[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532152#comment-14532152 ] Laljo John Pullokkaran commented on HIVE-10526: --- Most of the failures are unrelated except for two. Uploaded new patch. CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account -- Key: HIVE-10526 URL: https://issues.apache.org/jira/browse/HIVE-10526 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: 0.12.0 Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 1.2.0 Attachments: HIVE-10526.1.patch, HIVE-10526.2.patch, HIVE-10526.3.patch, HIVE-10526.4.patch, HIVE-10526.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533405#comment-14533405 ] Laljo John Pullokkaran commented on HIVE-10526: --- Committed to master/trunk, branch-1.2. CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account -- Key: HIVE-10526 URL: https://issues.apache.org/jira/browse/HIVE-10526 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: 0.12.0 Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 1.2.0 Attachments: HIVE-10526.1.patch, HIVE-10526.2.patch, HIVE-10526.3.patch, HIVE-10526.4.patch, HIVE-10526.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14531344#comment-14531344 ] Ashutosh Chauhan commented on HIVE-10526: - +1 LGTM CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account -- Key: HIVE-10526 URL: https://issues.apache.org/jira/browse/HIVE-10526 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: 0.12.0 Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 1.2.0 Attachments: HIVE-10526.1.patch, HIVE-10526.2.patch, HIVE-10526.3.patch, HIVE-10526.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[
https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14531871#comment-14531871
]
Hive QA commented on HIVE-10526:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12730890/HIVE-10526.3.patch
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 8904 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_parts
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_unencrypted_tbl
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_load_data_to_encrypted_tables
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_disallow_transform
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_droppartition
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_sba_drop_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_alterpart_loc
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropView
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessing
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessingCustomSetWhitelistAppend
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3787/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3787/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3787/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12730890 - PreCommit-HIVE-TRUNK-Build
CBO (Calcite Return Path): HiveCost epsilon comparison should take row count
in to account
--
Key: HIVE-10526
URL: https://issues.apache.org/jira/browse/HIVE-10526
Project: Hive
Issue Type: Sub-task
Components: CBO
Affects Versions: 0.12.0
Reporter: Laljo John Pullokkaran
Assignee: Laljo John Pullokkaran
Fix For: 1.2.0
Attachments: HIVE-10526.1.patch, HIVE-10526.2.patch,
HIVE-10526.3.patch, HIVE-10526.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14529502#comment-14529502 ] Laljo John Pullokkaran commented on HIVE-10526: --- uploaded modified patch last week. For some reason QA run didn't kick in. CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account -- Key: HIVE-10526 URL: https://issues.apache.org/jira/browse/HIVE-10526 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: 0.12.0 Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 1.2.0 Attachments: HIVE-10526.1.patch, HIVE-10526.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14529584#comment-14529584 ] Sushanth Sowmyan commented on HIVE-10526: - I don't see this picked up in the test commit queue, and it's possible it'll fail out saying it's already processed this file, so I'm going to re-upload .1.patch as .2.patch and manually submit this into the queue. CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account -- Key: HIVE-10526 URL: https://issues.apache.org/jira/browse/HIVE-10526 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: 0.12.0 Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 1.2.0 Attachments: HIVE-10526.1.patch, HIVE-10526.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[
https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522862#comment-14522862
]
Hive QA commented on HIVE-10526:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12728980/HIVE-10526.patch
{color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 8830 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join_pkfk
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_alt_syntax
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join32
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join32_lessSize
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join33
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_alt_syntax
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_4
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3676/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3676/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3676/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 28 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12728980 - PreCommit-HIVE-TRUNK-Build
CBO (Calcite Return Path): HiveCost epsilon comparison should take row count
in to account
--
Key: HIVE-10526
URL: https://issues.apache.org/jira/browse/HIVE-10526
Project: Hive
Issue Type: Sub-task
Components: CBO
Reporter: Laljo John Pullokkaran
Assignee: Laljo John Pullokkaran
Fix For: 1.2.0
Attachments: HIVE-10526.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523520#comment-14523520 ] Ashutosh Chauhan commented on HIVE-10526: - [~jpullokkaran] failures are related. My assumption is these are same set of plan updates which we brought in at the time of merge of cbo branch into trunk. Now that old cost model is fixed, we are getting plans as before the merge. So, these plan changes should be ok. Can you double check and then update the golden files. CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account -- Key: HIVE-10526 URL: https://issues.apache.org/jira/browse/HIVE-10526 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: 0.12.0 Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 1.2.0 Attachments: HIVE-10526.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523540#comment-14523540 ] Laljo John Pullokkaran commented on HIVE-10526: --- [~ashutoshc] The plan changes are expected. Let me analyze them, will upload the modified patch. CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account -- Key: HIVE-10526 URL: https://issues.apache.org/jira/browse/HIVE-10526 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: 0.12.0 Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 1.2.0 Attachments: HIVE-10526.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518393#comment-14518393 ] Laljo John Pullokkaran commented on HIVE-10526: --- [~ashutoshc] Could you review this? CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account -- Key: HIVE-10526 URL: https://issues.apache.org/jira/browse/HIVE-10526 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 1.2.0 Attachments: HIVE-10526.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518440#comment-14518440 ] Laljo John Pullokkaran commented on HIVE-10526: --- I am not sure about that. We are trying to say if Latency (CPU + IO) is less than epsilon and if row count difference is less than epsilon. By separating CPU, IO we may loose the additive effect. CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account -- Key: HIVE-10526 URL: https://issues.apache.org/jira/browse/HIVE-10526 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 1.2.0 Attachments: HIVE-10526.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[
https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518417#comment-14518417
]
Ashutosh Chauhan commented on HIVE-10526:
-
Wondering if its better to separate io cpu cost as well :
{code}
return (this == other) ||
((Math.abs((this.io) - (other.getio())) RelOptUtil.EPSILON)
((Math.abs((this.cpu) - (other.getCpu())) RelOptUtil.EPSILON)
(Math.abs((this.rowCount - other.getRows())) RelOptUtil.EPSILON));
{code}
CBO (Calcite Return Path): HiveCost epsilon comparison should take row count
in to account
--
Key: HIVE-10526
URL: https://issues.apache.org/jira/browse/HIVE-10526
Project: Hive
Issue Type: Sub-task
Components: CBO
Reporter: Laljo John Pullokkaran
Assignee: Laljo John Pullokkaran
Fix For: 1.2.0
Attachments: HIVE-10526.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
