[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14533405#comment-14533405 ] Laljo John Pullokkaran commented on HIVE-10526: --- Committed to master/trunk, branch-1.2. > CBO (Calcite Return Path): HiveCost epsilon comparison should take row count > in to account > -- > > Key: HIVE-10526 > URL: https://issues.apache.org/jira/browse/HIVE-10526 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 0.12.0 >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Fix For: 1.2.0 > > Attachments: HIVE-10526.1.patch, HIVE-10526.2.patch, > HIVE-10526.3.patch, HIVE-10526.4.patch, HIVE-10526.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14532152#comment-14532152 ] Laljo John Pullokkaran commented on HIVE-10526: --- Most of the failures are unrelated except for two. Uploaded new patch. > CBO (Calcite Return Path): HiveCost epsilon comparison should take row count > in to account > -- > > Key: HIVE-10526 > URL: https://issues.apache.org/jira/browse/HIVE-10526 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 0.12.0 >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Fix For: 1.2.0 > > Attachments: HIVE-10526.1.patch, HIVE-10526.2.patch, > HIVE-10526.3.patch, HIVE-10526.4.patch, HIVE-10526.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14531871#comment-14531871 ] Hive QA commented on HIVE-10526: {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12730890/HIVE-10526.3.patch {color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 8904 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_parts org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_unencrypted_tbl org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_load_data_to_encrypted_tables org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_2 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_disallow_transform org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_droppartition org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_sba_drop_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_alterpart_loc org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropView org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessing org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessingCustomSetWhitelistAppend {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3787/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3787/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3787/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 25 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12730890 - PreCommit-HIVE-TRUNK-Build > CBO (Calcite Return Path): HiveCost epsilon comparison should take row count > in to account > -- > > Key: HIVE-10526 > URL: https://issues.apache.org/jira/browse/HIVE-10526 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 0.12.0 >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Fix For: 1.2.0 > > Attachments: HIVE-10526.1.patch, HIVE-10526.2.patch, > HIVE-10526.3.patch, HIVE-10526.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14531344#comment-14531344 ] Ashutosh Chauhan commented on HIVE-10526: - +1 LGTM > CBO (Calcite Return Path): HiveCost epsilon comparison should take row count > in to account > -- > > Key: HIVE-10526 > URL: https://issues.apache.org/jira/browse/HIVE-10526 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 0.12.0 >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Fix For: 1.2.0 > > Attachments: HIVE-10526.1.patch, HIVE-10526.2.patch, > HIVE-10526.3.patch, HIVE-10526.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529584#comment-14529584 ] Sushanth Sowmyan commented on HIVE-10526: - I don't see this picked up in the test commit queue, and it's possible it'll fail out saying it's already processed this file, so I'm going to re-upload .1.patch as .2.patch and manually submit this into the queue. > CBO (Calcite Return Path): HiveCost epsilon comparison should take row count > in to account > -- > > Key: HIVE-10526 > URL: https://issues.apache.org/jira/browse/HIVE-10526 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 0.12.0 >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Fix For: 1.2.0 > > Attachments: HIVE-10526.1.patch, HIVE-10526.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529502#comment-14529502 ] Laljo John Pullokkaran commented on HIVE-10526: --- uploaded modified patch last week. For some reason QA run didn't kick in. > CBO (Calcite Return Path): HiveCost epsilon comparison should take row count > in to account > -- > > Key: HIVE-10526 > URL: https://issues.apache.org/jira/browse/HIVE-10526 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 0.12.0 >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Fix For: 1.2.0 > > Attachments: HIVE-10526.1.patch, HIVE-10526.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523540#comment-14523540 ] Laljo John Pullokkaran commented on HIVE-10526: --- [~ashutoshc] The plan changes are expected. Let me analyze them, will upload the modified patch. > CBO (Calcite Return Path): HiveCost epsilon comparison should take row count > in to account > -- > > Key: HIVE-10526 > URL: https://issues.apache.org/jira/browse/HIVE-10526 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 0.12.0 >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Fix For: 1.2.0 > > Attachments: HIVE-10526.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523520#comment-14523520 ] Ashutosh Chauhan commented on HIVE-10526: - [~jpullokkaran] failures are related. My assumption is these are same set of plan updates which we brought in at the time of merge of cbo branch into trunk. Now that old cost model is fixed, we are getting plans as before the merge. So, these plan changes should be ok. Can you double check and then update the golden files. > CBO (Calcite Return Path): HiveCost epsilon comparison should take row count > in to account > -- > > Key: HIVE-10526 > URL: https://issues.apache.org/jira/browse/HIVE-10526 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 0.12.0 >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Fix For: 1.2.0 > > Attachments: HIVE-10526.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14522862#comment-14522862 ] Hive QA commented on HIVE-10526: {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12728980/HIVE-10526.patch {color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 8830 tests executed *Failed tests:* {noformat} TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a TEST-*.xml file TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join_pkfk org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_alt_syntax org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_4 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join32 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join32_lessSize org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join33 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_alt_syntax org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_4 org.apache.hive.jdbc.TestSSL.testSSLVersion {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3676/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3676/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3676/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 28 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12728980 - PreCommit-HIVE-TRUNK-Build > CBO (Calcite Return Path): HiveCost epsilon comparison should take row count > in to account > -- > > Key: HIVE-10526 > URL: https://issues.apache.org/jira/browse/HIVE-10526 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Fix For: 1.2.0 > > Attachments: HIVE-10526.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520037#comment-14520037 ] Laljo John Pullokkaran commented on HIVE-10526: --- yes. Will add comment. > CBO (Calcite Return Path): HiveCost epsilon comparison should take row count > in to account > -- > > Key: HIVE-10526 > URL: https://issues.apache.org/jira/browse/HIVE-10526 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Fix For: 1.2.0 > > Attachments: HIVE-10526.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519884#comment-14519884 ] Ashutosh Chauhan commented on HIVE-10526: - Is it because CPU & IO are normalized to same domain (query runtime ?) while # of rows are not. If so, it will be good to add this note as a comment there. Otherwise, looks good +1. > CBO (Calcite Return Path): HiveCost epsilon comparison should take row count > in to account > -- > > Key: HIVE-10526 > URL: https://issues.apache.org/jira/browse/HIVE-10526 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Fix For: 1.2.0 > > Attachments: HIVE-10526.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518440#comment-14518440 ] Laljo John Pullokkaran commented on HIVE-10526: --- I am not sure about that. We are trying to say if Latency (CPU + IO) is less than epsilon and if row count difference is less than epsilon. By separating CPU, IO we may loose the additive effect. > CBO (Calcite Return Path): HiveCost epsilon comparison should take row count > in to account > -- > > Key: HIVE-10526 > URL: https://issues.apache.org/jira/browse/HIVE-10526 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Fix For: 1.2.0 > > Attachments: HIVE-10526.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518417#comment-14518417 ] Ashutosh Chauhan commented on HIVE-10526: - Wondering if its better to separate io & cpu cost as well : {code} return (this == other) || ((Math.abs((this.io) - (other.getio())) < RelOptUtil.EPSILON) &&((Math.abs((this.cpu) - (other.getCpu())) < RelOptUtil.EPSILON) && (Math.abs((this.rowCount - other.getRows())) < RelOptUtil.EPSILON)); {code} > CBO (Calcite Return Path): HiveCost epsilon comparison should take row count > in to account > -- > > Key: HIVE-10526 > URL: https://issues.apache.org/jira/browse/HIVE-10526 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Fix For: 1.2.0 > > Attachments: HIVE-10526.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10526) CBO (Calcite Return Path): HiveCost epsilon comparison should take row count in to account
[ https://issues.apache.org/jira/browse/HIVE-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518393#comment-14518393 ] Laljo John Pullokkaran commented on HIVE-10526: --- [~ashutoshc] Could you review this? > CBO (Calcite Return Path): HiveCost epsilon comparison should take row count > in to account > -- > > Key: HIVE-10526 > URL: https://issues.apache.org/jira/browse/HIVE-10526 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Fix For: 1.2.0 > > Attachments: HIVE-10526.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)