[jira] [Commented] (HIVE-12369) Native Vector GroupBy
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451542#comment-16451542 ] Matt McCline commented on HIVE-12369: - Putting this change into hibernation for now. Not a priority to work on it for now. > Native Vector GroupBy > - > > Key: HIVE-12369 > URL: https://issues.apache.org/jira/browse/HIVE-12369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch, > HIVE-12369.05.patch, HIVE-12369.06.patch, HIVE-12369.091.patch, > HIVE-12369.094.patch, HIVE-12369.095.patch, HIVE-12369.096.patch, > HIVE-12369.097.patch, HIVE-12369.098.patch, HIVE-12369.099.patch, > HIVE-12369.0991.patch, HIVE-12369.0992.patch, HIVE-12369.0993.patch > > > Implement Native Vector GroupBy using fast hash table technology developed > for Native Vector MapJoin, etc. > Patch is currently limited to a single COUNT aggregation. Or, no aggregation > also known as duplicate reduction. > Here are examples of new kinds of classes introduces that stored the count in > the slot table and don't allocate hash elements: > {noformat} > COUNT(column) VectorGroupByHashLongKeySingleCountColumnOperator > COUNT(key) VectorGroupByHashLongKeySingleCountKeyOperator > COUNT(*) VectorGroupByHashLongKeySingleCountStarOperator > {noformat} > And the duplicate reduction operator a single key. Example: > {noformat} > VectorGroupByHashLongKeyDuplicateReductionOperator > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-12369) Native Vector GroupBy (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16448869#comment-16448869 ] Hive QA commented on HIVE-12369: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12920303/HIVE-12369.0993.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 42 failed/errored test(s), 14290 tests executed *Failed tests:* {noformat} TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=93) [infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q] TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed out) (batchId=217) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_in_or_dup] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0] (batchId=17) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2] (batchId=39) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_smb] (batchId=176) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats] (batchId=168) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part] (batchId=168) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_orc_nested_column_pruning] (batchId=159) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] (batchId=105) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval] (batchId=98) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace] (batchId=98) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff] (batchId=98) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe] (batchId=98) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=225) org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion (batchId=228) org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=228) org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232) org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel (batchId=235) org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions (batchId=242) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert (batchId=242) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll (batchId=242) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent (batchId=242) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead (batchId=242) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes (batchId=242) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill (batchId=242) org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth (batchId=254) org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookieNegative (batchId=254) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/10435/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10435/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10435/
[jira] [Commented] (HIVE-12369) Native Vector GroupBy (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16448862#comment-16448862 ] Hive QA commented on HIVE-12369: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 49s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 45s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 57s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 47s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 9s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 39s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 12s{color} | {color:red} root: The patch generated 222 new + 1487 unchanged - 60 fixed = 1709 total (was 1547) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} itests/hive-jmh: The patch generated 9 new + 56 unchanged - 0 fixed = 65 total (was 56) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 52s{color} | {color:red} ql: The patch generated 183 new + 712 unchanged - 37 fixed = 895 total (was 749) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s{color} | {color:red} vector-code-gen: The patch generated 30 new + 293 unchanged - 23 fixed = 323 total (was 316) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 10s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 60m 30s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile xml | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-10435/dev-support/hive-personality.sh | | git revision | master / 4f67beb | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10435/yetus/diff-checkstyle-root.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10435/yetus/diff-checkstyle-itests_hive-jmh.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10435/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10435/yetus/diff-checkstyle-vector-code-gen.txt | | modules | C: common . itests itests/hive-jmh ql vector-code-gen U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-10435/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Native Vector GroupBy (Part 1) > -- > > Key: HIVE-12369 > URL: https://issues.apache.org/jira/browse/HIVE-12369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >
[jira] [Commented] (HIVE-12369) Native Vector GroupBy (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16448196#comment-16448196 ] Hive QA commented on HIVE-12369: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12920243/HIVE-12369.0992.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 34 failed/errored test(s), 14289 tests executed *Failed tests:* {noformat} TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=93) [infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q] TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed out) (batchId=217) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_in_or_dup] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0] (batchId=17) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2] (batchId=39) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats] (batchId=168) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=163) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] (batchId=105) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval] (batchId=98) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace] (batchId=98) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff] (batchId=98) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe] (batchId=98) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=225) org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion (batchId=228) org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=228) org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232) org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235) org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel (batchId=235) org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239) org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken (batchId=254) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth (batchId=254) org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookieNegative (batchId=254) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/10430/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10430/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10430/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 34 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12920243 - PreCommit-HIVE-Build > Native Vector GroupBy (Part 1) > -- > > Key: HIVE-12369 > URL: https://issues.apache.org/jira/browse/HIVE-12369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments:
[jira] [Commented] (HIVE-12369) Native Vector GroupBy (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16448180#comment-16448180 ] Hive QA commented on HIVE-12369: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 1s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 52s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 26s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 37s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 44s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 1s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 21s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 17s{color} | {color:red} root: The patch generated 280 new + 1501 unchanged - 46 fixed = 1781 total (was 1547) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} itests/hive-jmh: The patch generated 13 new + 56 unchanged - 0 fixed = 69 total (was 56) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 50s{color} | {color:red} ql: The patch generated 227 new + 712 unchanged - 37 fixed = 939 total (was 749) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s{color} | {color:red} vector-code-gen: The patch generated 40 new + 307 unchanged - 9 fixed = 347 total (was 316) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 46s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 58m 39s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile xml | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-10430/dev-support/hive-personality.sh | | git revision | master / 334c8ca | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10430/yetus/diff-checkstyle-root.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10430/yetus/diff-checkstyle-itests_hive-jmh.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10430/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10430/yetus/diff-checkstyle-vector-code-gen.txt | | modules | C: common . itests itests/hive-jmh ql vector-code-gen U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-10430/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Native Vector GroupBy (Part 1) > -- > > Key: HIVE-12369 > URL: https://issues.apache.org/jira/browse/HIVE-12369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >
[jira] [Commented] (HIVE-12369) Native Vector GroupBy (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447660#comment-16447660 ] Hive QA commented on HIVE-12369: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12920218/HIVE-12369.0991.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 50 failed/errored test(s), 14288 tests executed *Failed tests:* {noformat} TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=93) [infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q] TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed out) (batchId=217) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_in_or_dup] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_struct_type_vectorization] (batchId=71) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0] (batchId=17) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_decimal_date] (batchId=83) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_columnPruning] (batchId=83) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_struct_type_vectorization] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_complex_types_vectorization] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_map_type_vectorization] (batchId=153) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_struct_type_vectorization] (batchId=149) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_types_vectorization] (batchId=149) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets] (batchId=170) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_vectorization] (batchId=166) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_partitioned] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_vector_nohybridgrace] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin] (batchId=169) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats] (batchId=168) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_adaptor_usage_mode] (batchId=173) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_sort_8] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_orc_nested_column_pruning] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_part_project] (batchId=159) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=182) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_inner_join] (batchId=184) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join0] (batchId=184) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join1] (batchId=183) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join2] (batchId=182) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join3] (batchId=183) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join4] (batchId=185) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join5] (batchId=185) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez] (batchId=106) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] (batchId=105) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval] (batchId=98) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace] (batchId=98) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
[jira] [Commented] (HIVE-12369) Native Vector GroupBy (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447641#comment-16447641 ] Hive QA commented on HIVE-12369: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 1s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 49s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 24s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 22s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 39s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 3s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 19s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 8s{color} | {color:red} root: The patch generated 280 new + 1501 unchanged - 46 fixed = 1781 total (was 1547) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s{color} | {color:red} itests/hive-jmh: The patch generated 13 new + 56 unchanged - 0 fixed = 69 total (was 56) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 52s{color} | {color:red} ql: The patch generated 227 new + 712 unchanged - 37 fixed = 939 total (was 749) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s{color} | {color:red} vector-code-gen: The patch generated 40 new + 307 unchanged - 9 fixed = 347 total (was 316) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 51s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 57m 43s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile xml | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-10424/dev-support/hive-personality.sh | | git revision | master / a1524f7 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10424/yetus/diff-checkstyle-root.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10424/yetus/diff-checkstyle-itests_hive-jmh.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10424/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10424/yetus/diff-checkstyle-vector-code-gen.txt | | modules | C: common . itests itests/hive-jmh ql vector-code-gen U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-10424/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Native Vector GroupBy (Part 1) > -- > > Key: HIVE-12369 > URL: https://issues.apache.org/jira/browse/HIVE-12369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >
[jira] [Commented] (HIVE-12369) Native Vector GroupBy (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447345#comment-16447345 ] Hive QA commented on HIVE-12369: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12920198/HIVE-12369.099.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/10414/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10414/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10414/ Messages: {noformat} This message was trimmed, see log for full details error: src/test/results/clientpositive/llap/vector_binary_join_groupby.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_cast_constant.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_char_2.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_coalesce_2.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_complex_all.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_count_distinct.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_data_types.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_decimal_aggregate.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_decimal_precision.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_decimal_udf.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_distinct_2.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_3.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_cube1.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_grouping_id1.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_grouping_id2.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_grouping_id3.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_grouping_sets1.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_grouping_sets2.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_grouping_sets3_dec.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_grouping_sets4.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_grouping_sets5.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_grouping_sets6.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_grouping_sets_grouping.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_grouping_window.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_mapjoin.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_reduce.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_rollup1.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_sort_11.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_groupby_sort_8.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_grouping_sets.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_inner_join.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_join30.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_left_outer_join.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_leftsemi_mapjoin.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_number_compare_projection.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_orc_nested_column_pruning.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_orderby_5.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_outer_join1.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_outer_join2.q.out: does not exist in index error: src/test/results/clientpositive/llap/vector_outer_reference_windowed.q.out: does not exist in index error:
[jira] [Commented] (HIVE-12369) Native Vector GroupBy (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447200#comment-16447200 ] Matt McCline commented on HIVE-12369: - The days when I hate git. fatal: git apply: bad git-diff - inconsistent old filename on line 7887 > Native Vector GroupBy (Part 1) > -- > > Key: HIVE-12369 > URL: https://issues.apache.org/jira/browse/HIVE-12369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch, > HIVE-12369.05.patch, HIVE-12369.06.patch, HIVE-12369.091.patch, > HIVE-12369.094.patch, HIVE-12369.095.patch, HIVE-12369.096.patch, > HIVE-12369.097.patch, HIVE-12369.098.patch, HIVE-12369.099.patch > > > Implement Native Vector GroupBy using fast hash table technology developed > for Native Vector MapJoin, etc. > Patch is currently limited to a single COUNT aggregation. Or, no aggregation > also known as duplicate reduction. > Here are examples of new kinds of classes introduces that stored the count in > the slot table and don't allocate hash elements: > {noformat} > COUNT(column) VectorGroupByHashLongKeySingleCountColumnOperator > COUNT(key) VectorGroupByHashLongKeySingleCountKeyOperator > COUNT(*) VectorGroupByHashLongKeySingleCountStarOperator > {noformat} > And the duplicate reduction operator a single key. Example: > {noformat} > VectorGroupByHashLongKeyDuplicateReductionOperator > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-12369) Native Vector GroupBy (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447184#comment-16447184 ] Hive QA commented on HIVE-12369: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12920166/HIVE-12369.098.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/10405/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10405/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10405/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2018-04-22 09:54:48.706 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-10405/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2018-04-22 09:54:48.720 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 84481cf HIVE-19167: Map data type doesn't keep the order of the key/values pairs as read (Part 2, The Sequel or SQL) (Matt McCline, reviewed by Deepak Jaiswal) + git clean -f -d + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 84481cf HIVE-19167: Map data type doesn't keep the order of the key/values pairs as read (Part 2, The Sequel or SQL) (Matt McCline, reviewed by Deepak Jaiswal) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2018-04-22 09:54:57.409 + rm -rf ../yetus_PreCommit-HIVE-Build-10405 + mkdir ../yetus_PreCommit-HIVE-Build-10405 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-10405 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-10405/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: cannot apply binary patch to 'itests/qtest/output/.00_0.crc' without full index line Falling back to three-way merge... error: cannot apply binary patch to 'itests/qtest/output/.00_0.crc' without full index line error: itests/qtest/output/.00_0.crc: patch does not apply error: cannot apply binary patch to 'itests/qtest/output/00_0' without full index line Falling back to three-way merge... error: cannot apply binary patch to 'itests/qtest/output/00_0' without full index line error: itests/qtest/output/00_0: patch does not apply fatal: git apply: bad git-diff - inconsistent old filename on line 7887 fatal: git diff header lacks filename information when removing 2 leading pathname components (line 641) The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12920166 - PreCommit-HIVE-Build > Native Vector GroupBy (Part 1) > -- > > Key: HIVE-12369 > URL: https://issues.apache.org/jira/browse/HIVE-12369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch, > HIVE-12369.05.patch, HIVE-12369.06.patch, HIVE-12369.091.patch, > HIVE-12369.094.patch, HIVE-12369.095.patch, HIVE-12369.096.patch, > HIVE-12369.097.patch, HIVE-12369.098.patch > > > Implement Native Vector GroupBy using fast hash table technology developed > for Native Vector MapJoin, etc. > Patch is currently limited to a single COUNT aggregation. Or, no aggregation > also known as duplicate reduction. > Here are examples of new kinds of classes introduces that stored the count in > the slot table and don't allocate hash elements: >
[jira] [Commented] (HIVE-12369) Native Vector GroupBy (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442604#comment-16442604 ] Hive QA commented on HIVE-12369: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12919386/HIVE-12369.096.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 60 failed/errored test(s), 14284 tests executed *Failed tests:* {noformat} TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed out) (batchId=217) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_in_or_dup] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_smb] (batchId=92) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0] (batchId=17) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[tez_join_hash] (batchId=54) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_vectorization] (batchId=166) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_sort_8] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_orc_nested_column_pruning] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_part_project] (batchId=159) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez] (batchId=106) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] (batchId=105) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_notnull_constraint_violation] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[avro_non_nullable_union] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[cachingprintstream] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[check_constraint_violation] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[compute_stats_long] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[default_constraint_invalid_default_value_type] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[dyn_part3] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[dyn_part_max_per_node] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[dynamic_partitions_with_whitelist] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_acid_notnull] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_notnull_constraint] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_multi_into_notnull] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_overwrite_notnull_constraint] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insertsel_fail] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[merge_constraint_notnull] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[script_broken_pipe2] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[script_broken_pipe3] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[script_error] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[serde_regex2] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_2] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_2] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_implicit_gby] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notin_implicit_gby] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_corr_multi_rows] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_multi_rows] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_assert_true2]
[jira] [Commented] (HIVE-12369) Native Vector GroupBy (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442579#comment-16442579 ] Hive QA commented on HIVE-12369: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 47s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 40s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 29s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 37s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 23s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 22s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 13s{color} | {color:red} root: The patch generated 258 new + 1445 unchanged - 62 fixed = 1703 total (was 1507) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 50s{color} | {color:red} ql: The patch generated 221 new + 712 unchanged - 53 fixed = 933 total (was 765) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s{color} | {color:red} vector-code-gen: The patch generated 37 new + 307 unchanged - 9 fixed = 344 total (was 316) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 33s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 14s{color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 57m 45s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile xml | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-10295/dev-support/hive-personality.sh | | git revision | master / 878d6ee | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10295/yetus/diff-checkstyle-root.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10295/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10295/yetus/diff-checkstyle-vector-code-gen.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-10295/yetus/whitespace-eol.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-10295/yetus/patch-asflicense-problems.txt | | modules | C: common . itests ql vector-code-gen U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-10295/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Native Vector GroupBy (Part 1) > -- > > Key: HIVE-12369 > URL: https://issues.apache.org/jira/browse/HIVE-12369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority:
[jira] [Commented] (HIVE-12369) Native Vector GroupBy (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439976#comment-16439976 ] Hive QA commented on HIVE-12369: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12919129/HIVE-12369.095.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 83 failed/errored test(s), 13826 tests executed *Failed tests:* {noformat} TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed out) (batchId=247) TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed out) (batchId=247) TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=96)
[jira] [Commented] (HIVE-12369) Native Vector GroupBy (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439863#comment-16439863 ] Hive QA commented on HIVE-12369: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 14s{color} | {color:red} /data/hiveptest/logs/PreCommit-HIVE-Build-10255/patches/PreCommit-HIVE-Build-10255.patch does not apply to master. Rebase required? Wrong Branch? See http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-10255/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Native Vector GroupBy (Part 1) > -- > > Key: HIVE-12369 > URL: https://issues.apache.org/jira/browse/HIVE-12369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch, > HIVE-12369.05.patch, HIVE-12369.06.patch, HIVE-12369.091.patch, > HIVE-12369.094.patch, HIVE-12369.095.patch > > > Implement Native Vector GroupBy using fast hash table technology developed > for Native Vector MapJoin, etc. > Patch is currently limited to a single Long key with a single COUNT > aggregation. Or, a single Long key and no aggregation also known as > duplicate reduction. > 3 new classes introduces that stored the count in the slot table and don't > allocate hash elements: > {noformat} > COUNT(column) VectorGroupByHashLongKeyCountColumnOperator > COUNT(key) VectorGroupByHashLongKeyCountKeyOperator > COUNT(*) VectorGroupByHashLongKeyCountStarOperator > {noformat} > And the duplicate reduction operator a single Long key: > {noformat} > VectorGroupByHashLongKeyDuplicateReductionOperator > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-12369) Native Vector GroupBy
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437333#comment-16437333 ] Hive QA commented on HIVE-12369: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12918705/HIVE-12369.094.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 115 failed/errored test(s), 13799 tests executed *Failed tests:* {noformat} TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed out) (batchId=247) TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed out) (batchId=247) TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=95)
[jira] [Commented] (HIVE-12369) Native Vector GroupBy
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437300#comment-16437300 ] Hive QA commented on HIVE-12369: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 44s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 11s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 55s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 25s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 10s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 5s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 23s{color} | {color:red} root: The patch generated 320 new + 1088 unchanged - 51 fixed = 1408 total (was 1139) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 48s{color} | {color:red} ql: The patch generated 288 new + 356 unchanged - 42 fixed = 644 total (was 398) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s{color} | {color:red} vector-code-gen: The patch generated 32 new + 307 unchanged - 9 fixed = 339 total (was 316) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 7s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 55m 32s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile xml | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-10184/dev-support/hive-personality.sh | | git revision | master / 6f9090c | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10184/yetus/diff-checkstyle-root.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10184/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10184/yetus/diff-checkstyle-vector-code-gen.txt | | modules | C: common . ql vector-code-gen U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-10184/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Native Vector GroupBy > - > > Key: HIVE-12369 > URL: https://issues.apache.org/jira/browse/HIVE-12369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch, > HIVE-12369.05.patch, HIVE-12369.06.patch, HIVE-12369.091.patch, > HIVE-12369.094.patch > > > Implement Native Vector GroupBy using fast hash table technology developed > for Native Vector MapJoin, etc. > Patch is currently
[jira] [Commented] (HIVE-12369) Native Vector GroupBy
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427975#comment-16427975 ] Hive QA commented on HIVE-12369: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12917609/HIVE-12369.091.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 324 failed/errored test(s), 13579 tests executed *Failed tests:* {noformat} TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=252) TestCopyUtils - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed out) (batchId=246) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=252) TestExportImport - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed out) (batchId=246) TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=252) TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=252) TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=96)
[jira] [Commented] (HIVE-12369) Native Vector GroupBy
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427906#comment-16427906 ] Hive QA commented on HIVE-12369: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 33s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 13s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 34s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 4s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 46s{color} | {color:red} ql: The patch generated 229 new + 336 unchanged - 22 fixed = 565 total (was 358) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 16s{color} | {color:red} The patch generated 50 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 19m 39s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-10030/dev-support/hive-personality.sh | | git revision | master / 2f802e9 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10030/yetus/diff-checkstyle-ql.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-10030/yetus/patch-asflicense-problems.txt | | modules | C: common ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-10030/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Native Vector GroupBy > - > > Key: HIVE-12369 > URL: https://issues.apache.org/jira/browse/HIVE-12369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch, > HIVE-12369.05.patch, HIVE-12369.06.patch, HIVE-12369.091.patch > > > Implement Native Vector GroupBy using fast hash table technology developed > for Native Vector MapJoin, etc. > Patch is currently limited to a single Long key with a single COUNT > aggregation. Or, a single Long key and no aggregation also known as > duplicate reduction. > 3 new classes introduces that stored the count in the slot table and don't > allocate hash elements: > {noformat} > COUNT(column) VectorGroupByHashLongKeyCountColumnOperator > COUNT(key) VectorGroupByHashLongKeyCountKeyOperator > COUNT(*) VectorGroupByHashLongKeyCountStarOperator > {noformat} > And the duplicate reduction operator a single Long key: > {noformat} > VectorGroupByHashLongKeyDuplicateReductionOperator > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-12369) Native Vector GroupBy
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217981#comment-16217981 ] Hive QA commented on HIVE-12369: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12879115/HIVE-12369.06.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7458/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7458/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7458/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-10-25 01:49:44.492 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-7458/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-10-25 01:49:44.495 + cd apache-github-source-source + git fetch origin >From https://github.com/apache/hive 42e70a3..84950cf master -> origin/master 74034f1..84e107b branch-2 -> origin/branch-2 + git reset --hard HEAD HEAD is now at 42e70a3 HIVE-17471 : Vectorization: Enable hive.vectorized.row.identifier.enabled to true by default (Sergey Shelukhin, reviewed by Matt McCline) + git clean -f -d Removing common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig Removing itests/src/test/resources/testconfiguration.properties.orig Removing ql/src/test/queries/clientpositive/acid_vectorization_original.q Removing ql/src/test/results/clientpositive/llap/acid_vectorization_original.q.out Removing ql/src/test/results/clientpositive/tez/acid_vectorization_original.q.out Removing standalone-metastore/src/gen/org/ + git checkout master Already on 'master' Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded. (use "git pull" to update your local branch) + git reset --hard origin/master HEAD is now at 84950cf HIVE-17764 : alter view fails when hive.metastore.disallow.incompatible.col.type.changes set to true (Janaki Lahorani, reviewed by Andrew Sherman and Vihang Karajgaonkar) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-10-25 01:49:49.788 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java:20 error: ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java: patch does not apply error: patch failed: ql/src/test/results/clientpositive/llap/sysdb.q.out:3346 error: ql/src/test/results/clientpositive/llap/sysdb.q.out: patch does not apply error: patch failed: ql/src/test/results/clientpositive/llap/vector_aggregate_9.q.out:147 error: ql/src/test/results/clientpositive/llap/vector_aggregate_9.q.out: patch does not apply error: patch failed: ql/src/test/results/clientpositive/llap/vector_between_in.q.out:157 error: ql/src/test/results/clientpositive/llap/vector_between_in.q.out: patch does not apply error: patch failed: ql/src/test/results/clientpositive/llap/vector_count_distinct.q.out:1327 error: ql/src/test/results/clientpositive/llap/vector_count_distinct.q.out: patch does not apply error: patch failed: ql/src/test/results/clientpositive/llap/vector_decimal_precision.q.out:589 error: ql/src/test/results/clientpositive/llap/vector_decimal_precision.q.out: patch does not apply error: ql/src/test/results/clientpositive/llap/vector_empty_where.q.out: No such file or directory error: patch failed: ql/src/test/results/clientpositive/llap/vector_groupby_grouping_id2.q.out:609 error: ql/src/test/results/clientpositive/llap/vector_groupby_grouping_id2.q.out: patch does not apply error: patch failed:
[jira] [Commented] (HIVE-12369) Native Vector GroupBy
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216452#comment-16216452 ] Teddy Choi commented on HIVE-12369: --- I tried to merge this patch to current master branch, but I got many conflicts. After resolving them, I got NPE in TestVectorGroupByOperator. Would you rebase this patch? Thanks. > Native Vector GroupBy > - > > Key: HIVE-12369 > URL: https://issues.apache.org/jira/browse/HIVE-12369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch, > HIVE-12369.05.patch, HIVE-12369.06.patch > > > Implement Native Vector GroupBy using fast hash table technology developed > for Native Vector MapJoin, etc. > Patch is currently limited to a single Long key, aggregation on Long columns, > no more than 31 columns. > 3 new classes introduces that stored the count in the slot table and don't > allocate hash elements: > {noformat} > COUNT(column) VectorGroupByHashOneLongKeyCountColumnOperator > COUNT(key) VectorGroupByHashOneLongKeyCountKeyOperator > COUNT(*) VectorGroupByHashOneLongKeyCountStarOperator > {noformat} > And a new class that aggregates a single Long key: > {noformat} > VectorGroupByHashOneLongKeyOperator > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-12369) Native Vector GroupBy
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113621#comment-16113621 ] Sergey Shelukhin commented on HIVE-12369: - Finished the first iteration of the diff #3. My main comment is that the patch needs more comments, esp. for things that are used elsewhere and hard to understand without context. Also left some logic-specific/etc. comments on RB. I didn't read all the code in great detail in the parts that look very similar for different types and cases. > Native Vector GroupBy > - > > Key: HIVE-12369 > URL: https://issues.apache.org/jira/browse/HIVE-12369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch, > HIVE-12369.05.patch, HIVE-12369.06.patch > > > Implement Native Vector GroupBy using fast hash table technology developed > for Native Vector MapJoin, etc. > Patch is currently limited to a single Long key, aggregation on Long columns, > no more than 31 columns. > 3 new classes introduces that stored the count in the slot table and don't > allocate hash elements: > {noformat} > COUNT(column) VectorGroupByHashOneLongKeyCountColumnOperator > COUNT(key) VectorGroupByHashOneLongKeyCountKeyOperator > COUNT(*) VectorGroupByHashOneLongKeyCountStarOperator > {noformat} > And a new class that aggregates a single Long key: > {noformat} > VectorGroupByHashOneLongKeyOperator > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-12369) Native Vector GroupBy
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112186#comment-16112186 ] Matt McCline commented on HIVE-12369: - Yes, I think you continue reviewing. The path that is implemented is One Long Key and groupByMode == HASH. There are UNDONEs for *subsequent* JIRAs that later adds Aggregation of non-Long data types, Fixed Length Keys / Variable Length Keys, and the other groupByModes. And later adds Grouping Sets, Empty Aggregation (i.e. GroupBy on key that has no aggregations that does duplicate key elimination), too. > Native Vector GroupBy > - > > Key: HIVE-12369 > URL: https://issues.apache.org/jira/browse/HIVE-12369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch, > HIVE-12369.05.patch, HIVE-12369.06.patch > > > Implement Native Vector GroupBy using fast hash table technology developed > for Native Vector MapJoin, etc. > Patch is currently limited to a single Long key, aggregation on Long columns, > no more than 31 columns. > 3 new classes introduces that stored the count in the slot table and don't > allocate hash elements: > {noformat} > COUNT(column) VectorGroupByHashOneLongKeyCountColumnOperator > COUNT(key) VectorGroupByHashOneLongKeyCountKeyOperator > COUNT(*) VectorGroupByHashOneLongKeyCountStarOperator > {noformat} > And a new class that aggregates a single Long key: > {noformat} > VectorGroupByHashOneLongKeyOperator > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-12369) Native Vector GroupBy
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112093#comment-16112093 ] Sergey Shelukhin commented on HIVE-12369: - Hmm. Reviewed most of page one. Should I be reviewing this? It looks like lots of stuff is UNDONE (not implemented?). I can also review stuff now and then the diff of the diffs, but I wonder if it makes sense, i.e. whether the upcoming changes are going to be reasonable in scope for that. > Native Vector GroupBy > - > > Key: HIVE-12369 > URL: https://issues.apache.org/jira/browse/HIVE-12369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch, > HIVE-12369.05.patch, HIVE-12369.06.patch > > > Implement Native Vector GroupBy using fast hash table technology developed > for Native Vector MapJoin, etc. > Patch is currently limited to a single Long key, aggregation on Long columns, > no more than 31 columns. > 3 new classes introduces that stored the count in the slot table and don't > allocate hash elements: > {noformat} > COUNT(column) VectorGroupByHashOneLongKeyCountColumnOperator > COUNT(key) VectorGroupByHashOneLongKeyCountKeyOperator > COUNT(*) VectorGroupByHashOneLongKeyCountStarOperator > {noformat} > And a new class that aggregates a single Long key: > {noformat} > VectorGroupByHashOneLongKeyOperator > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-12369) Native Vector GroupBy
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112077#comment-16112077 ] Sergey Shelukhin commented on HIVE-12369: - I started reviewing this... it will take some time, probably over a couple days. Will publish review in parts > Native Vector GroupBy > - > > Key: HIVE-12369 > URL: https://issues.apache.org/jira/browse/HIVE-12369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch, > HIVE-12369.05.patch, HIVE-12369.06.patch > > > Implement Native Vector GroupBy using fast hash table technology developed > for Native Vector MapJoin, etc. > Patch is currently limited to a single Long key, aggregation on Long columns, > no more than 31 columns. > 3 new classes introduces that stored the count in the slot table and don't > allocate hash elements: > {noformat} > COUNT(column) VectorGroupByHashOneLongKeyCountColumnOperator > COUNT(key) VectorGroupByHashOneLongKeyCountKeyOperator > COUNT(*) VectorGroupByHashOneLongKeyCountStarOperator > {noformat} > And a new class that aggregates a single Long key: > {noformat} > VectorGroupByHashOneLongKeyOperator > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-12369) Native Vector GroupBy
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103127#comment-16103127 ] Hive QA commented on HIVE-12369: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12879115/HIVE-12369.06.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 55 failed/errored test(s), 10620 tests executed *Failed tests:* {noformat} TestPerfCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] (batchId=51) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] (batchId=12) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=60) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=74) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_count] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_simple] (batchId=69) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_part] (batchId=29) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_part_varchar] (batchId=39) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=144) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_join_hash] (batchId=155) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_count] (batchId=147) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning] (batchId=168) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=168) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_inner_join] (batchId=170) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join1] (batchId=169) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join2] (batchId=168) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vectorization_limit] (batchId=99) org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver (batchId=89) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_between_in] (batchId=125) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_cast_constant] (batchId=105) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_count_distinct] (batchId=112) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_decimal_aggregate] (batchId=109) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_distinct_2] (batchId=123) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_groupby_3] (batchId=128) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_mapjoin_reduce] (batchId=135) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_orderby_5] (batchId=119) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_string_concat] (batchId=115) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_0] (batchId=136) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_12] (batchId=105) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_13] (batchId=122) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_14] (batchId=107) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_15] (batchId=128) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_16] (batchId=119) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_1] (batchId=126) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_2] (batchId=110) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_3] (batchId=134) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_4] (batchId=110) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_5] (batchId=125) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_9] (batchId=101) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_part] (batchId=114) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_short_regress] (batchId=122) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_case] (batchId=125) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_mapjoin] (batchId=132)
[jira] [Commented] (HIVE-12369) Native Vector GroupBy
[ https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102777#comment-16102777 ] Matt McCline commented on HIVE-12369: - #5 cratered. > Native Vector GroupBy > - > > Key: HIVE-12369 > URL: https://issues.apache.org/jira/browse/HIVE-12369 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch, > HIVE-12369.05.patch, HIVE-12369.06.patch > > > Implement Native Vector GroupBy using fast hash table technology developed > for Native Vector MapJoin, etc. > Patch is currently limited to a single Long key, aggregation on Long columns, > no more than 31 columns. > 3 new classes introduces that stored the count in the slot table and don't > allocate hash elements: > {noformat} > COUNT(column) VectorGroupByHashOneLongKeyCountColumnOperator > COUNT(key) VectorGroupByHashOneLongKeyCountKeyOperator > COUNT(*) VectorGroupByHashOneLongKeyCountStarOperator > {noformat} > And a new class that aggregates a single Long key: > {noformat} > VectorGroupByHashOneLongKeyOperator > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)