[jira] [Commented] (HIVE-18048) Vectorization: Support Struct type with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351972#comment-16351972 ] Colin Ma commented on HIVE-18048: - Hi [~Ferd], I checked the failed test cases, and they are not patch related, no vectorization or struct tyep related, eg: TestTriggersNoTezSessionPool, TestTablesCreateDropAlterTruncate, TestTablesList. > Vectorization: Support Struct type with vectorization > - > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Improvement >Reporter: Colin Ma >Assignee: Colin Ma >Priority: Major > Attachments: HIVE-18048.001.patch, HIVE-18048.002.patch, > HIVE-18048.003.patch, HIVE-18048.004.patch, HIVE-18048.005.patch, > HIVE-18048.006.patch, HIVE-18048.007.patch > > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > New UDF will be added to access the field of Struct. > Note: > * Nested complex type won't be tested in this ticket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18048) Vectorization: Support Struct type with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351969#comment-16351969 ] Ferdinand Xu commented on HIVE-18048: - Hi [~colinma], could you check the failed test cases? It looks like 4 extra test cases comparing with the last time. Thank you! > Vectorization: Support Struct type with vectorization > - > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Improvement >Reporter: Colin Ma >Assignee: Colin Ma >Priority: Major > Attachments: HIVE-18048.001.patch, HIVE-18048.002.patch, > HIVE-18048.003.patch, HIVE-18048.004.patch, HIVE-18048.005.patch, > HIVE-18048.006.patch, HIVE-18048.007.patch > > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > New UDF will be added to access the field of Struct. > Note: > * Nested complex type won't be tested in this ticket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18048) Vectorization: Support Struct type with vectorization
[
https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346502#comment-16346502
]
Hive QA commented on HIVE-18048:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908469/HIVE-18048.007.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 12968 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testGetIndexEmptyTableName[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerSlowQueryExecutionTime
(batchId=235)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerTotalLaunchedTasks
(batchId=235)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerVertexTotalTasks
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8943/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8943/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8943/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908469 - PreCommit-HIVE-Build
> Vectorization: Support Struct type with vectorization
> -
>
> Key: HIVE-18048
> URL: https://issues.apache.org/jira/browse/HIVE-18048
> Project: Hive
> Issue Type: Improvement
>Reporter: Colin Ma
>Assignee: Colin Ma
>Priority: Major
> Attachments: HIVE-18048.001.patch, HIVE-18048.002.patch,
> HIVE-18048.003.patch, HIVE-18048.004.patch, HIVE-18048.005.patch,
> HIVE-18048.006.patch, HIVE-18048.007.patch
>
>
> Struct type is not supported in MapWork with vectorization, it should be
> supported to improve the performance.
> New UDF will be added to access the field of Struct.
> Note:
> * Nested complex type won't be tested in this ticket.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18048) Vectorization: Support Struct type with vectorization
[
https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346489#comment-16346489
]
Hive QA commented on HIVE-18048:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 79e4740 |
| Default Java | 1.8.0_111 |
| modules | C: ql . itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8943/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Support Struct type with vectorization
> -
>
> Key: HIVE-18048
> URL: https://issues.apache.org/jira/browse/HIVE-18048
> Project: Hive
> Issue Type: Improvement
>Reporter: Colin Ma
>Assignee: Colin Ma
>Priority: Major
> Attachments: HIVE-18048.001.patch, HIVE-18048.002.patch,
> HIVE-18048.003.patch, HIVE-18048.004.patch, HIVE-18048.005.patch,
> HIVE-18048.006.patch, HIVE-18048.007.patch
>
>
> Struct type is not supported in MapWork with vectorization, it should be
> supported to improve the performance.
> New UDF will be added to access the field of Struct.
> Note:
> * Nested complex type won't be tested in this ticket.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18048) Vectorization: Support Struct type with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346388#comment-16346388 ] Ferdinand Xu commented on HIVE-18048: - LGTM +1 pending on the test. > Vectorization: Support Struct type with vectorization > - > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Improvement >Reporter: Colin Ma >Assignee: Colin Ma >Priority: Major > Attachments: HIVE-18048.001.patch, HIVE-18048.002.patch, > HIVE-18048.003.patch, HIVE-18048.004.patch, HIVE-18048.005.patch, > HIVE-18048.006.patch, HIVE-18048.007.patch > > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > New UDF will be added to access the field of Struct. > Note: > * Nested complex type won't be tested in this ticket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18048) Vectorization: Support Struct type with vectorization
[
https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344907#comment-16344907
]
Hive QA commented on HIVE-18048:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908282/HIVE-18048.006.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 12866 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8925/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8925/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8925/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908282 - PreCommit-HIVE-Build
> Vectorization: Support Struct type with vectorization
> -
>
> Key: HIVE-18048
> URL: https://issues.apache.org/jira/browse/HIVE-18048
> Project: Hive
> Issue Type: Improvement
>Reporter: Colin Ma
>Assignee: Colin Ma
>Priority: Major
> Attachments: HIVE-18048.001.patch, HIVE-18048.002.patch,
> HIVE-18048.003.patch, HIVE-18048.004.patch, HIVE-18048.005.patch,
> HIVE-18048.006.patch
>
>
> Struct type is not supported in MapWork with vectorization, it should be
> supported to improve the performance.
> New UDF will be added to access the field of Struct.
> Note:
> * Filter operator won't be supported.
> * Nested complex type won't be tested in this ticket.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18048) Vectorization: Support Struct type with vectorization
[
https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344881#comment-16344881
]
Hive QA commented on HIVE-18048:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 9a9f7de |
| Default Java | 1.8.0_111 |
| modules | C: ql . itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8925/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Support Struct type with vectorization
> -
>
> Key: HIVE-18048
> URL: https://issues.apache.org/jira/browse/HIVE-18048
> Project: Hive
> Issue Type: Improvement
>Reporter: Colin Ma
>Assignee: Colin Ma
>Priority: Major
> Attachments: HIVE-18048.001.patch, HIVE-18048.002.patch,
> HIVE-18048.003.patch, HIVE-18048.004.patch, HIVE-18048.005.patch,
> HIVE-18048.006.patch
>
>
> Struct type is not supported in MapWork with vectorization, it should be
> supported to improve the performance.
> New UDF will be added to access the field of Struct.
> Note:
> * Filter operator won't be supported.
> * Nested complex type won't be tested in this ticket.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18048) Vectorization: Support Struct type with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344596#comment-16344596 ] Colin Ma commented on HIVE-18048: - [~Ferd], thanks for review. The patch is updated as your comments, new tests for ORC and filter operator are added. > Vectorization: Support Struct type with vectorization > - > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Improvement >Reporter: Colin Ma >Assignee: Colin Ma >Priority: Major > Attachments: HIVE-18048.001.patch, HIVE-18048.002.patch, > HIVE-18048.003.patch, HIVE-18048.004.patch, HIVE-18048.005.patch, > HIVE-18048.006.patch > > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > New UDF will be added to access the field of Struct. > Note: > * Filter operator won't be supported. > * Nested complex type won't be tested in this ticket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
