[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299154#comment-16299154 ] Matt McCline commented on HIVE-17994: - Committed to master. > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch, > HIVE-17994.03.patch, HIVE-17994.04.patch, HIVE-17994.05.patch, > HIVE-17994.06.patch, vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299153#comment-16299153 ] Matt McCline commented on HIVE-17994: - Thank you [~teddy.choi] for your review and [~gopalv] for your help. > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch, > HIVE-17994.03.patch, HIVE-17994.04.patch, HIVE-17994.05.patch, > HIVE-17994.06.patch, vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298864#comment-16298864 ] Gopal V commented on HIVE-17994: [~teddy.choi]: this looks like it is a threading only issue - my guess is that the hash memory is interleaved across NUMA zones and gets moved around for some reason, when so many threads read it (AFAIK hash-lookups are just reads, so won't trigger NUMA migrations, but there might be something else going on). > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch, > HIVE-17994.03.patch, HIVE-17994.04.patch, HIVE-17994.05.patch, > HIVE-17994.06.patch, vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298831#comment-16298831 ] Teddy Choi commented on HIVE-17994: --- +1. LGTM. I didn't expect that it takes long to handle types with enum. It would be much better with this patch. Thanks. > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch, > HIVE-17994.03.patch, HIVE-17994.04.patch, HIVE-17994.05.patch, > HIVE-17994.06.patch, vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[
https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293788#comment-16293788
]
Hive QA commented on HIVE-17994:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12902481/HIVE-17994.06.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 11528 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[quotedid_smb]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_10]
(batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_7]
(batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=113)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=209)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=226)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
(batchId=236)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=233)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8290/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8290/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8290/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12902481 - PreCommit-HIVE-Build
> Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
> --
>
> Key: HIVE-17994
> URL: https://issues.apache.org/jira/browse/HIVE-17994
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Minor
> Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch,
> HIVE-17994.03.patch, HIVE-17994.04.patch, HIVE-17994.05.patch,
> HIVE-17994.06.patch, vec-serialize-hashmap.png
>
>
> On machines with slower NUMA, the hashmap lookup for
> TypeInfo::getPrimitiveCategory is the slowest part of the vectorized
> serialization loops. The static object references run hot with the NUMA
> access speeds penalizing half the threads.
> This lookup is done for every column, for every row - though vectorization
> enforces that this type cannot change at all.
> !vec-serialize-hashmap.png!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[
https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293774#comment-16293774
]
Hive QA commented on HIVE-17994:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
35s{color} | {color:red} ql: The patch generated 9 new + 279 unchanged - 1
fixed = 288 total (was 280) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / f52e8b4 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8290/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8290/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8290/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
> --
>
> Key: HIVE-17994
> URL: https://issues.apache.org/jira/browse/HIVE-17994
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Minor
> Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch,
> HIVE-17994.03.patch, HIVE-17994.04.patch, HIVE-17994.05.patch,
> HIVE-17994.06.patch, vec-serialize-hashmap.png
>
>
> On machines with slower NUMA, the hashmap lookup for
> TypeInfo::getPrimitiveCategory is the slowest part of the vectorized
> serialization loops. The static object references run hot with the NUMA
> access speeds penalizing half the threads.
> This lookup is done for every column, for every row - though vectorization
> enforces that this type cannot change at all.
> !vec-serialize-hashmap.png!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264901#comment-16264901 ] Gopal V commented on HIVE-17994: The TypeInfo is a singleton object (from the the factory object), so it looks like I've got class-loader sharing race conditions with this (i.e thread 1 class loader showing up for thread 2). Is there some way to restrict this patch to just vectorization code on the task side (like holding the PrimitiveType[] in the initialize in the operator?) and not accidentally kick-in because the HS2 planner hasn't got locks (& separate class loaders for each query). > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch, > HIVE-17994.03.patch, HIVE-17994.04.patch, HIVE-17994.05.patch, > vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264854#comment-16264854 ] Matt McCline commented on HIVE-17994: - Not clear any of the test failures are related. TestTriggersWorkloadManager timed out but works on my laptop. TestCliDriver windowing_range_multiorder.q succeeds on my laptop. > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch, > HIVE-17994.03.patch, HIVE-17994.04.patch, HIVE-17994.05.patch, > vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[
https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264842#comment-16264842
]
Hive QA commented on HIVE-17994:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899107/HIVE-17994.05.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 11410 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_range_multiorder]
(batchId=7)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query64]
(batchId=244)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers1
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
(batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7991/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7991/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7991/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899107 - PreCommit-HIVE-Build
> Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
> --
>
> Key: HIVE-17994
> URL: https://issues.apache.org/jira/browse/HIVE-17994
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Minor
> Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch,
> HIVE-17994.03.patch, HIVE-17994.04.patch, HIVE-17994.05.patch,
> vec-serialize-hashmap.png
>
>
> On machines with slower NUMA, the hashmap lookup for
> TypeInfo::getPrimitiveCategory is the slowest part of the vectorized
> serialization loops. The static object references run hot with the NUMA
> access speeds penalizing half the threads.
> This lookup is done for every column, for every row - though vectorization
> enforces that this type cannot change at all.
> !vec-serialize-hashmap.png!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[
https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264815#comment-16264815
]
Hive QA commented on HIVE-17994:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 8m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / d9924ab |
| Default Java | 1.8.0_111 |
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7991/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
> --
>
> Key: HIVE-17994
> URL: https://issues.apache.org/jira/browse/HIVE-17994
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Minor
> Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch,
> HIVE-17994.03.patch, HIVE-17994.04.patch, HIVE-17994.05.patch,
> vec-serialize-hashmap.png
>
>
> On machines with slower NUMA, the hashmap lookup for
> TypeInfo::getPrimitiveCategory is the slowest part of the vectorized
> serialization loops. The static object references run hot with the NUMA
> access speeds penalizing half the threads.
> This lookup is done for every column, for every row - though vectorization
> enforces that this type cannot change at all.
> !vec-serialize-hashmap.png!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[
https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264181#comment-16264181
]
Hive QA commented on HIVE-17994:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899027/HIVE-17994.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 11410 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_range_multiorder]
(batchId=7)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression1
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression10
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression2
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression3
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression4
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression5
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression7
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression9
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testMultipleConditions_noTranslation
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testMultipleConditions_withTranslation
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testSimpleCondition_noTranslation
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testSimpleCondition_withDateType
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testSimpleCondition_withTranslation
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testSimpleCondition_withVariedCaseMappings
(batchId=205)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7986/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7986/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7986/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899027 - PreCommit-HIVE-Build
> Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
> --
>
> Key: HIVE-17994
> URL: https://issues.apache.org/jira/browse/HIVE-17994
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Minor
> Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch,
> HIVE-17994.03.patch, vec-serialize-hashmap.png
>
>
> On machines with slower NUMA, the hashmap lookup for
> TypeInfo::getPrimitiveCategory is the slowest part of the vectorized
> serialization loops. The static object references run hot with the NUMA
> access speeds penalizing half the threads.
> This lookup is done for every column, for every row - though vectorization
> enforces that this type cannot change at all.
> !vec-serialize-hashmap.png!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[
https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264120#comment-16264120
]
Hive QA commented on HIVE-17994:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} serde: The patch generated 1 new + 4 unchanged - 1
fixed = 5 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / d9924ab |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7986/yetus/diff-checkstyle-serde.txt
|
| modules | C: serde ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7986/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
> --
>
> Key: HIVE-17994
> URL: https://issues.apache.org/jira/browse/HIVE-17994
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Minor
> Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch,
> HIVE-17994.03.patch, vec-serialize-hashmap.png
>
>
> On machines with slower NUMA, the hashmap lookup for
> TypeInfo::getPrimitiveCategory is the slowest part of the vectorized
> serialization loops. The static object references run hot with the NUMA
> access speeds penalizing half the threads.
> This lookup is done for every column, for every row - though vectorization
> enforces that this type cannot change at all.
> !vec-serialize-hashmap.png!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264026#comment-16264026 ] Matt McCline commented on HIVE-17994: - Ok, with patch #3 -- trying out transients and no new non-transient members. > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch, > vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[
https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264023#comment-16264023
]
Hive QA commented on HIVE-17994:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899009/HIVE-17994.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 43 failed/errored test(s), 11410 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestBigintSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestBooleanSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestCharSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestDateSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestDecimalSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestDoubleSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestFloatSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestTimestampSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestVarcharSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression1
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression10
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression2
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression3
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression4
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression5
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression7
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression9
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testMultiPartColsInData
(batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers1
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerTotalTasks
(batchId=231)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testMultipleConditions_noTranslation
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testMultipleConditions_withTranslation
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testSimpleCondition_noTranslation
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testSimpleCondition_withDateType
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testSimpleCondition_withTranslation
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testSimpleCondition_withVariedCaseMappings
(batchId=205)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7984/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7984/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7984/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[
https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263954#comment-16263954
]
Hive QA commented on HIVE-17994:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} serde: The patch generated 1 new + 4 unchanged - 1
fixed = 5 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / c5c2986 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7984/yetus/diff-checkstyle-serde.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7984/yetus/patch-asflicense-problems.txt
|
| modules | C: serde ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7984/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
> --
>
> Key: HIVE-17994
> URL: https://issues.apache.org/jira/browse/HIVE-17994
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Minor
> Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch,
> vec-serialize-hashmap.png
>
>
> On machines with slower NUMA, the hashmap lookup for
> TypeInfo::getPrimitiveCategory is the slowest part of the vectorized
> serialization loops. The static object references run hot with the NUMA
> access speeds penalizing half the threads.
> This lookup is done for every column, for every row - though vectorization
> enforces that this type cannot change at all.
> !vec-serialize-hashmap.png!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[
https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263890#comment-16263890
]
Hive QA commented on HIVE-17994:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898986/HIVE-17994.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 32 failed/errored test(s), 11410 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[date_timestamp_prec]
(batchId=87)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testQueueing
(batchId=282)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestBigintSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestBooleanSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestCharSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestDateSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestDecimalSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestDoubleSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestFloatSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestTimestampSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestVarcharSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression1
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression10
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression2
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression3
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression4
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression5
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression7
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression9
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testMultipleConditions_noTranslation
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testMultipleConditions_withTranslation
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testSimpleCondition_noTranslation
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testSimpleCondition_withDateType
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testSimpleCondition_withTranslation
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testSimpleCondition_withVariedCaseMappings
(batchId=205)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7982/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7982/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7982/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 32 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12898986 - PreCommit-HIVE-Build
> Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
> --
>
> Key: HIVE-17994
> URL: https://issues.apache.org/jira/browse/HIVE-17994
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Minor
> Attachments: HIVE-17994.01.patch, vec-serialize-hashmap.png
>
>
> On machines with slower NUMA, the hashmap lookup for
> TypeInfo::getPrimitiveCategory is the slowest part of the vectorized
> serialization loops. The static object references run hot with the NUMA
> access speeds penalizing half the threads.
> This lookup is done for every column, for every row
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[
https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263860#comment-16263860
]
Hive QA commented on HIVE-17994:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} serde: The patch generated 2 new + 4 unchanged - 1
fixed = 6 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / c5c2986 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7982/yetus/diff-checkstyle-serde.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7982/yetus/patch-asflicense-problems.txt
|
| modules | C: serde ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7982/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
> --
>
> Key: HIVE-17994
> URL: https://issues.apache.org/jira/browse/HIVE-17994
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Minor
> Attachments: HIVE-17994.01.patch, vec-serialize-hashmap.png
>
>
> On machines with slower NUMA, the hashmap lookup for
> TypeInfo::getPrimitiveCategory is the slowest part of the vectorized
> serialization loops. The static object references run hot with the NUMA
> access speeds penalizing half the threads.
> This lookup is done for every column, for every row - though vectorization
> enforces that this type cannot change at all.
> !vec-serialize-hashmap.png!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
