[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[ https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709109#comment-16709109 ] ASF GitHub Bot commented on HIVE-20932: --- Github user b-slim closed the pull request at: https://github.com/apache/hive/pull/493 > Vectorize Druid Storage Handler Reader > -- > > Key: HIVE-20932 > URL: https://issues.apache.org/jira/browse/HIVE-20932 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch, > HIVE-20932.5.patch, HIVE-20932.6.patch, HIVE-20932.7.patch, > HIVE-20932.8.patch, HIVE-20932.8.patch, HIVE-20932.8.patch, HIVE-20932.patch > > > This patch aims at adding support for vectorize read of data from Druid to > Hive. > [~t3rmin4t0r] suggested that this will improve the performance of the top > level operators that supports vectorization. > As a first cut am just adding a wrapper around the existing Record Reader to > read up to 1024 row at a time. > Future work will be to avoid going via old reader and convert straight the > Json (smile format) to Vector primitive types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[ https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699537#comment-16699537 ] Gopal V commented on HIVE-20932: [~bslim]: LGTM - +1 minor nit: there's a new array list allocation for each loop, which seems somewhat of a GC thrash for no good reason. Making a DruidSerdeRow class extending ArrayList would fix that & make it less functional, but more allocation friendly. > Vectorize Druid Storage Handler Reader > -- > > Key: HIVE-20932 > URL: https://issues.apache.org/jira/browse/HIVE-20932 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch, > HIVE-20932.5.patch, HIVE-20932.6.patch, HIVE-20932.7.patch, > HIVE-20932.8.patch, HIVE-20932.8.patch, HIVE-20932.8.patch, HIVE-20932.patch > > > This patch aims at adding support for vectorize read of data from Druid to > Hive. > [~t3rmin4t0r] suggested that this will improve the performance of the top > level operators that supports vectorization. > As a first cut am just adding a wrapper around the existing Record Reader to > read up to 1024 row at a time. > Future work will be to avoid going via old reader and convert straight the > Json (smile format) to Vector primitive types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[ https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695545#comment-16695545 ] slim bouguerra commented on HIVE-20932: --- [~nishantbangarwa] any more comments? > Vectorize Druid Storage Handler Reader > -- > > Key: HIVE-20932 > URL: https://issues.apache.org/jira/browse/HIVE-20932 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch, > HIVE-20932.5.patch, HIVE-20932.6.patch, HIVE-20932.7.patch, > HIVE-20932.8.patch, HIVE-20932.8.patch, HIVE-20932.8.patch, HIVE-20932.patch > > > This patch aims at adding support for vectorize read of data from Druid to > Hive. > [~t3rmin4t0r] suggested that this will improve the performance of the top > level operators that supports vectorization. > As a first cut am just adding a wrapper around the existing Record Reader to > read up to 1024 row at a time. > Future work will be to avoid going via old reader and convert straight the > Json (smile format) to Vector primitive types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695543#comment-16695543
]
Hive QA commented on HIVE-20932:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949138/HIVE-20932.8.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15538 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15038/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15038/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15038/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949138 - PreCommit-HIVE-Build
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch,
> HIVE-20932.5.patch, HIVE-20932.6.patch, HIVE-20932.7.patch,
> HIVE-20932.8.patch, HIVE-20932.8.patch, HIVE-20932.8.patch, HIVE-20932.patch
>
>
> This patch aims at adding support for vectorize read of data from Druid to
> Hive.
> [~t3rmin4t0r] suggested that this will improve the performance of the top
> level operators that supports vectorization.
> As a first cut am just adding a wrapper around the existing Record Reader to
> read up to 1024 row at a time.
> Future work will be to avoid going via old reader and convert straight the
> Json (smile format) to Vector primitive types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695527#comment-16695527
]
Hive QA commented on HIVE-20932:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} druid-handler in master has 4 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15038/dev-support/hive-personality.sh
|
| git revision | master / ddf3b6c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql druid-handler U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15038/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch,
> HIVE-20932.5.patch, HIVE-20932.6.patch, HIVE-20932.7.patch,
> HIVE-20932.8.patch, HIVE-20932.8.patch, HIVE-20932.8.patch, HIVE-20932.patch
>
>
> This patch aims at adding support for vectorize read of data from Druid to
> Hive.
> [~t3rmin4t0r] suggested that this will improve the performance of the top
> level operators that supports vectorization.
> As a first cut am just adding a wrapper around the existing Record Reader to
> read up to 1024 row at a time.
> Future work will be to avoid going via old reader and convert straight the
> Json (smile format) to Vector primitive types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695508#comment-16695508
]
Hive QA commented on HIVE-20932:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949127/HIVE-20932.8.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 15538 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=232)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15037/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15037/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15037/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949127 - PreCommit-HIVE-Build
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch,
> HIVE-20932.5.patch, HIVE-20932.6.patch, HIVE-20932.7.patch,
> HIVE-20932.8.patch, HIVE-20932.8.patch, HIVE-20932.patch
>
>
> This patch aims at adding support for vectorize read of data from Druid to
> Hive.
> [~t3rmin4t0r] suggested that this will improve the performance of the top
> level operators that supports vectorization.
> As a first cut am just adding a wrapper around the existing Record Reader to
> read up to 1024 row at a time.
> Future work will be to avoid going via old reader and convert straight the
> Json (smile format) to Vector primitive types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695481#comment-16695481
]
Hive QA commented on HIVE-20932:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} druid-handler in master has 4 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15037/dev-support/hive-personality.sh
|
| git revision | master / ddf3b6c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql druid-handler U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15037/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch,
> HIVE-20932.5.patch, HIVE-20932.6.patch, HIVE-20932.7.patch,
> HIVE-20932.8.patch, HIVE-20932.8.patch, HIVE-20932.patch
>
>
> This patch aims at adding support for vectorize read of data from Druid to
> Hive.
> [~t3rmin4t0r] suggested that this will improve the performance of the top
> level operators that supports vectorization.
> As a first cut am just adding a wrapper around the existing Record Reader to
> read up to 1024 row at a time.
> Future work will be to avoid going via old reader and convert straight the
> Json (smile format) to Vector primitive types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695347#comment-16695347
]
Hive QA commented on HIVE-20932:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949117/HIVE-20932.8.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15546 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=197)
[druidmini_masking.q,druidmini_joins.q,druid_timestamptz.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15034/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15034/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15034/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949117 - PreCommit-HIVE-Build
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch,
> HIVE-20932.5.patch, HIVE-20932.6.patch, HIVE-20932.7.patch,
> HIVE-20932.8.patch, HIVE-20932.patch
>
>
> This patch aims at adding support for vectorize read of data from Druid to
> Hive.
> [~t3rmin4t0r] suggested that this will improve the performance of the top
> level operators that supports vectorization.
> As a first cut am just adding a wrapper around the existing Record Reader to
> read up to 1024 row at a time.
> Future work will be to avoid going via old reader and convert straight the
> Json (smile format) to Vector primitive types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695326#comment-16695326
]
Hive QA commented on HIVE-20932:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
50s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} druid-handler in master has 4 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15034/dev-support/hive-personality.sh
|
| git revision | master / f5b14fc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql druid-handler U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15034/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch,
> HIVE-20932.5.patch, HIVE-20932.6.patch, HIVE-20932.7.patch,
> HIVE-20932.8.patch, HIVE-20932.patch
>
>
> This patch aims at adding support for vectorize read of data from Druid to
> Hive.
> [~t3rmin4t0r] suggested that this will improve the performance of the top
> level operators that supports vectorization.
> As a first cut am just adding a wrapper around the existing Record Reader to
> read up to 1024 row at a time.
> Future work will be to avoid going via old reader and convert straight the
> Json (smile format) to Vector primitive types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695252#comment-16695252
]
Hive QA commented on HIVE-20932:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949099/HIVE-20932.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15536 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=195)
[druidmini_test_ts.q,druidmini_expressions.q,druid_timestamptz2.q,druidmini_test_alter.q,druidkafkamini_csv.q]
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=196)
[druidmini_dynamic_partition.q,druidmini_test1.q,druidkafkamini_basic.q,druidmini_test_insert.q,druidkafkamini_delimited.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15032/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15032/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15032/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949099 - PreCommit-HIVE-Build
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch,
> HIVE-20932.5.patch, HIVE-20932.6.patch, HIVE-20932.7.patch, HIVE-20932.patch
>
>
> This patch aims at adding support for vectorize read of data from Druid to
> Hive.
> [~t3rmin4t0r] suggested that this will improve the performance of the top
> level operators that supports vectorization.
> As a first cut am just adding a wrapper around the existing Record Reader to
> read up to 1024 row at a time.
> Future work will be to avoid going via old reader and convert straight the
> Json (smile format) to Vector primitive types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695220#comment-16695220
]
Hive QA commented on HIVE-20932:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} druid-handler in master has 4 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} druid-handler: The patch generated 1 new + 0 unchanged
- 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15032/dev-support/hive-personality.sh
|
| git revision | master / f5b14fc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15032/yetus/diff-checkstyle-druid-handler.txt
|
| modules | C: ql druid-handler U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15032/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch,
> HIVE-20932.5.patch, HIVE-20932.6.patch, HIVE-20932.7.patch, HIVE-20932.patch
>
>
> This patch aims at adding support for vectorize read of data from Druid to
> Hive.
> [~t3rmin4t0r] suggested that this will improve the performance of the top
> level operators that supports vectorization.
> As a first cut am just adding a wrapper around the existing Record Reader to
> read up to 1024 row at a time.
> Future work will be to avoid going via old reader and convert straight the
> Json (smile format) to Vector primitive types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695156#comment-16695156
]
Hive QA commented on HIVE-20932:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949082/HIVE-20932.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15546 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestTxnCommands.testMergeOnTezEdges (batchId=324)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15031/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15031/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15031/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949082 - PreCommit-HIVE-Build
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch,
> HIVE-20932.5.patch, HIVE-20932.6.patch, HIVE-20932.patch
>
>
> This patch aims at adding support for vectorize read of data from Druid to
> Hive.
> [~t3rmin4t0r] suggested that this will improve the performance of the top
> level operators that supports vectorization.
> As a first cut am just adding a wrapper around the existing Record Reader to
> read up to 1024 row at a time.
> Future work will be to avoid going via old reader and convert straight the
> Json (smile format) to Vector primitive types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695101#comment-16695101
]
Hive QA commented on HIVE-20932:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} druid-handler in master has 4 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
11s{color} | {color:red} druid-handler generated 1 new + 0 unchanged - 0 fixed
= 1 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15031/dev-support/hive-personality.sh
|
| git revision | master / 9389a5a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15031/yetus/diff-javadoc-javadoc-druid-handler.txt
|
| modules | C: ql druid-handler U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15031/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch,
> HIVE-20932.5.patch, HIVE-20932.6.patch, HIVE-20932.patch
>
>
> This patch aims at adding support for vectorize read of data from Druid to
> Hive.
> [~t3rmin4t0r] suggested that this will improve the performance of the top
> level operators that supports vectorization.
> As a first cut am just adding a wrapper around the existing Record Reader to
> read up to 1024 row at a time.
> Future work will be to avoid going via old reader and convert straight the
> Json (smile format) to Vector primitive types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694922#comment-16694922
]
Hive QA commented on HIVE-20932:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949064/HIVE-20932.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15546 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=197)
[druidmini_masking.q,druidmini_joins.q,druid_timestamptz.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15029/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15029/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15029/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949064 - PreCommit-HIVE-Build
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch,
> HIVE-20932.5.patch, HIVE-20932.patch
>
>
> This patch aims at adding support for vectorize read of data from Druid to
> Hive.
> [~t3rmin4t0r] suggested that this will improve the performance of the top
> level operators that supports vectorization.
> As a first cut am just adding a wrapper around the existing Record Reader to
> read up to 1024 row at a time.
> Future work will be to avoid going via old reader and convert straight the
> Json (smile format) to Vector primitive types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[ https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694920#comment-16694920 ] ASF GitHub Bot commented on HIVE-20932: --- GitHub user b-slim opened a pull request: https://github.com/apache/hive/pull/493 HIVE-20932 Adding Vectorize code to Druid storage handler (Slim B) You can merge this pull request into a Git repository by running: $ git pull https://github.com/b-slim/hive HIVE-20932 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/493.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #493 commit cbf555f2b54a3b5eaca0c98c7067b420ad488c08 Author: Slim Bouguerra Date: 2018-11-20T22:49:45Z HIVE-20932 Adding Vectorize code to Druid storage handler (Slim B) Change-Id: I1a95bbe0f1d0e3a5452111cd1d2262d4253dbdcb > Vectorize Druid Storage Handler Reader > -- > > Key: HIVE-20932 > URL: https://issues.apache.org/jira/browse/HIVE-20932 > Project: Hive > Issue Type: Improvement >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch, > HIVE-20932.5.patch, HIVE-20932.patch > > > This patch aims at adding support for vectorize read of data from Druid to > Hive. > [~t3rmin4t0r] suggested that this will improve the performance of the top > level operators that supports vectorization. > As a first cut am just adding a wrapper around the existing Record Reader to > read up to 1024 row at a time. > Future work will be to avoid going via old reader and convert straight the > Json (smile format) to Vector primitive types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694883#comment-16694883
]
Hive QA commented on HIVE-20932:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} druid-handler in master has 4 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} druid-handler: The patch generated 1 new + 0 unchanged
- 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
11s{color} | {color:red} druid-handler generated 1 new + 0 unchanged - 0 fixed
= 1 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15029/dev-support/hive-personality.sh
|
| git revision | master / 9389a5a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15029/yetus/diff-checkstyle-druid-handler.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15029/yetus/diff-javadoc-javadoc-druid-handler.txt
|
| modules | C: ql druid-handler U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15029/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch,
> HIVE-20932.5.patch, HIVE-20932.patch
>
>
> This patch aims at adding support for vectorize read of data from Druid to
> Hive.
> [~t3rmin4t0r] suggested that this will improve the performance of the top
> level operators that supports vectorization.
> As a first cut am just adding a wrapper around the existing Record Reader to
> read up to 1024 row at a time.
> Future work will be to avoid going via old reader and convert straight the
> Json (smile format) to Vector primitive types.
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[ https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694846#comment-16694846 ] slim bouguerra commented on HIVE-20932: --- [~nishantbangarwa] This should enable Vectorize pipeline thus less object creation and SIMD on arithmetic therefore faster of course in theory :D Working on real benchmark. > Vectorize Druid Storage Handler Reader > -- > > Key: HIVE-20932 > URL: https://issues.apache.org/jira/browse/HIVE-20932 > Project: Hive > Issue Type: Improvement >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch, > HIVE-20932.5.patch, HIVE-20932.patch > > > This patch aims at adding support for vectorize read of data from Druid to > Hive. > [~t3rmin4t0r] suggested that this will improve the performance of the top > level operators that supports vectorization. > As a first cut am just adding a wrapper around the existing Record Reader to > read up to 1024 row at a time. > Future work will be to avoid going via old reader and convert straight the > Json (smile format) to Vector primitive types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[ https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692797#comment-16692797 ] Nishant Bangarwa commented on HIVE-20932: - [~bslim] Can you also share some benchmark results regarding the performance gains ? > Vectorize Druid Storage Handler Reader > -- > > Key: HIVE-20932 > URL: https://issues.apache.org/jira/browse/HIVE-20932 > Project: Hive > Issue Type: Improvement >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch, HIVE-20932.patch > > > This patch aims at adding support for vectorize read of data from Druid to > Hive. > [~t3rmin4t0r] suggested that this will improve the performance of the top > level operators that supports vectorization. > As a first cut am just adding a wrapper around the existing Record Reader to > read up to 1024 row at a time. > Future work will be to avoid going via old reader and convert straight the > Json (smile format) to Vector primitive types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692713#comment-16692713
]
Hive QA commented on HIVE-20932:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12948791/HIVE-20932.4.patch
{color:green}SUCCESS:{color} +1 due to 15 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15544 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15002/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15002/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15002/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12948791 - PreCommit-HIVE-Build
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch, HIVE-20932.patch
>
>
> This patch aims at adding support for vectorize read of data from Druid to
> Hive.
> [~t3rmin4t0r] suggested that this will improve the performance of the top
> level operators that supports vectorization.
> As a first cut am just adding a wrapper around the existing Record Reader to
> read up to 1024 row at a time.
> Future work will be to avoid going via old reader and convert straight the
> Json (smile format) to Vector primitive types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692712#comment-16692712
]
Hive QA commented on HIVE-20932:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
37s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} druid-handler in master has 4 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} storage-api: The patch generated 1 new + 17 unchanged
- 2 fixed = 18 total (was 19) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} serde: The patch generated 0 new + 50 unchanged - 2
fixed = 50 total (was 52) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} druid-handler: The patch generated 0 new + 0
unchanged - 193 fixed = 0 total (was 193) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} root: The patch generated 1 new + 67 unchanged - 4
fixed = 68 total (was 71) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
10s{color} | {color:red} druid-handler generated 4 new + 0 unchanged - 0 fixed
= 4 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 6m
12s{color} | {color:red} root generated 4 new + 377 unchanged - 0 fixed = 381
total (was 377) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense xml javac javadoc findbugs checkstyle
compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15002/dev-support/hive-personality.sh
|
| git revision | master / 1fe471d |
| Default Java | 1.8.0_111
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[ https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692549#comment-16692549 ] slim bouguerra commented on HIVE-20932: --- [~t3rmin4t0r] and [~teddy.choi] the most important class to look at druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidVectorizedWrapper.java org.apache.hadoop.hive.druid.serde.DruidSerDe#deserializeAsPrimitive org.apache.hadoop.hive.druid.io.DruidQueryBasedInputFormat#getRecordReader > Vectorize Druid Storage Handler Reader > -- > > Key: HIVE-20932 > URL: https://issues.apache.org/jira/browse/HIVE-20932 > Project: Hive > Issue Type: Improvement >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch, HIVE-20932.patch > > > This patch aims at adding support for vectorize read of data from Druid to > Hive. > [~t3rmin4t0r] suggested that this will improve the performance of the top > level operators that supports vectorization. > As a first cut am just adding a wrapper around the existing Record Reader to > read up to 1024 row at a time. > Future work will be to avoid going via old reader and convert straight the > Json (smile format) to Vector primitive types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690358#comment-16690358
]
Hive QA commented on HIVE-20932:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12948574/HIVE-20932.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14972/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14972/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14972/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12948574/HIVE-20932.3.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12948574 - PreCommit-HIVE-Build
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20932.3.patch, HIVE-20932.patch
>
>
> This patch aims at adding support for vectorize read of data from Druid to
> Hive.
> [~t3rmin4t0r] suggested that this will improve the performance of the top
> level operators that supports vectorization.
> As a first cut am just adding a wrapper around the existing Record Reader to
> read up to 1024 row at a time.
> Future work will be to avoid going via old reader and convert straight the
> Json (smile format) to Vector primitive types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690357#comment-16690357
]
Hive QA commented on HIVE-20932:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12948574/HIVE-20932.3.patch
{color:green}SUCCESS:{color} +1 due to 15 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15544 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14971/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14971/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14971/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12948574 - PreCommit-HIVE-Build
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20932.3.patch, HIVE-20932.patch
>
>
> This patch aims at adding support for vectorize read of data from Druid to
> Hive.
> [~t3rmin4t0r] suggested that this will improve the performance of the top
> level operators that supports vectorization.
> As a first cut am just adding a wrapper around the existing Record Reader to
> read up to 1024 row at a time.
> Future work will be to avoid going via old reader and convert straight the
> Json (smile format) to Vector primitive types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690354#comment-16690354
]
Hive QA commented on HIVE-20932:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} druid-handler in master has 4 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 2316 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} The patch . passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} druid-handler: The patch generated 0 new + 0
unchanged - 193 fixed = 0 total (was 193) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
34s{color} | {color:red} druid-handler generated 1 new + 3 unchanged - 1 fixed
= 4 total (was 4) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:druid-handler |
| | Boxing/unboxing to parse a primitive
org.apache.hadoop.hive.druid.serde.DruidSerDe.doDeserialize(Writable) At
DruidSerDe.java:org.apache.hadoop.hive.druid.serde.DruidSerDe.doDeserialize(Writable)
At DruidSerDe.java:[line 452] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense xml javac javadoc findbugs checkstyle
compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14971/dev-support/hive-personality.sh
|
| git revision | master / fd5f34f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14971/yetus/new-findbugs-druid-handler.html
|
| modules | C: . druid-handler ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14971/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690101#comment-16690101
]
Hive QA commented on HIVE-20932:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12948559/HIVE-20932.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15544 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=334)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14967/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14967/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14967/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12948559 - PreCommit-HIVE-Build
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20932.patch
>
>
> This patch aims at adding support for vectorize read of data from Druid to
> Hive.
> [~t3rmin4t0r] suggested that this will improve the performance of the top
> level operators that supports vectorization.
> As a first cut am just adding a wrapper around the existing Record Reader to
> read up to 1024 row at a time.
> Future work will be to avoid going via old reader and convert straight the
> Json (smile format) to Vector primitive types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20932) Vectorize Druid Storage Handler Reader
[
https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690056#comment-16690056
]
Hive QA commented on HIVE-20932:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} druid-handler in master has 4 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
47s{color} | {color:blue} ql in master has 2316 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} druid-handler: The patch generated 3 new + 77
unchanged - 2 fixed = 80 total (was 79) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red} druid-handler generated 1 new + 3 unchanged - 1 fixed
= 4 total (was 4) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:druid-handler |
| | Boxing/unboxing to parse a primitive
org.apache.hadoop.hive.druid.serde.DruidSerDe.doDeserialize(Writable) At
DruidSerDe.java:org.apache.hadoop.hive.druid.serde.DruidSerDe.doDeserialize(Writable)
At DruidSerDe.java:[line 452] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14967/dev-support/hive-personality.sh
|
| git revision | master / fd5f34f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14967/yetus/diff-checkstyle-druid-handler.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14967/yetus/new-findbugs-druid-handler.html
|
| modules | C: druid-handler ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14967/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorize Druid Storage Handler Reader
> --
>
> Key: HIVE-20932
> URL: https://issues.apache.org/jira/browse/HIVE-20932
> Project: Hive
> Issue Type: Improvement
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20932.patch
>
>
> This patch aims at adding support for vectorize read of data from Druid to
> Hive.
> [~t3rmin4t0r] suggested that this will improve the performance of the top
> level operators that supports vectorization.
>
