[jira] [Commented] (HIVE-22079) Post order walker for iterating over expression tree
[
https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926345#comment-16926345
]
Hive QA commented on HIVE-22079:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
28s{color} | {color:blue} ql in master has 2246 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 9 new + 16 unchanged - 0 fixed
= 25 total (was 16) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18517/dev-support/hive-personality.sh
|
| git revision | master / 1dfa2d8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18517/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18517/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Post order walker for iterating over expression tree
>
>
> Key: HIVE-22079
> URL: https://issues.apache.org/jira/browse/HIVE-22079
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer, Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch,
> HIVE-22079.3.patch
>
>
> Current {{DefaultGraphWalker}} is used to iterate over an expression tree.
> This walker uses hash map to keep track of visited/processed nodes. If an
> expression tree is large this adds significant overhead due to map lookup.
> For an expression trees we can instead use post order traversal and avoid
> using map.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[
https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926337#comment-16926337
]
Hive QA commented on HIVE-20113:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12979886/HIVE-20113.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 16718 tests
executed
*Failed tests:*
{noformat}
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=233)
TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=233)
org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthHttp.testAuthorization1
(batchId=298)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18516/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18516/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18516/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12979886 - PreCommit-HIVE-Build
> Shuffle avoidance: Disable 1-1 edges for sorted shuffle
>
>
> Key: HIVE-20113
> URL: https://issues.apache.org/jira/browse/HIVE-20113
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Labels: Branch3Candidate
> Attachments: HIVE-20113.1.patch, HIVE-20113.2.patch,
> HIVE-20113.3.patch, HIVE-20113.4.patch, HIVE-20113.4.patch,
> HIVE-20113.5.patch, HIVE-20113.6.patch, HIVE-20113.7.patch
>
>
> The sorted shuffle avoidance can have some issues when the shuffle data gets
> broken up into multiple chunks on disk.
> The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to
> have a final merge at all, it should open a single compressed file and write
> a single index entry.
> Until the shuffle issue is resolved & a lot more testing, it is prudent to
> disable the optimization for sorted shuffle edges and stop rewriting the
> RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD).
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[ https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926333#comment-16926333 ] Vineet Garg commented on HIVE-20113: [~jcamachorodriguez] Can you please take a look? > Shuffle avoidance: Disable 1-1 edges for sorted shuffle > > > Key: HIVE-20113 > URL: https://issues.apache.org/jira/browse/HIVE-20113 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: Branch3Candidate > Attachments: HIVE-20113.1.patch, HIVE-20113.2.patch, > HIVE-20113.3.patch, HIVE-20113.4.patch, HIVE-20113.4.patch, > HIVE-20113.5.patch, HIVE-20113.6.patch, HIVE-20113.7.patch > > > The sorted shuffle avoidance can have some issues when the shuffle data gets > broken up into multiple chunks on disk. > The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to > have a final merge at all, it should open a single compressed file and write > a single index entry. > Until the shuffle issue is resolved & a lot more testing, it is prudent to > disable the optimization for sorted shuffle edges and stop rewriting the > RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22164) Vectorized Limit operator returns wrong number of results with offset
[ https://issues.apache.org/jira/browse/HIVE-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22164: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~rameshkumar] > Vectorized Limit operator returns wrong number of results with offset > - > > Key: HIVE-22164 > URL: https://issues.apache.org/jira/browse/HIVE-22164 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22164.1.patch > > > Vectorized Limit operator returns wrong number of results with offset -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22164) Vectorized Limit operator returns wrong number of results with offset
[ https://issues.apache.org/jira/browse/HIVE-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926329#comment-16926329 ] Vineet Garg commented on HIVE-22164: LGTM +1 > Vectorized Limit operator returns wrong number of results with offset > - > > Key: HIVE-22164 > URL: https://issues.apache.org/jira/browse/HIVE-22164 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22164.1.patch > > > Vectorized Limit operator returns wrong number of results with offset -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22181) Same query but different result occasionally
[
https://issues.apache.org/jira/browse/HIVE-22181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qingfa zhou updated HIVE-22181:
---
Attachment: image-2019-09-10-12-41-16-717.png
> Same query but different result occasionally
>
>
> Key: HIVE-22181
> URL: https://issues.apache.org/jira/browse/HIVE-22181
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.3.0
> Environment: hadoop：2.7.3.2.6.1.0-129
> hive：2.3.0
> tez：0.8.4
> java：1.8.0_212
>Reporter: qingfa zhou
>Priority: Major
> Attachments: WX20190910-123604.png, WX20190910-123621.png,
> WX20190910-123641.png, image-2019-09-10-12-40-34-958.png,
> image-2019-09-10-12-40-53-338.png, image-2019-09-10-12-41-16-717.png,
> image-2019-09-10-12-41-35-056.png
>
>
> h3. 1.Phenomenon
> Running the same SQL(hive on tez) several times will result in
> inconsistent results . The following results are generated respectively
> 1)
> OK
> 3951864 808
> 2)
> OK
> 1822979 353
> But the first result is true.
> h3. 2.SQL
> select
> count(1),count(distinct store_code)
> from
> (
> select
> store_code,
> all_pay_id,
> payable_price,
> row_number()over(partition by store_code order by payable_price desc) as rw,
> count(1)over(partition by store_code) as store_user
> from
> (
> select
> store_code,
> all_pay_id,
> sum(payable_price) as payable_price
> from data_promotion.mdw_user_promotion_shopping_behaivor_four_week_detail a
> where dt='20190904'
> group by store_code,all_pay_id
> ) a
> order by rw
> ) a
> where rw/store_user<=0.8
> ;
>
> h3. 3.Troubleshooting
> I lookup task result from tez web ui，give the result as follows：
> the first:
> !image-2019-09-10-12-40-34-958.png!
> the second:
> !image-2019-09-10-12-40-53-338.png!
> so , The second RECORDS_OUT_INTERMEDIATE_Reducer_4's result is much
> lower than the first.
> This is detailed info of the second result from reducer_3 to reducer_4
> !image-2019-09-10-12-41-16-717.png!
> This is execution plan of the sql:
> OK
> Vertex dependency in root stage
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
> Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
> Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Reducer 6
> File Output Operator [FS_24]
> Group By Operator [GBY_22] (rows=1 width=128)
> Output:["_col0","_col1"],aggregations:["count(VALUE._col0)","count(DISTINCT
> KEY._col0:0._col0)"]
> <-Reducer 5 [SIMPLE_EDGE]
> SHUFFLE [RS_21]
> Group By Operator [GBY_20] (rows=5619870 width=2328)
> Output:["_col0","_col1","_col2"],aggregations:["count(1)","count(DISTINCT
> _col0)"],keys:_col0
> Select Operator [SEL_17] (rows=5619870 width=2328)
> Output:["_col0"]
> <-Reducer 4 [SIMPLE_EDGE]
> SHUFFLE [RS_16]
> Select Operator [SEL_14] (rows=5619870 width=2328)
> Output:["_col0","_col3"]
> Filter Operator [FIL_25] (rows=5619870 width=2328)
> predicate:((_col0 / count_window_1) <= 0.8)
> PTF Operator [PTF_13] (rows=16859610 width=2328)
> Function definitions:[{},
> {"name:":"windowingtablefunction","order by:":"_col1 ASC NULLS
> FIRST","partition by:":"_col1"}
> ]
> Select Operator [SEL_12] (rows=16859610 width=2328)
> Output:["_col0","_col1"]
> <-Reducer 3 [SIMPLE_EDGE]
> SHUFFLE [RS_11]
> PartitionCols:_col0
> Select Operator [SEL_10] (rows=16859610 width=2328)
> Output:["_col0","row_number_window_0"]
> PTF Operator [PTF_9] (rows=16859610 width=2328)
> Function definitions:[{},
> {"name:":"windowingtablefunction","order by:":"_col2 DESC NULLS
> LAST","partition by:":"_col0"}
> ]
> Select Operator [SEL_8] (rows=16859610 width=2328)
> Output:["_col0","_col2"]
> <-Reducer 2 [SIMPLE_EDGE]
> SHUFFLE [RS_7]
> PartitionCols:_col0
> Select Operator [SEL_6] (rows=16859610 width=2328)
> Output:["_col0","_col2"]
> Group By Operator [GBY_5] (rows=16859610 width=2328)
>
> Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0,
> KEY._col1
> <-Map 1 [SIMPLE_EDGE]
> SHUFFLE [RS_4]
> PartitionCols:_col0, _col1
> Group By Operator [GBY_3] (rows=33719220 width=2328)
>
> Output:["_col0","_col1","_col2"],aggregations:["sum(payable_price)"],keys:store_code,
> all_pay_id
> Select Operator [SEL_2] (rows=33719220 width=2328)
> Output:["store_code","all_pay_id","payable_price"]
> TableScan [TS_0] (rows=33719220 width=2328)
>
> data_promotion@mdw_user_promotion_shopping_behaivor_four_week_detail,a,Tbl:COMPLETE,Col:NONE,Output:["all_pay_id","store_code","payable_price"]
>
> Reducer 4 show PTF , so I don't know if this result has anything to do
> with PTF? Can you help me.
>
>
--
This message was sent by Atlassian Jira
[jira] [Updated] (HIVE-22181) Same query but different result occasionally
[
https://issues.apache.org/jira/browse/HIVE-22181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qingfa zhou updated HIVE-22181:
---
Description:
h3. 1.Phenomenon
Running the same SQL(hive on tez) several times will result in inconsistent
results . The following results are generated respectively
1)
OK
3951864 808
2)
OK
1822979 353
But the first result is true.
h3. 2.SQL
select
count(1),count(distinct store_code)
from
(
select
store_code,
all_pay_id,
payable_price,
row_number()over(partition by store_code order by payable_price desc) as rw,
count(1)over(partition by store_code) as store_user
from
(
select
store_code,
all_pay_id,
sum(payable_price) as payable_price
from data_promotion.mdw_user_promotion_shopping_behaivor_four_week_detail a
where dt='20190904'
group by store_code,all_pay_id
) a
order by rw
) a
where rw/store_user<=0.8
;
h3. 3.Troubleshooting
I lookup task result from tez web ui，give the result as follows：
the first:
!image-2019-09-10-12-40-34-958.png!
the second:
!image-2019-09-10-12-40-53-338.png!
so , The second RECORDS_OUT_INTERMEDIATE_Reducer_4's result is much lower
than the first.
This is detailed info of the second result from reducer_3 to reducer_4
!image-2019-09-10-12-41-16-717.png!
This is execution plan of the sql:
OK
Vertex dependency in root stage
Reducer 2 <- Map 1 (SIMPLE_EDGE)
Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
Stage-0
Fetch Operator
limit:-1
Stage-1
Reducer 6
File Output Operator [FS_24]
Group By Operator [GBY_22] (rows=1 width=128)
Output:["_col0","_col1"],aggregations:["count(VALUE._col0)","count(DISTINCT
KEY._col0:0._col0)"]
<-Reducer 5 [SIMPLE_EDGE]
SHUFFLE [RS_21]
Group By Operator [GBY_20] (rows=5619870 width=2328)
Output:["_col0","_col1","_col2"],aggregations:["count(1)","count(DISTINCT
_col0)"],keys:_col0
Select Operator [SEL_17] (rows=5619870 width=2328)
Output:["_col0"]
<-Reducer 4 [SIMPLE_EDGE]
SHUFFLE [RS_16]
Select Operator [SEL_14] (rows=5619870 width=2328)
Output:["_col0","_col3"]
Filter Operator [FIL_25] (rows=5619870 width=2328)
predicate:((_col0 / count_window_1) <= 0.8)
PTF Operator [PTF_13] (rows=16859610 width=2328)
Function definitions:[{},
{"name:":"windowingtablefunction","order by:":"_col1 ASC NULLS
FIRST","partition by:":"_col1"}
]
Select Operator [SEL_12] (rows=16859610 width=2328)
Output:["_col0","_col1"]
<-Reducer 3 [SIMPLE_EDGE]
SHUFFLE [RS_11]
PartitionCols:_col0
Select Operator [SEL_10] (rows=16859610 width=2328)
Output:["_col0","row_number_window_0"]
PTF Operator [PTF_9] (rows=16859610 width=2328)
Function definitions:[{},
{"name:":"windowingtablefunction","order by:":"_col2 DESC NULLS
LAST","partition by:":"_col0"}
]
Select Operator [SEL_8] (rows=16859610 width=2328)
Output:["_col0","_col2"]
<-Reducer 2 [SIMPLE_EDGE]
SHUFFLE [RS_7]
PartitionCols:_col0
Select Operator [SEL_6] (rows=16859610 width=2328)
Output:["_col0","_col2"]
Group By Operator [GBY_5] (rows=16859610 width=2328)
Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0,
KEY._col1
<-Map 1 [SIMPLE_EDGE]
SHUFFLE [RS_4]
PartitionCols:_col0, _col1
Group By Operator [GBY_3] (rows=33719220 width=2328)
Output:["_col0","_col1","_col2"],aggregations:["sum(payable_price)"],keys:store_code,
all_pay_id
Select Operator [SEL_2] (rows=33719220 width=2328)
Output:["store_code","all_pay_id","payable_price"]
TableScan [TS_0] (rows=33719220 width=2328)
data_promotion@mdw_user_promotion_shopping_behaivor_four_week_detail,a,Tbl:COMPLETE,Col:NONE,Output:["all_pay_id","store_code","payable_price"]
Reducer 4 show PTF , so I don't know if this result has anything to do with
PTF? Can you help me.
was:
h3. 1.Phenomenon
Running the same SQL(hive on tez) several times will result in inconsistent
results . The following results are generated respectively
1)
OK
3951864 808
2)
OK
1822979 353
But the first result is true.
h3. 2.SQL
select
count(1),count(distinct store_code)
from
(
select
store_code,
all_pay_id,
payable_price,
row_number()over(partition by store_code order by payable_price desc) as rw,
count(1)over(partition by store_code) as store_user
from
(
select
store_code,
all_pay_id,
sum(payable_price) as payable_price
from data_promotion.mdw_user_promotion_shopping_behaivor_four_week_detail a
where dt='20190904'
group by store_code,all_pay_id
) a
order by rw
) a
where rw/store_user<=0.8
;
h3. 3.Troubleshooting
I lookup task result from tez web ui，give the result as follows：
the first:
the second:
!image-2019-09-10-12-18-08-769.png!
so , The second
[jira] [Updated] (HIVE-22181) Same query but different result occasionally
[
https://issues.apache.org/jira/browse/HIVE-22181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qingfa zhou updated HIVE-22181:
---
Attachment: image-2019-09-10-12-41-35-056.png
> Same query but different result occasionally
>
>
> Key: HIVE-22181
> URL: https://issues.apache.org/jira/browse/HIVE-22181
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.3.0
> Environment: hadoop：2.7.3.2.6.1.0-129
> hive：2.3.0
> tez：0.8.4
> java：1.8.0_212
>Reporter: qingfa zhou
>Priority: Major
> Attachments: WX20190910-123604.png, WX20190910-123621.png,
> WX20190910-123641.png, image-2019-09-10-12-40-34-958.png,
> image-2019-09-10-12-40-53-338.png, image-2019-09-10-12-41-16-717.png,
> image-2019-09-10-12-41-35-056.png
>
>
> h3. 1.Phenomenon
> Running the same SQL(hive on tez) several times will result in
> inconsistent results . The following results are generated respectively
> 1)
> OK
> 3951864 808
> 2)
> OK
> 1822979 353
> But the first result is true.
> h3. 2.SQL
> select
> count(1),count(distinct store_code)
> from
> (
> select
> store_code,
> all_pay_id,
> payable_price,
> row_number()over(partition by store_code order by payable_price desc) as rw,
> count(1)over(partition by store_code) as store_user
> from
> (
> select
> store_code,
> all_pay_id,
> sum(payable_price) as payable_price
> from data_promotion.mdw_user_promotion_shopping_behaivor_four_week_detail a
> where dt='20190904'
> group by store_code,all_pay_id
> ) a
> order by rw
> ) a
> where rw/store_user<=0.8
> ;
>
> h3. 3.Troubleshooting
> I lookup task result from tez web ui，give the result as follows：
> the first:
> !image-2019-09-10-12-40-34-958.png!
> the second:
> !image-2019-09-10-12-40-53-338.png!
> so , The second RECORDS_OUT_INTERMEDIATE_Reducer_4's result is much
> lower than the first.
> This is detailed info of the second result from reducer_3 to reducer_4
> !image-2019-09-10-12-41-35-056.png!
> This is execution plan of the sql:
> OK
> Vertex dependency in root stage
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
> Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
> Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Reducer 6
> File Output Operator [FS_24]
> Group By Operator [GBY_22] (rows=1 width=128)
> Output:["_col0","_col1"],aggregations:["count(VALUE._col0)","count(DISTINCT
> KEY._col0:0._col0)"]
> <-Reducer 5 [SIMPLE_EDGE]
> SHUFFLE [RS_21]
> Group By Operator [GBY_20] (rows=5619870 width=2328)
> Output:["_col0","_col1","_col2"],aggregations:["count(1)","count(DISTINCT
> _col0)"],keys:_col0
> Select Operator [SEL_17] (rows=5619870 width=2328)
> Output:["_col0"]
> <-Reducer 4 [SIMPLE_EDGE]
> SHUFFLE [RS_16]
> Select Operator [SEL_14] (rows=5619870 width=2328)
> Output:["_col0","_col3"]
> Filter Operator [FIL_25] (rows=5619870 width=2328)
> predicate:((_col0 / count_window_1) <= 0.8)
> PTF Operator [PTF_13] (rows=16859610 width=2328)
> Function definitions:[{},
> {"name:":"windowingtablefunction","order by:":"_col1 ASC NULLS
> FIRST","partition by:":"_col1"}
> ]
> Select Operator [SEL_12] (rows=16859610 width=2328)
> Output:["_col0","_col1"]
> <-Reducer 3 [SIMPLE_EDGE]
> SHUFFLE [RS_11]
> PartitionCols:_col0
> Select Operator [SEL_10] (rows=16859610 width=2328)
> Output:["_col0","row_number_window_0"]
> PTF Operator [PTF_9] (rows=16859610 width=2328)
> Function definitions:[{},
> {"name:":"windowingtablefunction","order by:":"_col2 DESC NULLS
> LAST","partition by:":"_col0"}
> ]
> Select Operator [SEL_8] (rows=16859610 width=2328)
> Output:["_col0","_col2"]
> <-Reducer 2 [SIMPLE_EDGE]
> SHUFFLE [RS_7]
> PartitionCols:_col0
> Select Operator [SEL_6] (rows=16859610 width=2328)
> Output:["_col0","_col2"]
> Group By Operator [GBY_5] (rows=16859610 width=2328)
>
> Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0,
> KEY._col1
> <-Map 1 [SIMPLE_EDGE]
> SHUFFLE [RS_4]
> PartitionCols:_col0, _col1
> Group By Operator [GBY_3] (rows=33719220 width=2328)
>
> Output:["_col0","_col1","_col2"],aggregations:["sum(payable_price)"],keys:store_code,
> all_pay_id
> Select Operator [SEL_2] (rows=33719220 width=2328)
> Output:["store_code","all_pay_id","payable_price"]
> TableScan [TS_0] (rows=33719220 width=2328)
>
> data_promotion@mdw_user_promotion_shopping_behaivor_four_week_detail,a,Tbl:COMPLETE,Col:NONE,Output:["all_pay_id","store_code","payable_price"]
>
> Reducer 4 show PTF , so I don't know if this result has anything to do
> with PTF? Can you help me.
>
>
--
This message was sent by Atlassian Jira
[jira] [Updated] (HIVE-22181) Same query but different result occasionally
[
https://issues.apache.org/jira/browse/HIVE-22181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qingfa zhou updated HIVE-22181:
---
Description:
h3. 1.Phenomenon
Running the same SQL(hive on tez) several times will result in inconsistent
results . The following results are generated respectively
1)
OK
3951864 808
2)
OK
1822979 353
But the first result is true.
h3. 2.SQL
select
count(1),count(distinct store_code)
from
(
select
store_code,
all_pay_id,
payable_price,
row_number()over(partition by store_code order by payable_price desc) as rw,
count(1)over(partition by store_code) as store_user
from
(
select
store_code,
all_pay_id,
sum(payable_price) as payable_price
from data_promotion.mdw_user_promotion_shopping_behaivor_four_week_detail a
where dt='20190904'
group by store_code,all_pay_id
) a
order by rw
) a
where rw/store_user<=0.8
;
h3. 3.Troubleshooting
I lookup task result from tez web ui，give the result as follows：
the first:
!image-2019-09-10-12-40-34-958.png!
the second:
!image-2019-09-10-12-40-53-338.png!
so , The second RECORDS_OUT_INTERMEDIATE_Reducer_4's result is much lower
than the first.
This is detailed info of the second result from reducer_3 to reducer_4
!image-2019-09-10-12-41-35-056.png!
This is execution plan of the sql:
OK
Vertex dependency in root stage
Reducer 2 <- Map 1 (SIMPLE_EDGE)
Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
Stage-0
Fetch Operator
limit:-1
Stage-1
Reducer 6
File Output Operator [FS_24]
Group By Operator [GBY_22] (rows=1 width=128)
Output:["_col0","_col1"],aggregations:["count(VALUE._col0)","count(DISTINCT
KEY._col0:0._col0)"]
<-Reducer 5 [SIMPLE_EDGE]
SHUFFLE [RS_21]
Group By Operator [GBY_20] (rows=5619870 width=2328)
Output:["_col0","_col1","_col2"],aggregations:["count(1)","count(DISTINCT
_col0)"],keys:_col0
Select Operator [SEL_17] (rows=5619870 width=2328)
Output:["_col0"]
<-Reducer 4 [SIMPLE_EDGE]
SHUFFLE [RS_16]
Select Operator [SEL_14] (rows=5619870 width=2328)
Output:["_col0","_col3"]
Filter Operator [FIL_25] (rows=5619870 width=2328)
predicate:((_col0 / count_window_1) <= 0.8)
PTF Operator [PTF_13] (rows=16859610 width=2328)
Function definitions:[{},
{"name:":"windowingtablefunction","order by:":"_col1 ASC NULLS
FIRST","partition by:":"_col1"}
]
Select Operator [SEL_12] (rows=16859610 width=2328)
Output:["_col0","_col1"]
<-Reducer 3 [SIMPLE_EDGE]
SHUFFLE [RS_11]
PartitionCols:_col0
Select Operator [SEL_10] (rows=16859610 width=2328)
Output:["_col0","row_number_window_0"]
PTF Operator [PTF_9] (rows=16859610 width=2328)
Function definitions:[{},
{"name:":"windowingtablefunction","order by:":"_col2 DESC NULLS
LAST","partition by:":"_col0"}
]
Select Operator [SEL_8] (rows=16859610 width=2328)
Output:["_col0","_col2"]
<-Reducer 2 [SIMPLE_EDGE]
SHUFFLE [RS_7]
PartitionCols:_col0
Select Operator [SEL_6] (rows=16859610 width=2328)
Output:["_col0","_col2"]
Group By Operator [GBY_5] (rows=16859610 width=2328)
Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0,
KEY._col1
<-Map 1 [SIMPLE_EDGE]
SHUFFLE [RS_4]
PartitionCols:_col0, _col1
Group By Operator [GBY_3] (rows=33719220 width=2328)
Output:["_col0","_col1","_col2"],aggregations:["sum(payable_price)"],keys:store_code,
all_pay_id
Select Operator [SEL_2] (rows=33719220 width=2328)
Output:["store_code","all_pay_id","payable_price"]
TableScan [TS_0] (rows=33719220 width=2328)
data_promotion@mdw_user_promotion_shopping_behaivor_four_week_detail,a,Tbl:COMPLETE,Col:NONE,Output:["all_pay_id","store_code","payable_price"]
Reducer 4 show PTF , so I don't know if this result has anything to do with
PTF? Can you help me.
was:
h3. 1.Phenomenon
Running the same SQL(hive on tez) several times will result in inconsistent
results . The following results are generated respectively
1)
OK
3951864 808
2)
OK
1822979 353
But the first result is true.
h3. 2.SQL
select
count(1),count(distinct store_code)
from
(
select
store_code,
all_pay_id,
payable_price,
row_number()over(partition by store_code order by payable_price desc) as rw,
count(1)over(partition by store_code) as store_user
from
(
select
store_code,
all_pay_id,
sum(payable_price) as payable_price
from data_promotion.mdw_user_promotion_shopping_behaivor_four_week_detail a
where dt='20190904'
group by store_code,all_pay_id
) a
order by rw
) a
where rw/store_user<=0.8
;
h3. 3.Troubleshooting
I lookup task result from tez web ui，give the result as follows：
the first:
!image-2019-09-10-12-40-34-958.png!
the second:
!image-2019-09-10-12-40-53-338.png!
so , The
[jira] [Updated] (HIVE-22181) Same query but different result occasionally
[
https://issues.apache.org/jira/browse/HIVE-22181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qingfa zhou updated HIVE-22181:
---
Attachment: image-2019-09-10-12-40-53-338.png
> Same query but different result occasionally
>
>
> Key: HIVE-22181
> URL: https://issues.apache.org/jira/browse/HIVE-22181
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.3.0
> Environment: hadoop：2.7.3.2.6.1.0-129
> hive：2.3.0
> tez：0.8.4
> java：1.8.0_212
>Reporter: qingfa zhou
>Priority: Major
> Attachments: WX20190910-123604.png, WX20190910-123621.png,
> WX20190910-123641.png, image-2019-09-10-12-40-34-958.png,
> image-2019-09-10-12-40-53-338.png
>
>
> h3. 1.Phenomenon
> Running the same SQL(hive on tez) several times will result in
> inconsistent results . The following results are generated respectively
> 1)
> OK
> 3951864 808
> 2)
> OK
> 1822979 353
> But the first result is true.
> h3. 2.SQL
> select
> count(1),count(distinct store_code)
> from
> (
> select
> store_code,
> all_pay_id,
> payable_price,
> row_number()over(partition by store_code order by payable_price desc) as rw,
> count(1)over(partition by store_code) as store_user
> from
> (
> select
> store_code,
> all_pay_id,
> sum(payable_price) as payable_price
> from data_promotion.mdw_user_promotion_shopping_behaivor_four_week_detail a
> where dt='20190904'
> group by store_code,all_pay_id
> ) a
> order by rw
> ) a
> where rw/store_user<=0.8
> ;
>
> h3. 3.Troubleshooting
> I lookup task result from tez web ui，give the result as follows：
> the first:
> the second:
> !image-2019-09-10-12-18-08-769.png!
> so , The second RECORDS_OUT_INTERMEDIATE_Reducer_4's result is much
> lower than the first.
> This is detailed info of the second result from reducer_3 to reducer_4
> !image-2019-09-10-12-26-41-681.png!
> This is execution plan of the sql:
> OK
> Vertex dependency in root stage
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
> Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
> Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Reducer 6
> File Output Operator [FS_24]
> Group By Operator [GBY_22] (rows=1 width=128)
> Output:["_col0","_col1"],aggregations:["count(VALUE._col0)","count(DISTINCT
> KEY._col0:0._col0)"]
> <-Reducer 5 [SIMPLE_EDGE]
> SHUFFLE [RS_21]
> Group By Operator [GBY_20] (rows=5619870 width=2328)
> Output:["_col0","_col1","_col2"],aggregations:["count(1)","count(DISTINCT
> _col0)"],keys:_col0
> Select Operator [SEL_17] (rows=5619870 width=2328)
> Output:["_col0"]
> <-Reducer 4 [SIMPLE_EDGE]
> SHUFFLE [RS_16]
> Select Operator [SEL_14] (rows=5619870 width=2328)
> Output:["_col0","_col3"]
> Filter Operator [FIL_25] (rows=5619870 width=2328)
> predicate:((_col0 / count_window_1) <= 0.8)
> PTF Operator [PTF_13] (rows=16859610 width=2328)
> Function definitions:[{},\\{"name:":"windowingtablefunction","order
> by:":"_col1 ASC NULLS FIRST","partition by:":"_col1"}]
> Select Operator [SEL_12] (rows=16859610 width=2328)
> Output:["_col0","_col1"]
> <-Reducer 3 [SIMPLE_EDGE]
> SHUFFLE [RS_11]
> PartitionCols:_col0
> Select Operator [SEL_10] (rows=16859610 width=2328)
> Output:["_col0","row_number_window_0"]
> PTF Operator [PTF_9] (rows=16859610 width=2328)
> Function definitions:[{},\\{"name:":"windowingtablefunction","order
> by:":"_col2 DESC NULLS LAST","partition by:":"_col0"}]
> Select Operator [SEL_8] (rows=16859610 width=2328)
> Output:["_col0","_col2"]
> <-Reducer 2 [SIMPLE_EDGE]
> SHUFFLE [RS_7]
> PartitionCols:_col0
> Select Operator [SEL_6] (rows=16859610 width=2328)
> Output:["_col0","_col2"]
> Group By Operator [GBY_5] (rows=16859610 width=2328)
>
> Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0,
> KEY._col1
> <-Map 1 [SIMPLE_EDGE]
> SHUFFLE [RS_4]
> PartitionCols:_col0, _col1
> Group By Operator [GBY_3] (rows=33719220 width=2328)
>
> Output:["_col0","_col1","_col2"],aggregations:["sum(payable_price)"],keys:store_code,
> all_pay_id
> Select Operator [SEL_2] (rows=33719220 width=2328)
> Output:["store_code","all_pay_id","payable_price"]
> TableScan [TS_0] (rows=33719220 width=2328)
>
> data_promotion@mdw_user_promotion_shopping_behaivor_four_week_detail,a,Tbl:COMPLETE,Col:NONE,Output:["all_pay_id","store_code","payable_price"]
>
> Reducer 4 show PTF , so I don't know if this result has anything to do
> with PTF? Can you help me.
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22181) Same query but different result occasionally
[
https://issues.apache.org/jira/browse/HIVE-22181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qingfa zhou updated HIVE-22181:
---
Attachment: image-2019-09-10-12-40-34-958.png
> Same query but different result occasionally
>
>
> Key: HIVE-22181
> URL: https://issues.apache.org/jira/browse/HIVE-22181
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.3.0
> Environment: hadoop：2.7.3.2.6.1.0-129
> hive：2.3.0
> tez：0.8.4
> java：1.8.0_212
>Reporter: qingfa zhou
>Priority: Major
> Attachments: WX20190910-123604.png, WX20190910-123621.png,
> WX20190910-123641.png, image-2019-09-10-12-40-34-958.png,
> image-2019-09-10-12-40-53-338.png
>
>
> h3. 1.Phenomenon
> Running the same SQL(hive on tez) several times will result in
> inconsistent results . The following results are generated respectively
> 1)
> OK
> 3951864 808
> 2)
> OK
> 1822979 353
> But the first result is true.
> h3. 2.SQL
> select
> count(1),count(distinct store_code)
> from
> (
> select
> store_code,
> all_pay_id,
> payable_price,
> row_number()over(partition by store_code order by payable_price desc) as rw,
> count(1)over(partition by store_code) as store_user
> from
> (
> select
> store_code,
> all_pay_id,
> sum(payable_price) as payable_price
> from data_promotion.mdw_user_promotion_shopping_behaivor_four_week_detail a
> where dt='20190904'
> group by store_code,all_pay_id
> ) a
> order by rw
> ) a
> where rw/store_user<=0.8
> ;
>
> h3. 3.Troubleshooting
> I lookup task result from tez web ui，give the result as follows：
> the first:
> the second:
> !image-2019-09-10-12-18-08-769.png!
> so , The second RECORDS_OUT_INTERMEDIATE_Reducer_4's result is much
> lower than the first.
> This is detailed info of the second result from reducer_3 to reducer_4
> !image-2019-09-10-12-26-41-681.png!
> This is execution plan of the sql:
> OK
> Vertex dependency in root stage
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
> Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
> Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Reducer 6
> File Output Operator [FS_24]
> Group By Operator [GBY_22] (rows=1 width=128)
> Output:["_col0","_col1"],aggregations:["count(VALUE._col0)","count(DISTINCT
> KEY._col0:0._col0)"]
> <-Reducer 5 [SIMPLE_EDGE]
> SHUFFLE [RS_21]
> Group By Operator [GBY_20] (rows=5619870 width=2328)
> Output:["_col0","_col1","_col2"],aggregations:["count(1)","count(DISTINCT
> _col0)"],keys:_col0
> Select Operator [SEL_17] (rows=5619870 width=2328)
> Output:["_col0"]
> <-Reducer 4 [SIMPLE_EDGE]
> SHUFFLE [RS_16]
> Select Operator [SEL_14] (rows=5619870 width=2328)
> Output:["_col0","_col3"]
> Filter Operator [FIL_25] (rows=5619870 width=2328)
> predicate:((_col0 / count_window_1) <= 0.8)
> PTF Operator [PTF_13] (rows=16859610 width=2328)
> Function definitions:[{},\\{"name:":"windowingtablefunction","order
> by:":"_col1 ASC NULLS FIRST","partition by:":"_col1"}]
> Select Operator [SEL_12] (rows=16859610 width=2328)
> Output:["_col0","_col1"]
> <-Reducer 3 [SIMPLE_EDGE]
> SHUFFLE [RS_11]
> PartitionCols:_col0
> Select Operator [SEL_10] (rows=16859610 width=2328)
> Output:["_col0","row_number_window_0"]
> PTF Operator [PTF_9] (rows=16859610 width=2328)
> Function definitions:[{},\\{"name:":"windowingtablefunction","order
> by:":"_col2 DESC NULLS LAST","partition by:":"_col0"}]
> Select Operator [SEL_8] (rows=16859610 width=2328)
> Output:["_col0","_col2"]
> <-Reducer 2 [SIMPLE_EDGE]
> SHUFFLE [RS_7]
> PartitionCols:_col0
> Select Operator [SEL_6] (rows=16859610 width=2328)
> Output:["_col0","_col2"]
> Group By Operator [GBY_5] (rows=16859610 width=2328)
>
> Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0,
> KEY._col1
> <-Map 1 [SIMPLE_EDGE]
> SHUFFLE [RS_4]
> PartitionCols:_col0, _col1
> Group By Operator [GBY_3] (rows=33719220 width=2328)
>
> Output:["_col0","_col1","_col2"],aggregations:["sum(payable_price)"],keys:store_code,
> all_pay_id
> Select Operator [SEL_2] (rows=33719220 width=2328)
> Output:["store_code","all_pay_id","payable_price"]
> TableScan [TS_0] (rows=33719220 width=2328)
>
> data_promotion@mdw_user_promotion_shopping_behaivor_four_week_detail,a,Tbl:COMPLETE,Col:NONE,Output:["all_pay_id","store_code","payable_price"]
>
> Reducer 4 show PTF , so I don't know if this result has anything to do
> with PTF? Can you help me.
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22181) Same query but different result occasionally
[
https://issues.apache.org/jira/browse/HIVE-22181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qingfa zhou updated HIVE-22181:
---
Description:
h3. 1.Phenomenon
Running the same SQL(hive on tez) several times will result in inconsistent
results . The following results are generated respectively
1)
OK
3951864 808
2)
OK
1822979 353
But the first result is true.
h3. 2.SQL
select
count(1),count(distinct store_code)
from
(
select
store_code,
all_pay_id,
payable_price,
row_number()over(partition by store_code order by payable_price desc) as rw,
count(1)over(partition by store_code) as store_user
from
(
select
store_code,
all_pay_id,
sum(payable_price) as payable_price
from data_promotion.mdw_user_promotion_shopping_behaivor_four_week_detail a
where dt='20190904'
group by store_code,all_pay_id
) a
order by rw
) a
where rw/store_user<=0.8
;
h3. 3.Troubleshooting
I lookup task result from tez web ui，give the result as follows：
the first:
the second:
!image-2019-09-10-12-18-08-769.png!
so , The second RECORDS_OUT_INTERMEDIATE_Reducer_4's result is much lower
than the first.
This is detailed info of the second result from reducer_3 to reducer_4
!image-2019-09-10-12-26-41-681.png!
This is execution plan of the sql:
OK
Vertex dependency in root stage
Reducer 2 <- Map 1 (SIMPLE_EDGE)
Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
Stage-0
Fetch Operator
limit:-1
Stage-1
Reducer 6
File Output Operator [FS_24]
Group By Operator [GBY_22] (rows=1 width=128)
Output:["_col0","_col1"],aggregations:["count(VALUE._col0)","count(DISTINCT
KEY._col0:0._col0)"]
<-Reducer 5 [SIMPLE_EDGE]
SHUFFLE [RS_21]
Group By Operator [GBY_20] (rows=5619870 width=2328)
Output:["_col0","_col1","_col2"],aggregations:["count(1)","count(DISTINCT
_col0)"],keys:_col0
Select Operator [SEL_17] (rows=5619870 width=2328)
Output:["_col0"]
<-Reducer 4 [SIMPLE_EDGE]
SHUFFLE [RS_16]
Select Operator [SEL_14] (rows=5619870 width=2328)
Output:["_col0","_col3"]
Filter Operator [FIL_25] (rows=5619870 width=2328)
predicate:((_col0 / count_window_1) <= 0.8)
PTF Operator [PTF_13] (rows=16859610 width=2328)
Function definitions:[{},\\{"name:":"windowingtablefunction","order
by:":"_col1 ASC NULLS FIRST","partition by:":"_col1"}]
Select Operator [SEL_12] (rows=16859610 width=2328)
Output:["_col0","_col1"]
<-Reducer 3 [SIMPLE_EDGE]
SHUFFLE [RS_11]
PartitionCols:_col0
Select Operator [SEL_10] (rows=16859610 width=2328)
Output:["_col0","row_number_window_0"]
PTF Operator [PTF_9] (rows=16859610 width=2328)
Function definitions:[{},\\{"name:":"windowingtablefunction","order
by:":"_col2 DESC NULLS LAST","partition by:":"_col0"}]
Select Operator [SEL_8] (rows=16859610 width=2328)
Output:["_col0","_col2"]
<-Reducer 2 [SIMPLE_EDGE]
SHUFFLE [RS_7]
PartitionCols:_col0
Select Operator [SEL_6] (rows=16859610 width=2328)
Output:["_col0","_col2"]
Group By Operator [GBY_5] (rows=16859610 width=2328)
Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0,
KEY._col1
<-Map 1 [SIMPLE_EDGE]
SHUFFLE [RS_4]
PartitionCols:_col0, _col1
Group By Operator [GBY_3] (rows=33719220 width=2328)
Output:["_col0","_col1","_col2"],aggregations:["sum(payable_price)"],keys:store_code,
all_pay_id
Select Operator [SEL_2] (rows=33719220 width=2328)
Output:["store_code","all_pay_id","payable_price"]
TableScan [TS_0] (rows=33719220 width=2328)
data_promotion@mdw_user_promotion_shopping_behaivor_four_week_detail,a,Tbl:COMPLETE,Col:NONE,Output:["all_pay_id","store_code","payable_price"]
Reducer 4 show PTF , so I don't know if this result has anything to do with
PTF? Can you help me.
was:
h3. 1.Phenomenon
Running the same SQL(hive on tez) several times will result in inconsistent
results . The following results are generated respectively
1)
OK
3951864 808
2)
OK
1822979 353
But the first result is true.
h3. 2.SQL
select
count(1),count(distinct store_code)
from
(
select
store_code,
all_pay_id,
payable_price,
row_number()over(partition by store_code order by payable_price desc) as rw,
count(1)over(partition by store_code) as store_user
from
(
select
store_code,
all_pay_id,
sum(payable_price) as payable_price
from data_promotion.mdw_user_promotion_shopping_behaivor_four_week_detail a
where dt='20190904'
group by store_code,all_pay_id
) a
order by rw
) a
where rw/store_user<=0.8
;
h3. 3.Troubleshooting
I lookup task result from tez web ui，give the result as follows：
the first: !image-2019-09-10-12-16-45-385.png!
the second:
!image-2019-09-10-12-18-08-769.png!
so , The second RECORDS_OUT_INTERMEDIATE_Reducer_4's
[jira] [Commented] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[
https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926323#comment-16926323
]
Hive QA commented on HIVE-20113:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
21s{color} | {color:blue} ql in master has 2246 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 1 new + 347 unchanged - 1
fixed = 348 total (was 348) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18516/dev-support/hive-personality.sh
|
| git revision | master / d244da5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18516/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18516/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Shuffle avoidance: Disable 1-1 edges for sorted shuffle
>
>
> Key: HIVE-20113
> URL: https://issues.apache.org/jira/browse/HIVE-20113
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Labels: Branch3Candidate
> Attachments: HIVE-20113.1.patch, HIVE-20113.2.patch,
> HIVE-20113.3.patch, HIVE-20113.4.patch, HIVE-20113.4.patch,
> HIVE-20113.5.patch, HIVE-20113.6.patch, HIVE-20113.7.patch
>
>
> The sorted shuffle avoidance can have some issues when the shuffle data gets
> broken up into multiple chunks on disk.
> The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to
> have a final merge at all, it should open a single compressed file and write
> a single index entry.
> Until the shuffle issue is resolved & a lot more testing, it is prudent to
> disable the optimization for sorted shuffle edges and stop rewriting the
> RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD).
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22079) Post order walker for iterating over expression tree
[
https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22079:
---
Status: Open (was: Patch Available)
> Post order walker for iterating over expression tree
>
>
> Key: HIVE-22079
> URL: https://issues.apache.org/jira/browse/HIVE-22079
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer, Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch,
> HIVE-22079.3.patch
>
>
> Current {{DefaultGraphWalker}} is used to iterate over an expression tree.
> This walker uses hash map to keep track of visited/processed nodes. If an
> expression tree is large this adds significant overhead due to map lookup.
> For an expression trees we can instead use post order traversal and avoid
> using map.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22079) Post order walker for iterating over expression tree
[
https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22079:
---
Status: Patch Available (was: Open)
> Post order walker for iterating over expression tree
>
>
> Key: HIVE-22079
> URL: https://issues.apache.org/jira/browse/HIVE-22079
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer, Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch,
> HIVE-22079.3.patch
>
>
> Current {{DefaultGraphWalker}} is used to iterate over an expression tree.
> This walker uses hash map to keep track of visited/processed nodes. If an
> expression tree is large this adds significant overhead due to map lookup.
> For an expression trees we can instead use post order traversal and avoid
> using map.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22079) Post order walker for iterating over expression tree
[
https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22079:
---
Attachment: HIVE-22079.3.patch
> Post order walker for iterating over expression tree
>
>
> Key: HIVE-22079
> URL: https://issues.apache.org/jira/browse/HIVE-22079
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer, Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch,
> HIVE-22079.3.patch
>
>
> Current {{DefaultGraphWalker}} is used to iterate over an expression tree.
> This walker uses hash map to keep track of visited/processed nodes. If an
> expression tree is large this adds significant overhead due to map lookup.
> For an expression trees we can instead use post order traversal and avoid
> using map.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926313#comment-16926313
]
Hive QA commented on HIVE-15956:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12979885/HIVE-15956.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16750 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18515/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18515/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18515/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12979885 - PreCommit-HIVE-Build
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-15956.2.patch, HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926285#comment-16926285
]
Hive QA commented on HIVE-15956:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
6s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 412 unchanged - 0 fixed = 413 total (was 412) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
14s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 179 unchanged - 2 fixed = 180 total (was 181) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | org.apache.hadoop.hive.metastore.ObjectStore$18.run(List) concatenates
strings using + in a loop At ObjectStore.java:in a loop At
ObjectStore.java:[line 8771] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18515/dev-support/hive-personality.sh
|
| git revision | master / d244da5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18515/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18515/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18515/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-15956.2.patch, HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table
[jira] [Commented] (HIVE-22158) HMS Translation layer - Disallow non-ACID MANAGED tables.
[
https://issues.apache.org/jira/browse/HIVE-22158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926275#comment-16926275
]
Hive QA commented on HIVE-22158:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12979870/HIVE-22158.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16750 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18514/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18514/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18514/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12979870 - PreCommit-HIVE-Build
> HMS Translation layer - Disallow non-ACID MANAGED tables.
> -
>
> Key: HIVE-22158
> URL: https://issues.apache.org/jira/browse/HIVE-22158
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22158.1.patch, HIVE-22158.1.patch,
> HIVE-22158.2.patch
>
>
> In the recent commits, we have allowed non-ACID MANAGED tables to be created
> by clients that have some form of ACID WRITE capabilities.
> I think it would make sense to disallow this entirely. MANAGED tables should
> be ACID tables only.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22158) HMS Translation layer - Disallow non-ACID MANAGED tables.
[
https://issues.apache.org/jira/browse/HIVE-22158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926241#comment-16926241
]
Hive QA commented on HIVE-22158:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
4s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 3 new + 402 unchanged - 3 fixed = 405 total (was 405) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} itests/hive-unit: The patch generated 4 new + 151
unchanged - 0 fixed = 155 total (was 151) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18514/dev-support/hive-personality.sh
|
| git revision | master / d244da5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18514/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18514/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: standalone-metastore/metastore-server itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18514/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HMS Translation layer - Disallow non-ACID MANAGED tables.
> -
>
> Key: HIVE-22158
> URL: https://issues.apache.org/jira/browse/HIVE-22158
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22158.1.patch, HIVE-22158.1.patch,
> HIVE-22158.2.patch
>
>
> In the recent commits, we have allowed non-ACID MANAGED tables to be created
> by clients
[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[
https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926228#comment-16926228
]
Hive QA commented on HIVE-22126:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12979861/HIVE-22126.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16750 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18513/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18513/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18513/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12979861 - PreCommit-HIVE-Build
> hive-exec packaging should shade guava
> --
>
> Key: HIVE-22126
> URL: https://issues.apache.org/jira/browse/HIVE-22126
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22126.01.patch
>
>
> The ql/pom.xml includes complete guava library into hive-exec.jar
> https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a
> problems for downstream clients of hive which have hive-exec.jar in their
> classpath since they are pinned to the same guava version as that of hive.
> We should shade guava classes so that other components which depend on
> hive-exec can independently use a different version of guava as needed.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[
https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926205#comment-16926205
]
Hive QA commented on HIVE-22126:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18513/dev-support/hive-personality.sh
|
| git revision | master / d244da5 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18513/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> hive-exec packaging should shade guava
> --
>
> Key: HIVE-22126
> URL: https://issues.apache.org/jira/browse/HIVE-22126
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22126.01.patch
>
>
> The ql/pom.xml includes complete guava library into hive-exec.jar
> https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a
> problems for downstream clients of hive which have hive-exec.jar in their
> classpath since they are pinned to the same guava version as that of hive.
> We should shade guava classes so that other components which depend on
> hive-exec can independently use a different version of guava as needed.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926202#comment-16926202
]
Hive QA commented on HIVE-21884:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
18s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
16s{color} | {color:blue} standalone-metastore/metastore-common in master has
32 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 61 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
7s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 2246 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/util in master has 44 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 1 new + 495 unchanged - 0 fixed = 496 total (was 495) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} common: The patch generated 2 new + 367 unchanged - 0
fixed = 369 total (was 367) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch metastore passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
25s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 18 new + 1250 unchanged - 15 fixed = 1268 total (was 1265) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 61 new + 224 unchanged - 0
fixed = 285 total (was 224) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} service: The patch generated 0 new + 38 unchanged -
1 fixed = 38 total (was 39) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
10s{color} | {color:red} root: The patch generated 82 new + 2421 unchanged - 16
fixed = 2503 total (was 2437) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch hcatalog-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch util passed checkstyle {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 17 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 77 line(s) with tabs. {color} |
| {color:green}+1{color} |
[jira] [Commented] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926185#comment-16926185
]
Hive QA commented on HIVE-21884:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12979859/HIVE-21844.07.patch
{color:green}SUCCESS:{color} +1 due to 9 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 65 failed/errored test(s), 16784 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_materialized_view_rewrite_ssb]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timeseries]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz2]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_topn]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_dynamic_partition]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_extractTime]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_floorTime]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_alter]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_ts]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_avro]
(batchId=297)
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_basic]
(batchId=297)
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_csv]
(batchId=297)
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_delimited]
(batchId=297)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[table_nonprintable]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[temp_table_external]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[strict_managed_tables_sysdb]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb_schq]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_join_hash]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_case_when_2]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_udf_string_to_boolean]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucket4]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_partitioner]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_semijoin]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_map_operators]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_merge]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[insert_overwrite_directory2]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge4]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge_incompat1]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[root_dir_external_table]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_2]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_3]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_4]
(batchId=195)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_schq1]
(batchId=102)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=104)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ctas_noperm_loc]
(batchId=104)
[jira] [Assigned] (HIVE-21985) LLAP IO: Log schema evolution incompatibilities at WARN level always
[ https://issues.apache.org/jira/browse/HIVE-21985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mustafa Iman reassigned HIVE-21985: --- Assignee: Mustafa Iman > LLAP IO: Log schema evolution incompatibilities at WARN level always > > > Key: HIVE-21985 > URL: https://issues.apache.org/jira/browse/HIVE-21985 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Mustafa Iman >Priority: Major > > When reading incompatible schema, LLAP IO simply skips over the file and does > not cache it. > The logging at WARN level would be useful and simplify the root-cause via > logs. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22076) JDK11: Remove ParallelGC in debug.sh
[ https://issues.apache.org/jira/browse/HIVE-22076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22076: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Gopal! > JDK11: Remove ParallelGC in debug.sh > > > Key: HIVE-22076 > URL: https://issues.apache.org/jira/browse/HIVE-22076 > Project: Hive > Issue Type: Bug > Components: Diagnosability >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22076.1.patch > > > The JDK debug mode no longer depends on ParallelGC > This was a workaround for JDK6 bug - > https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6862295 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22076) JDK11: Remove ParallelGC in debug.sh
[ https://issues.apache.org/jira/browse/HIVE-22076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926181#comment-16926181 ] Ashutosh Chauhan commented on HIVE-22076: - +1 > JDK11: Remove ParallelGC in debug.sh > > > Key: HIVE-22076 > URL: https://issues.apache.org/jira/browse/HIVE-22076 > Project: Hive > Issue Type: Bug > Components: Diagnosability >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Minor > Attachments: HIVE-22076.1.patch > > > The JDK debug mode no longer depends on ParallelGC > This was a workaround for JDK6 bug - > https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6862295 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-9223) HiveServer2 on Tez doesn't support concurrent queries within one session
[ https://issues.apache.org/jira/browse/HIVE-9223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926160#comment-16926160 ] Tak Lon (Stephen) Wu commented on HIVE-9223: we're also seeing this issue, is this Jira still valid for tracking ? > HiveServer2 on Tez doesn't support concurrent queries within one session > > > Key: HIVE-9223 > URL: https://issues.apache.org/jira/browse/HIVE-9223 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Pala M Muthaia >Priority: Major > > When a user submits multiple queries in the same HS2 session (using thrift > interface) concurrently, the query goes through the same TezSessionState and > ends up being submitted to the same Tez AM, and the second query fails with > the error "App master already running a DAG" > Is this by design? I looked into the code, and the comments as well as the > code suggest support only for serial execution of queries within the same > HiveServer2 session (on tez). > This works for CLI environment but in a server, it is plausible that client > sends multiple concurrent queries under the same session (e.g: a web app that > executes queries for user, such as Cloudera Hue). So shouldn't HS2 on Tez > implementation support concurrent queries? -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22164) Vectorized Limit operator returns wrong number of results with offset
[ https://issues.apache.org/jira/browse/HIVE-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926137#comment-16926137 ] Richard Zhang commented on HIVE-22164: -- hi Vineet: Can you do a review of the above pull request? > Vectorized Limit operator returns wrong number of results with offset > - > > Key: HIVE-22164 > URL: https://issues.apache.org/jira/browse/HIVE-22164 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22164.1.patch > > > Vectorized Limit operator returns wrong number of results with offset -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22059) hive-exec jar doesn't contain (fasterxml) jackson library
[
https://issues.apache.org/jira/browse/HIVE-22059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926105#comment-16926105
]
Hive QA commented on HIVE-22059:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12979858/HIVE-22059.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 632 failed/errored test(s), 16015 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=113)
[bucketmapjoin4.q,bucket_map_join_spark4.q,union21.q,groupby2_noskew.q,timestamp_2.q,date_join1.q,mergejoins.q,smb_mapjoin_11.q,auto_sortmerge_join_3.q,mapjoin_test_outer.q,vectorization_9.q,merge2.q,groupby6_noskew.q,auto_join_without_localtask.q,multi_join_union.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=114)
[join_cond_pushdown_unqual4.q,union_remove_7.q,join13.q,join_vc.q,groupby_cube1.q,parquet_vectorization_2.q,bucket_map_join_spark2.q,sample3.q,smb_mapjoin_19.q,union23.q,union.q,union31.q,cbo_udf_udaf.q,ptf_decimal.q,bucketmapjoin2.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=115)
[parallel_join1.q,union27.q,union12.q,groupby7_map_multi_single_reducer.q,varchar_join1.q,join7.q,join_reorder4.q,skewjoinopt2.q,bucketsortoptimize_insert_2.q,smb_mapjoin_17.q,script_env_var1.q,groupby7_map.q,bucketsortoptimize_insert_8.q,stats16.q,union20.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=116)
[ptf_general_queries.q,auto_join_reordering_values.q,sample2.q,join1.q,decimal_join.q,mapjoin_subquery2.q,join32_lessSize.q,mapjoin1.q,skewjoinopt18.q,union_remove_18.q,join25.q,groupby3.q,groupby9.q,bucketsortoptimize_insert_6.q,ctas.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=117)
[groupby_map_ppr.q,nullgroup4_multi_distinct.q,join_rc.q,union14.q,order2.q,smb_mapjoin_12.q,vector_cast_constant.q,union_remove_4.q,parquet_vectorization_1.q,auto_join11.q,udaf_collect_set.q,vectorization_12.q,groupby_sort_skew_1_23.q,smb_mapjoin_25.q,skewjoinopt12.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=118)
[skewjoinopt15.q,auto_join18.q,list_bucket_dml_2.q,input1_limit.q,load_dyn_part3.q,union_remove_14.q,auto_sortmerge_join_14.q,auto_sortmerge_join_15.q,union10.q,bucket_map_join_tez2.q,groupby5_map_skew.q,load_dyn_part7.q,join_reorder.q,bucketmapjoin8.q,union34.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=119)
[avro_joins.q,parquet_vectorization_8.q,auto_join14.q,vectorization_14.q,auto_join26.q,stats1.q,cbo_stats.q,union22.q,union_view.q,subquery_views.q,smb_mapjoin_22.q,stats15.q,ptf_matchpath.q,transform_ppr1.q,sample1.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=120)
[limit_pushdown2.q,leftsemijoin_mr.q,parquet_vectorization_0.q,skewjoinopt16.q,bucket3.q,skewjoinopt13.q,auto_sortmerge_join_6.q,bucketmapjoin9.q,auto_join15.q,union_remove_24.q,join22.q,sample4.q,multi_insert_gby.q,join33.q,join_cond_pushdown_unqual2.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=121)
[vector_decimal_aggregate.q,skewjoin_noskew.q,ppd_join3.q,auto_join23.q,join10.q,union_ppr.q,subquery_multi.q,join32.q,input18.q,cbo_simple_select.q,ptf.q,vectorized_nested_mapjoin.q,union18.q,groupby1.q,join_reorder2.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=122)
[order.q,join_merge_multi_expressions.q,skewjoinopt10.q,union_remove_11.q,vectorized_math_funcs.q,vectorization_2.q,skewjoinopt6.q,union_remove_19.q,join14.q,groupby_multi_single_reducer2.q,stats3.q,parquet_join.q,outer_join_ppr.q,join26.q,load_dyn_part10.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=123)
[skewjoinopt3.q,skewjoinopt19.q,timestamp_comparison.q,bucketmapjoin_negative.q,union5.q,insert_into1.q,vectorization_4.q,parquet_vectorization_10.q,vector_left_outer_join.q,decimal_1_1.q,semijoin.q,skewjoinopt9.q,smb_mapjoin_3.q,stats10.q,rcfile_bigdata.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=124)
[parquet_vectorization_limit.q,multi_insert_mixed.q,smb_mapjoin_4.q,join_cond_pushdown_3.q,insert1.q,union_remove_10.q,mapreduce2.q,udf_in_file.q,skewjoinopt5.q,auto_join12.q,skewjoin.q,vectorization_part_project.q,vector_count_distinct.q,nullgroup4.q,parquet_vectorization_12.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=125)
[jira] [Updated] (HIVE-22177) Beeline query for special character like £ does not give back result
[
https://issues.apache.org/jira/browse/HIVE-22177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanjay Mishra updated HIVE-22177:
-
Affects Version/s: 2.3.6
> Beeline query for special character like £ does not give back result
>
>
> Key: HIVE-22177
> URL: https://issues.apache.org/jira/browse/HIVE-22177
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.1.0, 2.3.6
> Environment: Cloudera 5.15.1 Hive 1.1.0
> beeline and Impala
>Reporter: Sanjay Mishra
>Priority: Major
>
> I have a table in hive with a column having special character like £. When I
> try to a query as simple as 'SELECT * FROM TABLE WHERE COL = "£"; ', the
> query does not come back with any result.
> The table data is UTF-8 encoded. The sql file that contains the above sql and
> run via beeline is also UTF-8 encoded.
> Interestingly, if I try to create the run the same query using impala against
> the same table, I get back the columns containing the £ symbol.
> Also when the data from the table is exported (SELECT * FROM THE TABLE and
> save the output to csv file) the column with special character is shown as
> NULL.
>
> Steps to replicate:
> # create test.sql file with the below content:
> #
> {code:java}
> CREATE TABLE T1 (test String);
> INSERT INTO T1 VALUES('£');
> {code}
> # beeline>!run test.sql
> # create another file test_select.sql with the below content:
> #
> {code:java}
> SELECT * FROM T1 WHERE test='£';{code}
> # beeline>!run test_select.sql
> # No results will be returned back
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22059) hive-exec jar doesn't contain (fasterxml) jackson library
[
https://issues.apache.org/jira/browse/HIVE-22059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926039#comment-16926039
]
Hive QA commented on HIVE-22059:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18511/dev-support/hive-personality.sh
|
| git revision | master / 9667f25 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18511/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> hive-exec jar doesn't contain (fasterxml) jackson library
> -
>
> Key: HIVE-22059
> URL: https://issues.apache.org/jira/browse/HIVE-22059
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22059.01.patch, HIVE-22059.02.patch,
> HIVE-22059.03.patch
>
>
> While deploying master branch into a container I've noticed that the jackson
> libraries are not 100% sure that are available at runtime - this is probably
> due to the fact that we are still using the "old" codehaus jackson and also
> the "new" fasterxml one.
> {code:java}
> ]Vertex killed, vertexName=Reducer 2,
> vertexId=vertex_1564408646590_0005_1_01, diagnostics=[Vertex received Kill in
> INITED state., Vertex vertex_1564408646590_0005_1_01 [Reducer 2]
> killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to
> VERTEX_FAILURE. failedVertices:1 killedVertices:1
> INFO : Completed executing
> command(queryId=vagrant_20190729141949_8d8c7f0d-0ac4-4d76-ba12-6ec01561b040);
> Time taken: 5.127 seconds
> INFO : Concurrency mode is disabled, not creating a lock manager
> Error: Error while processing statement: FAILED: Execution Error, return code
> 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed,
> vertexName=Map 1, vertexId=vertex_1564408646590_0005_1_00,
> diagnostics=[Vertex vertex_1564408646590_0005_1_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: _dummy_table initializer failed,
> vertex=vertex_1564408646590_0005_1_00 [Map 1],
> java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
> at org.apache.hadoop.hive.ql.exec.Utilities.(Utilities.java:226)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:428)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:508)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:488)
> at
>
[jira] [Updated] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[ https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20113: --- Attachment: HIVE-20113.7.patch > Shuffle avoidance: Disable 1-1 edges for sorted shuffle > > > Key: HIVE-20113 > URL: https://issues.apache.org/jira/browse/HIVE-20113 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: Branch3Candidate > Attachments: HIVE-20113.1.patch, HIVE-20113.2.patch, > HIVE-20113.3.patch, HIVE-20113.4.patch, HIVE-20113.4.patch, > HIVE-20113.5.patch, HIVE-20113.6.patch, HIVE-20113.7.patch > > > The sorted shuffle avoidance can have some issues when the shuffle data gets > broken up into multiple chunks on disk. > The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to > have a final merge at all, it should open a single compressed file and write > a single index entry. > Until the shuffle issue is resolved & a lot more testing, it is prudent to > disable the optimization for sorted shuffle edges and stop rewriting the > RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[ https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20113: --- Status: Open (was: Patch Available) > Shuffle avoidance: Disable 1-1 edges for sorted shuffle > > > Key: HIVE-20113 > URL: https://issues.apache.org/jira/browse/HIVE-20113 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: Branch3Candidate > Attachments: HIVE-20113.1.patch, HIVE-20113.2.patch, > HIVE-20113.3.patch, HIVE-20113.4.patch, HIVE-20113.4.patch, > HIVE-20113.5.patch, HIVE-20113.6.patch, HIVE-20113.7.patch > > > The sorted shuffle avoidance can have some issues when the shuffle data gets > broken up into multiple chunks on disk. > The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to > have a final merge at all, it should open a single compressed file and write > a single index entry. > Until the shuffle issue is resolved & a lot more testing, it is prudent to > disable the optimization for sorted shuffle edges and stop rewriting the > RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[ https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20113: --- Status: Patch Available (was: Open) > Shuffle avoidance: Disable 1-1 edges for sorted shuffle > > > Key: HIVE-20113 > URL: https://issues.apache.org/jira/browse/HIVE-20113 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: Branch3Candidate > Attachments: HIVE-20113.1.patch, HIVE-20113.2.patch, > HIVE-20113.3.patch, HIVE-20113.4.patch, HIVE-20113.4.patch, > HIVE-20113.5.patch, HIVE-20113.6.patch, HIVE-20113.7.patch > > > The sorted shuffle avoidance can have some issues when the shuffle data gets > broken up into multiple chunks on disk. > The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to > have a final merge at all, it should open a single compressed file and write > a single index entry. > Until the shuffle issue is resolved & a lot more testing, it is prudent to > disable the optimization for sorted shuffle edges and stop rewriting the > RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-15956:
--
Attachment: HIVE-15956.2.patch
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-15956.2.patch, HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko reassigned HIVE-15956:
-
Assignee: Denys Kuzmenko (was: Niklaus Xiao)
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22055) select count gives incorrect result after loading data from text file
[
https://issues.apache.org/jira/browse/HIVE-22055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925926#comment-16925926
]
Hive QA commented on HIVE-22055:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12979856/HIVE-22055.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16751 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18510/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18510/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18510/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12979856 - PreCommit-HIVE-Build
> select count gives incorrect result after loading data from text file
> -
>
> Key: HIVE-22055
> URL: https://issues.apache.org/jira/browse/HIVE-22055
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22055.2.patch
>
>
> Add one more load to mm_loaddata.q:
> Load data 3 times (both kv1.txt and kv2.txt contains 500 records)
> {code:java}
> create table load0_mm (key string, value string) stored as textfile
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> load data local inpath '../../data/files/kv1.txt' into table load0_mm;
> select count(1) from load0_mm;
> load data local inpath '../../data/files/kv2.txt' into table load0_mm;
> select count(1) from load0_mm;
> load data local inpath '../../data/files/kv2.txt' into table load0_mm;
> select count(1) from load0_mm;{code}
> Expected output
> {code:java}
> PREHOOK: query: load data local inpath '../../data/files/kv2.txt' into table
> load0_mm
> PREHOOK: type: LOAD
> A masked pattern was here
> PREHOOK: Output: default@load0_mm
> POSTHOOK: query: load data local inpath '../../data/files/kv2.txt' into table
> load0_mm
> POSTHOOK: type: LOAD
> A masked pattern was here
> POSTHOOK: Output: default@load0_mm
> PREHOOK: query: select count(1) from load0_mm
> PREHOOK: type: QUERY
> PREHOOK: Input: default@load0_mm
> A masked pattern was here
> POSTHOOK: query: select count(1) from load0_mm
> POSTHOOK: type: QUERY
> POSTHOOK: Input: default@load0_mm
> A masked pattern was here
> 1500{code}
> Got:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client Execution
> succeeded but contained differences (error code = 1) after executing
> mm_loaddata.q
> 63c63
> < 1480
> —
> > 1500
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22055) select count gives incorrect result after loading data from text file
[
https://issues.apache.org/jira/browse/HIVE-22055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925882#comment-16925882
]
Hive QA commented on HIVE-22055:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
8s{color} | {color:blue} ql in master has 2246 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} llap-server in master has 83 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18510/dev-support/hive-personality.sh
|
| git revision | master / 9667f25 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql llap-server itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18510/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> select count gives incorrect result after loading data from text file
> -
>
> Key: HIVE-22055
> URL: https://issues.apache.org/jira/browse/HIVE-22055
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22055.2.patch
>
>
> Add one more load to mm_loaddata.q:
> Load data 3 times (both kv1.txt and kv2.txt contains 500 records)
> {code:java}
> create table load0_mm (key string, value string) stored as textfile
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> load data local inpath '../../data/files/kv1.txt' into table load0_mm;
> select count(1) from load0_mm;
> load data local inpath '../../data/files/kv2.txt' into table load0_mm;
> select count(1) from load0_mm;
> load data local inpath '../../data/files/kv2.txt' into table load0_mm;
> select count(1) from load0_mm;{code}
> Expected output
> {code:java}
> PREHOOK: query: load data local inpath '../../data/files/kv2.txt' into table
> load0_mm
> PREHOOK: type: LOAD
> A masked
[jira] [Commented] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925857#comment-16925857
]
Hive QA commented on HIVE-20683:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12979857/HIVE-20683.10.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16751 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18509/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18509/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18509/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12979857 - PreCommit-HIVE-Build
> Add the Ability to push Dynamic Between and Bloom filters to Druid
> --
>
> Key: HIVE-20683
> URL: https://issues.apache.org/jira/browse/HIVE-20683
> Project: Hive
> Issue Type: New Feature
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20683.1.patch, HIVE-20683.10.patch,
> HIVE-20683.2.patch, HIVE-20683.3.patch, HIVE-20683.4.patch,
> HIVE-20683.5.patch, HIVE-20683.6.patch, HIVE-20683.8.patch, HIVE-20683.patch
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> For optimizing joins, Hive generates BETWEEN filter with min-max and BLOOM
> filter for filtering one side of semi-join.
> Druid 0.13.0 will have support for Bloom filters (Added via
> https://github.com/apache/incubator-druid/pull/6222)
> Implementation details -
> # Hive generates and passes the filters as part of 'filterExpr' in TableScan.
> # DruidQueryBasedRecordReader gets this filter passed as part of the conf.
> # During execution phase, before sending the query to druid in
> DruidQueryBasedRecordReader we will deserialize this filter, translate it
> into a DruidDimFilter and add it to existing DruidQuery. Tez executor
> already ensures that when we start reading results from the record reader,
> all the dynamic values are initialized.
> # Explaining a druid query also prints the query sent to druid as
> {{druid.json.query}}. We also need to make sure to update the druid query
> with the filters. During explain we do not have the actual values for the
> dynamic values, so instead of values we will print the dynamic expression
> itself as part of druid query.
> Note:- This work needs druid to be updated to version 0.13.0
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925785#comment-16925785
]
Hive QA commented on HIVE-20683:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
8s{color} | {color:blue} ql in master has 2246 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} druid-handler in master has 3 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
16s{color} | {color:blue} itests/qtest-druid in master has 7 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
8s{color} | {color:red} druid-handler: The patch generated 1 new + 0 unchanged
- 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
27s{color} | {color:red} root: The patch generated 1 new + 35 unchanged - 0
fixed = 36 total (was 35) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18509/dev-support/hive-personality.sh
|
| git revision | master / 9667f25 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18509/yetus/diff-checkstyle-druid-handler.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18509/yetus/diff-checkstyle-root.txt
|
| modules | C: ql druid-handler . itests itests/qtest-druid U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18509/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add the Ability to push Dynamic Between and Bloom filters to Druid
> --
>
> Key: HIVE-20683
> URL: https://issues.apache.org/jira/browse/HIVE-20683
> Project: Hive
> Issue Type: New Feature
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
[jira] [Updated] (HIVE-22158) HMS Translation layer - Disallow non-ACID MANAGED tables.
[ https://issues.apache.org/jira/browse/HIVE-22158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-22158: - Status: Patch Available (was: Open) Attached a patch to fix the failing test. The test failure is a resultant of a managed table being converted to external table. The replication code is not setup with the appropriate config properties to be able to work with external table locations. > HMS Translation layer - Disallow non-ACID MANAGED tables. > - > > Key: HIVE-22158 > URL: https://issues.apache.org/jira/browse/HIVE-22158 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-22158.1.patch, HIVE-22158.1.patch, > HIVE-22158.2.patch > > > In the recent commits, we have allowed non-ACID MANAGED tables to be created > by clients that have some form of ACID WRITE capabilities. > I think it would make sense to disallow this entirely. MANAGED tables should > be ACID tables only. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22158) HMS Translation layer - Disallow non-ACID MANAGED tables.
[ https://issues.apache.org/jira/browse/HIVE-22158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-22158: - Attachment: HIVE-22158.2.patch > HMS Translation layer - Disallow non-ACID MANAGED tables. > - > > Key: HIVE-22158 > URL: https://issues.apache.org/jira/browse/HIVE-22158 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-22158.1.patch, HIVE-22158.1.patch, > HIVE-22158.2.patch > > > In the recent commits, we have allowed non-ACID MANAGED tables to be created > by clients that have some form of ACID WRITE capabilities. > I think it would make sense to disallow this entirely. MANAGED tables should > be ACID tables only. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22158) HMS Translation layer - Disallow non-ACID MANAGED tables.
[ https://issues.apache.org/jira/browse/HIVE-22158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-22158: - Status: Open (was: Patch Available) > HMS Translation layer - Disallow non-ACID MANAGED tables. > - > > Key: HIVE-22158 > URL: https://issues.apache.org/jira/browse/HIVE-22158 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-22158.1.patch, HIVE-22158.1.patch, > HIVE-22158.2.patch > > > In the recent commits, we have allowed non-ACID MANAGED tables to be created > by clients that have some form of ACID WRITE capabilities. > I think it would make sense to disallow this entirely. MANAGED tables should > be ACID tables only. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22083) Values of tag order cannot be null, so it can be "byte" instead of "Byte"
[
https://issues.apache.org/jira/browse/HIVE-22083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925742#comment-16925742
]
Hive QA commented on HIVE-22083:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12979855/HIVE-22083.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18508/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18508/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18508/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-09-09 14:36:27.499
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-18508/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-09-09 14:36:27.514
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 9667f25 HIVE-21942: Remove useless MetastoreType enum from
AbstractCliConfig (Imre Molnar via Laszlo Bodor)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 9667f25 HIVE-21942: Remove useless MetastoreType enum from
AbstractCliConfig (Imre Molnar via Laszlo Bodor)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-09-09 14:36:29.097
+ rm -rf ../yetus_PreCommit-HIVE-Build-18508
+ mkdir ../yetus_PreCommit-HIVE-Build-18508
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-18508
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-18508/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/SkewJoinHandler.java: does
not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/SparkHashTableSinkOperator.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinBaseOperator.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/JoinReorder.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationOptimizer.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/BucketingSortingOpProcFactory.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenSparkSkewJoinProcessor.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java: does not
exist in index
error:
[jira] [Updated] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22126: - Status: Patch Available (was: Open) > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22126.01.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22126: - Attachment: HIVE-22126.01.patch > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22126.01.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-21508) ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
[
https://issues.apache.org/jira/browse/HIVE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925740#comment-16925740
]
Hive QA commented on HIVE-21508:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12979845/HIVE-21508.1.branch-2.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10587 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_semijoin_reduction2]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_semijoin_reduction]
(batchId=142)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=110)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion02
(batchId=264)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdate.testNonAcidToAcidConversion02
(batchId=276)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion02
(batchId=273)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18507/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18507/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18507/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12979845 - PreCommit-HIVE-Build
> ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
> --
>
> Key: HIVE-21508
> URL: https://issues.apache.org/jira/browse/HIVE-21508
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 3.2.0, 2.3.4
>Reporter: Adar Dembo
>Assignee: Ana Jalba
>Priority: Major
> Fix For: 4.0.0, 2.3.7
>
> Attachments: HIVE-21508.1.branch-2.3.patch, HIVE-21508.patch
>
>
> There's this block of code in {{HiveMetaStoreClient:resolveUris}} (called
> from the constructor) on master:
> {noformat}
> private URI metastoreUris[];
> ...
> if (MetastoreConf.getVar(conf,
> ConfVars.THRIFT_URI_SELECTION).equalsIgnoreCase("RANDOM")) {
> List uriList = Arrays.asList(metastoreUris);
> Collections.shuffle(uriList);
> metastoreUris = (URI[]) uriList.toArray();
> }
> {noformat}
> The cast to {{URI[]}} throws a {{ClassCastException}} beginning with JDK 10,
> possibly with JDK 9 as well. Note that {{THRIFT_URI_SELECTION}} defaults to
> {{RANDOM}} so this should affect anyone who creates a
> {{HiveMetaStoreClient}}. On master this can be overridden with {{SEQUENTIAL}}
> to avoid the broken case; I'm working against 2.3.4 where there's no such
> workaround.
> [Here's|https://stackoverflow.com/questions/51372788/array-cast-java-8-vs-java-9]
> a StackOverflow post that explains the issue in more detail. Interestingly,
> the author described the issue in the context of the HMS; not sure why there
> was no follow up with a Hive bug report.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-22126: Assignee: Laszlo Pinter > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Laszlo Pinter >Priority: Major > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21884: Attachment: HIVE-21844.07.patch > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21884.01.patch, > HIVE-21884.02.patch, HIVE-21884.03.patch, Scheduled queries2.pdf > > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22059) hive-exec jar doesn't contain (fasterxml) jackson library
[
https://issues.apache.org/jira/browse/HIVE-22059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-22059:
Attachment: HIVE-22059.03.patch
> hive-exec jar doesn't contain (fasterxml) jackson library
> -
>
> Key: HIVE-22059
> URL: https://issues.apache.org/jira/browse/HIVE-22059
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22059.01.patch, HIVE-22059.02.patch,
> HIVE-22059.03.patch
>
>
> While deploying master branch into a container I've noticed that the jackson
> libraries are not 100% sure that are available at runtime - this is probably
> due to the fact that we are still using the "old" codehaus jackson and also
> the "new" fasterxml one.
> {code:java}
> ]Vertex killed, vertexName=Reducer 2,
> vertexId=vertex_1564408646590_0005_1_01, diagnostics=[Vertex received Kill in
> INITED state., Vertex vertex_1564408646590_0005_1_01 [Reducer 2]
> killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to
> VERTEX_FAILURE. failedVertices:1 killedVertices:1
> INFO : Completed executing
> command(queryId=vagrant_20190729141949_8d8c7f0d-0ac4-4d76-ba12-6ec01561b040);
> Time taken: 5.127 seconds
> INFO : Concurrency mode is disabled, not creating a lock manager
> Error: Error while processing statement: FAILED: Execution Error, return code
> 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed,
> vertexName=Map 1, vertexId=vertex_1564408646590_0005_1_00,
> diagnostics=[Vertex vertex_1564408646590_0005_1_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: _dummy_table initializer failed,
> vertex=vertex_1564408646590_0005_1_00 [Map 1],
> java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
> at org.apache.hadoop.hive.ql.exec.Utilities.(Utilities.java:226)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:428)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:508)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:488)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:337)
> at
> org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:122)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassNotFoundException:
> com.fasterxml.jackson.databind.ObjectMapper
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 19 more
> ]Vertex killed, vertexName=Reducer 2,
> vertexId=vertex_1564408646590_0005_1_01, diagnostics=[Vertex received Kill in
> INITED state., Vertex vertex_1564408646590_0005_1_01 [Reducer 2]
> killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to
> VERTEX_FAILURE. failedVertices:1 killedVertices:1 (state=08S01,code=2)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22107) Correlated subquery producing wrong schema
[
https://issues.apache.org/jira/browse/HIVE-22107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925716#comment-16925716
]
Hankó Gergely commented on HIVE-22107:
--
Hi, [~vgarg], the file of the added test case
([subquery_notexists.q|https://github.com/apache/hive/pull/755/files#diff-386c2b9c0e9b16261791f06236dbd7cb])
is not added to any set in
itests/src/test/resources/testconfiguration.properties.
Is this intentional?
> Correlated subquery producing wrong schema
> --
>
> Key: HIVE-22107
> URL: https://issues.apache.org/jira/browse/HIVE-22107
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22107.1.patch, HIVE-22107.2.patch,
> HIVE-22107.3.patch, HIVE-22107.4.patch, HIVE-22107.5.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> *Repro*
> {code:sql}
> create table test(id int, name string,dept string);
> insert into test values(1,'a','it'),(2,'b','eee'),(NULL, 'c', 'cse');
> select distinct 'empno' as eid, a.id from test a where NOT EXISTS (select
> c.id from test c where a.id=c.id);
> {code}
> {code}
> +---++
> | eid | a.id |
> +---++
> | NULL | empno |
> +---++
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=308883=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308883
]
ASF GitHub Bot logged work on HIVE-20683:
-
Author: ASF GitHub Bot
Created on: 09/Sep/19 13:57
Start Date: 09/Sep/19 13:57
Worklog Time Spent: 10m
Work Description: nishantmonu51 commented on pull request #723:
[HIVE-20683] Add the Ability to push Dynamic Between and Bloom filters to Druid
URL: https://github.com/apache/hive/pull/723#discussion_r322256584
##
File path:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java
##
@@ -894,4 +945,255 @@ public static IndexSpec getIndexSpec(Configuration jc) {
ImmutableList aggregatorFactories =
aggregatorFactoryBuilder.build();
return Pair.of(dimensions, aggregatorFactories.toArray(new
AggregatorFactory[0]));
}
+
+ // Druid only supports String,Long,Float,Double selectors
+ private static Set druidSupportedTypeInfos =
ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo, TypeInfoFactory.charTypeInfo,
+ TypeInfoFactory.varcharTypeInfo, TypeInfoFactory.byteTypeInfo,
+ TypeInfoFactory.intTypeInfo, TypeInfoFactory.longTypeInfo,
+ TypeInfoFactory.shortTypeInfo, TypeInfoFactory.doubleTypeInfo
+ );
+
+ private static Set stringTypeInfos = ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo,
+ TypeInfoFactory.charTypeInfo, TypeInfoFactory.varcharTypeInfo
+ );
+
+
+ public static org.apache.druid.query.Query
addDynamicFilters(org.apache.druid.query.Query query,
+ ExprNodeGenericFuncDesc filterExpr, Configuration conf, boolean
resolveDynamicValues
+ ) {
+List virtualColumns = Arrays
+.asList(getVirtualColumns(query).getVirtualColumns());
+org.apache.druid.query.Query rv = query;
+DimFilter joinReductionFilter = toDruidFilter(filterExpr, conf,
virtualColumns,
+resolveDynamicValues
+);
+if(joinReductionFilter != null) {
+ String type = query.getType();
+ DimFilter filter = new AndDimFilter(joinReductionFilter,
query.getFilter());
+ switch (type) {
+ case org.apache.druid.query.Query.TIMESERIES:
+rv = Druids.TimeseriesQueryBuilder.copy((TimeseriesQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.TOPN:
+rv = new TopNQueryBuilder((TopNQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.GROUP_BY:
+rv = new GroupByQuery.Builder((GroupByQuery) query)
+.setDimFilter(filter)
+.setVirtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SCAN:
+rv = ScanQuery.ScanQueryBuilder.copy((ScanQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SELECT:
+rv = Druids.SelectQueryBuilder.copy((SelectQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ default:
+throw new UnsupportedOperationException("Unsupported Query type " +
type);
+ }
+}
+return rv;
+ }
+
+ @Nullable
+ private static DimFilter toDruidFilter(ExprNodeDesc filterExpr,
Configuration configuration,
+ List virtualColumns, boolean resolveDynamicValues
+ ) {
+if(filterExpr == null) {
+ return null;
+}
+Class genericUDFClass =
getGenericUDFClassFromExprDesc(filterExpr);
+if(FunctionRegistry.isOpAnd(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null && !delegates.isEmpty()) {
+return new AndDimFilter(delegates);
+ }
+}
+if(FunctionRegistry.isOpOr(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null) {
+return new OrDimFilter(delegates);
+ }
+} else
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=308884=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308884
]
ASF GitHub Bot logged work on HIVE-20683:
-
Author: ASF GitHub Bot
Created on: 09/Sep/19 13:57
Start Date: 09/Sep/19 13:57
Worklog Time Spent: 10m
Work Description: nishantmonu51 commented on issue #723: [HIVE-20683] Add
the Ability to push Dynamic Between and Bloom filters to Druid
URL: https://github.com/apache/hive/pull/723#issuecomment-529487533
@b-slim : Updated patch
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 308884)
Time Spent: 5h 10m (was: 5h)
> Add the Ability to push Dynamic Between and Bloom filters to Druid
> --
>
> Key: HIVE-20683
> URL: https://issues.apache.org/jira/browse/HIVE-20683
> Project: Hive
> Issue Type: New Feature
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20683.1.patch, HIVE-20683.10.patch,
> HIVE-20683.2.patch, HIVE-20683.3.patch, HIVE-20683.4.patch,
> HIVE-20683.5.patch, HIVE-20683.6.patch, HIVE-20683.8.patch, HIVE-20683.patch
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> For optimizing joins, Hive generates BETWEEN filter with min-max and BLOOM
> filter for filtering one side of semi-join.
> Druid 0.13.0 will have support for Bloom filters (Added via
> https://github.com/apache/incubator-druid/pull/6222)
> Implementation details -
> # Hive generates and passes the filters as part of 'filterExpr' in TableScan.
> # DruidQueryBasedRecordReader gets this filter passed as part of the conf.
> # During execution phase, before sending the query to druid in
> DruidQueryBasedRecordReader we will deserialize this filter, translate it
> into a DruidDimFilter and add it to existing DruidQuery. Tez executor
> already ensures that when we start reading results from the record reader,
> all the dynamic values are initialized.
> # Explaining a druid query also prints the query sent to druid as
> {{druid.json.query}}. We also need to make sure to update the druid query
> with the filters. During explain we do not have the actual values for the
> dynamic values, so instead of values we will print the dynamic expression
> itself as part of druid query.
> Note:- This work needs druid to be updated to version 0.13.0
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=308882=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308882
]
ASF GitHub Bot logged work on HIVE-20683:
-
Author: ASF GitHub Bot
Created on: 09/Sep/19 13:57
Start Date: 09/Sep/19 13:57
Worklog Time Spent: 10m
Work Description: nishantmonu51 commented on pull request #723:
[HIVE-20683] Add the Ability to push Dynamic Between and Bloom filters to Druid
URL: https://github.com/apache/hive/pull/723#discussion_r322256531
##
File path:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java
##
@@ -894,4 +945,255 @@ public static IndexSpec getIndexSpec(Configuration jc) {
ImmutableList aggregatorFactories =
aggregatorFactoryBuilder.build();
return Pair.of(dimensions, aggregatorFactories.toArray(new
AggregatorFactory[0]));
}
+
+ // Druid only supports String,Long,Float,Double selectors
+ private static Set druidSupportedTypeInfos =
ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo, TypeInfoFactory.charTypeInfo,
+ TypeInfoFactory.varcharTypeInfo, TypeInfoFactory.byteTypeInfo,
+ TypeInfoFactory.intTypeInfo, TypeInfoFactory.longTypeInfo,
+ TypeInfoFactory.shortTypeInfo, TypeInfoFactory.doubleTypeInfo
+ );
+
+ private static Set stringTypeInfos = ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo,
+ TypeInfoFactory.charTypeInfo, TypeInfoFactory.varcharTypeInfo
+ );
+
+
+ public static org.apache.druid.query.Query
addDynamicFilters(org.apache.druid.query.Query query,
+ ExprNodeGenericFuncDesc filterExpr, Configuration conf, boolean
resolveDynamicValues
+ ) {
+List virtualColumns = Arrays
+.asList(getVirtualColumns(query).getVirtualColumns());
+org.apache.druid.query.Query rv = query;
+DimFilter joinReductionFilter = toDruidFilter(filterExpr, conf,
virtualColumns,
+resolveDynamicValues
+);
+if(joinReductionFilter != null) {
+ String type = query.getType();
+ DimFilter filter = new AndDimFilter(joinReductionFilter,
query.getFilter());
+ switch (type) {
+ case org.apache.druid.query.Query.TIMESERIES:
+rv = Druids.TimeseriesQueryBuilder.copy((TimeseriesQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.TOPN:
+rv = new TopNQueryBuilder((TopNQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.GROUP_BY:
+rv = new GroupByQuery.Builder((GroupByQuery) query)
+.setDimFilter(filter)
+.setVirtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SCAN:
+rv = ScanQuery.ScanQueryBuilder.copy((ScanQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SELECT:
+rv = Druids.SelectQueryBuilder.copy((SelectQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ default:
+throw new UnsupportedOperationException("Unsupported Query type " +
type);
+ }
+}
+return rv;
+ }
+
+ @Nullable
+ private static DimFilter toDruidFilter(ExprNodeDesc filterExpr,
Configuration configuration,
+ List virtualColumns, boolean resolveDynamicValues
+ ) {
+if(filterExpr == null) {
+ return null;
+}
+Class genericUDFClass =
getGenericUDFClassFromExprDesc(filterExpr);
+if(FunctionRegistry.isOpAnd(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null && !delegates.isEmpty()) {
+return new AndDimFilter(delegates);
+ }
+}
+if(FunctionRegistry.isOpOr(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null) {
+return new OrDimFilter(delegates);
+ }
+} else
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=308881=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308881
]
ASF GitHub Bot logged work on HIVE-20683:
-
Author: ASF GitHub Bot
Created on: 09/Sep/19 13:57
Start Date: 09/Sep/19 13:57
Worklog Time Spent: 10m
Work Description: nishantmonu51 commented on pull request #723:
[HIVE-20683] Add the Ability to push Dynamic Between and Bloom filters to Druid
URL: https://github.com/apache/hive/pull/723#discussion_r322256384
##
File path:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java
##
@@ -91,17 +114,37 @@
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.NonSyncByteArrayInputStream;
Review comment:
removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 308881)
Time Spent: 4h 40m (was: 4.5h)
> Add the Ability to push Dynamic Between and Bloom filters to Druid
> --
>
> Key: HIVE-20683
> URL: https://issues.apache.org/jira/browse/HIVE-20683
> Project: Hive
> Issue Type: New Feature
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20683.1.patch, HIVE-20683.10.patch,
> HIVE-20683.2.patch, HIVE-20683.3.patch, HIVE-20683.4.patch,
> HIVE-20683.5.patch, HIVE-20683.6.patch, HIVE-20683.8.patch, HIVE-20683.patch
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> For optimizing joins, Hive generates BETWEEN filter with min-max and BLOOM
> filter for filtering one side of semi-join.
> Druid 0.13.0 will have support for Bloom filters (Added via
> https://github.com/apache/incubator-druid/pull/6222)
> Implementation details -
> # Hive generates and passes the filters as part of 'filterExpr' in TableScan.
> # DruidQueryBasedRecordReader gets this filter passed as part of the conf.
> # During execution phase, before sending the query to druid in
> DruidQueryBasedRecordReader we will deserialize this filter, translate it
> into a DruidDimFilter and add it to existing DruidQuery. Tez executor
> already ensures that when we start reading results from the record reader,
> all the dynamic values are initialized.
> # Explaining a druid query also prints the query sent to druid as
> {{druid.json.query}}. We also need to make sure to update the druid query
> with the filters. During explain we do not have the actual values for the
> dynamic values, so instead of values we will print the dynamic expression
> itself as part of druid query.
> Note:- This work needs druid to be updated to version 0.13.0
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=308880=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308880
]
ASF GitHub Bot logged work on HIVE-20683:
-
Author: ASF GitHub Bot
Created on: 09/Sep/19 13:57
Start Date: 09/Sep/19 13:57
Worklog Time Spent: 10m
Work Description: nishantmonu51 commented on pull request #723:
[HIVE-20683] Add the Ability to push Dynamic Between and Bloom filters to Druid
URL: https://github.com/apache/hive/pull/723#discussion_r322256307
##
File path:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java
##
@@ -894,4 +945,255 @@ public static IndexSpec getIndexSpec(Configuration jc) {
ImmutableList aggregatorFactories =
aggregatorFactoryBuilder.build();
return Pair.of(dimensions, aggregatorFactories.toArray(new
AggregatorFactory[0]));
}
+
+ // Druid only supports String,Long,Float,Double selectors
+ private static Set druidSupportedTypeInfos =
ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo, TypeInfoFactory.charTypeInfo,
+ TypeInfoFactory.varcharTypeInfo, TypeInfoFactory.byteTypeInfo,
+ TypeInfoFactory.intTypeInfo, TypeInfoFactory.longTypeInfo,
+ TypeInfoFactory.shortTypeInfo, TypeInfoFactory.doubleTypeInfo
+ );
+
+ private static Set stringTypeInfos = ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo,
+ TypeInfoFactory.charTypeInfo, TypeInfoFactory.varcharTypeInfo
+ );
+
+
+ public static org.apache.druid.query.Query
addDynamicFilters(org.apache.druid.query.Query query,
+ ExprNodeGenericFuncDesc filterExpr, Configuration conf, boolean
resolveDynamicValues
+ ) {
+List virtualColumns = Arrays
+.asList(getVirtualColumns(query).getVirtualColumns());
+org.apache.druid.query.Query rv = query;
+DimFilter joinReductionFilter = toDruidFilter(filterExpr, conf,
virtualColumns,
+resolveDynamicValues
+);
+if(joinReductionFilter != null) {
+ String type = query.getType();
+ DimFilter filter = new AndDimFilter(joinReductionFilter,
query.getFilter());
+ switch (type) {
+ case org.apache.druid.query.Query.TIMESERIES:
+rv = Druids.TimeseriesQueryBuilder.copy((TimeseriesQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.TOPN:
+rv = new TopNQueryBuilder((TopNQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.GROUP_BY:
+rv = new GroupByQuery.Builder((GroupByQuery) query)
+.setDimFilter(filter)
+.setVirtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SCAN:
+rv = ScanQuery.ScanQueryBuilder.copy((ScanQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SELECT:
+rv = Druids.SelectQueryBuilder.copy((SelectQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ default:
+throw new UnsupportedOperationException("Unsupported Query type " +
type);
+ }
+}
+return rv;
+ }
+
+ @Nullable
+ private static DimFilter toDruidFilter(ExprNodeDesc filterExpr,
Configuration configuration,
+ List virtualColumns, boolean resolveDynamicValues
+ ) {
+if(filterExpr == null) {
+ return null;
+}
+Class genericUDFClass =
getGenericUDFClassFromExprDesc(filterExpr);
+if(FunctionRegistry.isOpAnd(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null && !delegates.isEmpty()) {
Review comment:
removed null check
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 308880)
Time Spent: 4.5h (was: 4h 20m)
> Add the Ability to push Dynamic Between and Bloom filters to Druid
>
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=308878=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308878
]
ASF GitHub Bot logged work on HIVE-20683:
-
Author: ASF GitHub Bot
Created on: 09/Sep/19 13:56
Start Date: 09/Sep/19 13:56
Worklog Time Spent: 10m
Work Description: nishantmonu51 commented on pull request #723:
[HIVE-20683] Add the Ability to push Dynamic Between and Bloom filters to Druid
URL: https://github.com/apache/hive/pull/723#discussion_r322256133
##
File path:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java
##
@@ -894,4 +945,255 @@ public static IndexSpec getIndexSpec(Configuration jc) {
ImmutableList aggregatorFactories =
aggregatorFactoryBuilder.build();
return Pair.of(dimensions, aggregatorFactories.toArray(new
AggregatorFactory[0]));
}
+
+ // Druid only supports String,Long,Float,Double selectors
+ private static Set druidSupportedTypeInfos =
ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo, TypeInfoFactory.charTypeInfo,
+ TypeInfoFactory.varcharTypeInfo, TypeInfoFactory.byteTypeInfo,
+ TypeInfoFactory.intTypeInfo, TypeInfoFactory.longTypeInfo,
+ TypeInfoFactory.shortTypeInfo, TypeInfoFactory.doubleTypeInfo
+ );
+
+ private static Set stringTypeInfos = ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo,
+ TypeInfoFactory.charTypeInfo, TypeInfoFactory.varcharTypeInfo
+ );
+
+
+ public static org.apache.druid.query.Query
addDynamicFilters(org.apache.druid.query.Query query,
+ ExprNodeGenericFuncDesc filterExpr, Configuration conf, boolean
resolveDynamicValues
+ ) {
+List virtualColumns = Arrays
+.asList(getVirtualColumns(query).getVirtualColumns());
+org.apache.druid.query.Query rv = query;
+DimFilter joinReductionFilter = toDruidFilter(filterExpr, conf,
virtualColumns,
+resolveDynamicValues
+);
+if(joinReductionFilter != null) {
+ String type = query.getType();
+ DimFilter filter = new AndDimFilter(joinReductionFilter,
query.getFilter());
+ switch (type) {
+ case org.apache.druid.query.Query.TIMESERIES:
+rv = Druids.TimeseriesQueryBuilder.copy((TimeseriesQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.TOPN:
+rv = new TopNQueryBuilder((TopNQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.GROUP_BY:
+rv = new GroupByQuery.Builder((GroupByQuery) query)
+.setDimFilter(filter)
+.setVirtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SCAN:
+rv = ScanQuery.ScanQueryBuilder.copy((ScanQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SELECT:
+rv = Druids.SelectQueryBuilder.copy((SelectQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ default:
+throw new UnsupportedOperationException("Unsupported Query type " +
type);
+ }
+}
+return rv;
+ }
+
+ @Nullable
+ private static DimFilter toDruidFilter(ExprNodeDesc filterExpr,
Configuration configuration,
+ List virtualColumns, boolean resolveDynamicValues
+ ) {
+if(filterExpr == null) {
+ return null;
+}
+Class genericUDFClass =
getGenericUDFClassFromExprDesc(filterExpr);
+if(FunctionRegistry.isOpAnd(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null && !delegates.isEmpty()) {
+return new AndDimFilter(delegates);
+ }
+}
+if(FunctionRegistry.isOpOr(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null) {
+return new OrDimFilter(delegates);
+ }
+} else
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[ https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=308876=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308876 ] ASF GitHub Bot logged work on HIVE-20683: - Author: ASF GitHub Bot Created on: 09/Sep/19 13:56 Start Date: 09/Sep/19 13:56 Worklog Time Spent: 10m Work Description: nishantmonu51 commented on pull request #723: [HIVE-20683] Add the Ability to push Dynamic Between and Bloom filters to Druid URL: https://github.com/apache/hive/pull/723#discussion_r322256014 ## File path: ql/src/test/queries/clientpositive/druidmini_semijoin_reduction_all_types.q ## @@ -0,0 +1,144 @@ +--! qt:dataset:srcpart +--! qt:dataset:druid_table_alltypesorc +--! qt:dataset:alltypesorc + +set hive.compute.query.using.stats=false; +set hive.mapred.mode=nonstrict; +set hive.explain.user=false; +set hive.optimize.ppd=true; +set hive.ppd.remove.duplicatefilters=true; +set hive.tez.dynamic.partition.pruning=true; +set hive.tez.dynamic.semijoin.reduction=true; +set hive.optimize.metadataonly=false; +set hive.optimize.index.filter=true; +set hive.stats.autogather=true; +set hive.tez.bigtable.minsize.semijoin.reduction=1; +set hive.tez.min.bloom.filter.entries=1; +set hive.stats.fetch.column.stats=true; +set hive.disable.unsafe.external.table.operations=false; +set hive.tez.dynamic.semijoin.reduction.for.mapjoin=true; + +DROP TABLE IF EXISTS alltypesorc_small; +CREATE TABLE alltypesorc_small( +ctinyint TINYINT, +csmallint SMALLINT, +cint INT, +cbigint BIGINT, +cfloat FLOAT, +cdouble DOUBLE, +cstring1 STRING, +cstring2 STRING, +ctimestamp1 TIMESTAMP, +cboolean1 BOOLEAN, +cboolean2 BOOLEAN) +STORED AS ORC; +Insert into table alltypesorc_small +Select ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, cast(`__time` as timestamp), cboolean1, cboolean2 from druid_table_alltypesorc where cstring2 like '%a%' and cstring1 like '%a%'; +Select count(*) from alltypesorc_small; +Select count(*) from druid_table_alltypesorc; + +DESCRIBE druid_table_alltypesorc; +DESCRIBE alltypesorc_small; + +-- Test Joins on all column types one by one +-- String +set hive.disable.unsafe.external.table.operations=false; +EXPLAIN select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cstring1 = druid_table_alltypesorc.cstring1); +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cstring1 = druid_table_alltypesorc.cstring1); +set hive.disable.unsafe.external.table.operations=true; +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cstring1 = druid_table_alltypesorc.cstring1); + +-- tinyint +set hive.disable.unsafe.external.table.operations=false; +EXPLAIN select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.ctinyint = druid_table_alltypesorc.ctinyint); +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.ctinyint = druid_table_alltypesorc.ctinyint); +set hive.disable.unsafe.external.table.operations=true; +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.ctinyint = druid_table_alltypesorc.ctinyint); + +-- smallint +set hive.disable.unsafe.external.table.operations=false; +EXPLAIN select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.csmallint = druid_table_alltypesorc.csmallint); +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.csmallint = druid_table_alltypesorc.csmallint); +set hive.disable.unsafe.external.table.operations=true; +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.csmallint = druid_table_alltypesorc.csmallint); + +-- int +set hive.disable.unsafe.external.table.operations=false; +EXPLAIN select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cint = druid_table_alltypesorc.cint); +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cint = druid_table_alltypesorc.cint); +set hive.disable.unsafe.external.table.operations=true; +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cint = druid_table_alltypesorc.cint); + +-- bigint +set hive.disable.unsafe.external.table.operations=false; +EXPLAIN select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cbigint = druid_table_alltypesorc.cbigint); +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cbigint = druid_table_alltypesorc.cbigint); +set hive.disable.unsafe.external.table.operations=true; +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cbigint =
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=308879=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308879
]
ASF GitHub Bot logged work on HIVE-20683:
-
Author: ASF GitHub Bot
Created on: 09/Sep/19 13:56
Start Date: 09/Sep/19 13:56
Worklog Time Spent: 10m
Work Description: nishantmonu51 commented on pull request #723:
[HIVE-20683] Add the Ability to push Dynamic Between and Bloom filters to Druid
URL: https://github.com/apache/hive/pull/723#discussion_r322256224
##
File path:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java
##
@@ -894,4 +945,255 @@ public static IndexSpec getIndexSpec(Configuration jc) {
ImmutableList aggregatorFactories =
aggregatorFactoryBuilder.build();
return Pair.of(dimensions, aggregatorFactories.toArray(new
AggregatorFactory[0]));
}
+
+ // Druid only supports String,Long,Float,Double selectors
+ private static Set druidSupportedTypeInfos =
ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo, TypeInfoFactory.charTypeInfo,
+ TypeInfoFactory.varcharTypeInfo, TypeInfoFactory.byteTypeInfo,
+ TypeInfoFactory.intTypeInfo, TypeInfoFactory.longTypeInfo,
+ TypeInfoFactory.shortTypeInfo, TypeInfoFactory.doubleTypeInfo
+ );
+
+ private static Set stringTypeInfos = ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo,
+ TypeInfoFactory.charTypeInfo, TypeInfoFactory.varcharTypeInfo
+ );
+
+
+ public static org.apache.druid.query.Query
addDynamicFilters(org.apache.druid.query.Query query,
+ ExprNodeGenericFuncDesc filterExpr, Configuration conf, boolean
resolveDynamicValues
+ ) {
+List virtualColumns = Arrays
+.asList(getVirtualColumns(query).getVirtualColumns());
+org.apache.druid.query.Query rv = query;
+DimFilter joinReductionFilter = toDruidFilter(filterExpr, conf,
virtualColumns,
+resolveDynamicValues
+);
+if(joinReductionFilter != null) {
+ String type = query.getType();
+ DimFilter filter = new AndDimFilter(joinReductionFilter,
query.getFilter());
+ switch (type) {
+ case org.apache.druid.query.Query.TIMESERIES:
+rv = Druids.TimeseriesQueryBuilder.copy((TimeseriesQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.TOPN:
+rv = new TopNQueryBuilder((TopNQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.GROUP_BY:
+rv = new GroupByQuery.Builder((GroupByQuery) query)
+.setDimFilter(filter)
+.setVirtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SCAN:
+rv = ScanQuery.ScanQueryBuilder.copy((ScanQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SELECT:
+rv = Druids.SelectQueryBuilder.copy((SelectQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ default:
+throw new UnsupportedOperationException("Unsupported Query type " +
type);
+ }
+}
+return rv;
+ }
+
+ @Nullable
+ private static DimFilter toDruidFilter(ExprNodeDesc filterExpr,
Configuration configuration,
+ List virtualColumns, boolean resolveDynamicValues
+ ) {
+if(filterExpr == null) {
+ return null;
+}
+Class genericUDFClass =
getGenericUDFClassFromExprDesc(filterExpr);
+if(FunctionRegistry.isOpAnd(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null && !delegates.isEmpty()) {
+return new AndDimFilter(delegates);
+ }
+}
+if(FunctionRegistry.isOpOr(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null) {
Review comment:
fixed.
[jira] [Updated] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-20683:
Attachment: HIVE-20683.10.patch
> Add the Ability to push Dynamic Between and Bloom filters to Druid
> --
>
> Key: HIVE-20683
> URL: https://issues.apache.org/jira/browse/HIVE-20683
> Project: Hive
> Issue Type: New Feature
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20683.1.patch, HIVE-20683.10.patch,
> HIVE-20683.2.patch, HIVE-20683.3.patch, HIVE-20683.4.patch,
> HIVE-20683.5.patch, HIVE-20683.6.patch, HIVE-20683.8.patch, HIVE-20683.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> For optimizing joins, Hive generates BETWEEN filter with min-max and BLOOM
> filter for filtering one side of semi-join.
> Druid 0.13.0 will have support for Bloom filters (Added via
> https://github.com/apache/incubator-druid/pull/6222)
> Implementation details -
> # Hive generates and passes the filters as part of 'filterExpr' in TableScan.
> # DruidQueryBasedRecordReader gets this filter passed as part of the conf.
> # During execution phase, before sending the query to druid in
> DruidQueryBasedRecordReader we will deserialize this filter, translate it
> into a DruidDimFilter and add it to existing DruidQuery. Tez executor
> already ensures that when we start reading results from the record reader,
> all the dynamic values are initialized.
> # Explaining a druid query also prints the query sent to druid as
> {{druid.json.query}}. We also need to make sure to update the druid query
> with the filters. During explain we do not have the actual values for the
> dynamic values, so instead of values we will print the dynamic expression
> itself as part of druid query.
> Note:- This work needs druid to be updated to version 0.13.0
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=308877=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308877
]
ASF GitHub Bot logged work on HIVE-20683:
-
Author: ASF GitHub Bot
Created on: 09/Sep/19 13:56
Start Date: 09/Sep/19 13:56
Worklog Time Spent: 10m
Work Description: nishantmonu51 commented on pull request #723:
[HIVE-20683] Add the Ability to push Dynamic Between and Bloom filters to Druid
URL: https://github.com/apache/hive/pull/723#discussion_r322256075
##
File path:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java
##
@@ -894,4 +945,255 @@ public static IndexSpec getIndexSpec(Configuration jc) {
ImmutableList aggregatorFactories =
aggregatorFactoryBuilder.build();
return Pair.of(dimensions, aggregatorFactories.toArray(new
AggregatorFactory[0]));
}
+
+ // Druid only supports String,Long,Float,Double selectors
+ private static Set druidSupportedTypeInfos =
ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo, TypeInfoFactory.charTypeInfo,
+ TypeInfoFactory.varcharTypeInfo, TypeInfoFactory.byteTypeInfo,
+ TypeInfoFactory.intTypeInfo, TypeInfoFactory.longTypeInfo,
+ TypeInfoFactory.shortTypeInfo, TypeInfoFactory.doubleTypeInfo
+ );
+
+ private static Set stringTypeInfos = ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo,
+ TypeInfoFactory.charTypeInfo, TypeInfoFactory.varcharTypeInfo
+ );
+
+
+ public static org.apache.druid.query.Query
addDynamicFilters(org.apache.druid.query.Query query,
+ ExprNodeGenericFuncDesc filterExpr, Configuration conf, boolean
resolveDynamicValues
+ ) {
+List virtualColumns = Arrays
+.asList(getVirtualColumns(query).getVirtualColumns());
+org.apache.druid.query.Query rv = query;
+DimFilter joinReductionFilter = toDruidFilter(filterExpr, conf,
virtualColumns,
+resolveDynamicValues
+);
+if(joinReductionFilter != null) {
+ String type = query.getType();
+ DimFilter filter = new AndDimFilter(joinReductionFilter,
query.getFilter());
+ switch (type) {
+ case org.apache.druid.query.Query.TIMESERIES:
+rv = Druids.TimeseriesQueryBuilder.copy((TimeseriesQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.TOPN:
+rv = new TopNQueryBuilder((TopNQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.GROUP_BY:
+rv = new GroupByQuery.Builder((GroupByQuery) query)
+.setDimFilter(filter)
+.setVirtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SCAN:
+rv = ScanQuery.ScanQueryBuilder.copy((ScanQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SELECT:
+rv = Druids.SelectQueryBuilder.copy((SelectQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ default:
+throw new UnsupportedOperationException("Unsupported Query type " +
type);
+ }
+}
+return rv;
+ }
+
+ @Nullable
+ private static DimFilter toDruidFilter(ExprNodeDesc filterExpr,
Configuration configuration,
+ List virtualColumns, boolean resolveDynamicValues
+ ) {
+if(filterExpr == null) {
+ return null;
+}
+Class genericUDFClass =
getGenericUDFClassFromExprDesc(filterExpr);
+if(FunctionRegistry.isOpAnd(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null && !delegates.isEmpty()) {
+return new AndDimFilter(delegates);
+ }
+}
+if(FunctionRegistry.isOpOr(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null) {
+return new OrDimFilter(delegates);
+ }
+} else
[jira] [Updated] (HIVE-22055) select count gives incorrect result after loading data from text file
[
https://issues.apache.org/jira/browse/HIVE-22055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated HIVE-22055:
-
Status: Patch Available (was: Open)
> select count gives incorrect result after loading data from text file
> -
>
> Key: HIVE-22055
> URL: https://issues.apache.org/jira/browse/HIVE-22055
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22055.2.patch
>
>
> Add one more load to mm_loaddata.q:
> Load data 3 times (both kv1.txt and kv2.txt contains 500 records)
> {code:java}
> create table load0_mm (key string, value string) stored as textfile
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> load data local inpath '../../data/files/kv1.txt' into table load0_mm;
> select count(1) from load0_mm;
> load data local inpath '../../data/files/kv2.txt' into table load0_mm;
> select count(1) from load0_mm;
> load data local inpath '../../data/files/kv2.txt' into table load0_mm;
> select count(1) from load0_mm;{code}
> Expected output
> {code:java}
> PREHOOK: query: load data local inpath '../../data/files/kv2.txt' into table
> load0_mm
> PREHOOK: type: LOAD
> A masked pattern was here
> PREHOOK: Output: default@load0_mm
> POSTHOOK: query: load data local inpath '../../data/files/kv2.txt' into table
> load0_mm
> POSTHOOK: type: LOAD
> A masked pattern was here
> POSTHOOK: Output: default@load0_mm
> PREHOOK: query: select count(1) from load0_mm
> PREHOOK: type: QUERY
> PREHOOK: Input: default@load0_mm
> A masked pattern was here
> POSTHOOK: query: select count(1) from load0_mm
> POSTHOOK: type: QUERY
> POSTHOOK: Input: default@load0_mm
> A masked pattern was here
> 1500{code}
> Got:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client Execution
> succeeded but contained differences (error code = 1) after executing
> mm_loaddata.q
> 63c63
> < 1480
> —
> > 1500
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22055) select count gives incorrect result after loading data from text file
[
https://issues.apache.org/jira/browse/HIVE-22055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated HIVE-22055:
-
Attachment: (was: HIVE-22055.1.patch)
> select count gives incorrect result after loading data from text file
> -
>
> Key: HIVE-22055
> URL: https://issues.apache.org/jira/browse/HIVE-22055
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22055.2.patch
>
>
> Add one more load to mm_loaddata.q:
> Load data 3 times (both kv1.txt and kv2.txt contains 500 records)
> {code:java}
> create table load0_mm (key string, value string) stored as textfile
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> load data local inpath '../../data/files/kv1.txt' into table load0_mm;
> select count(1) from load0_mm;
> load data local inpath '../../data/files/kv2.txt' into table load0_mm;
> select count(1) from load0_mm;
> load data local inpath '../../data/files/kv2.txt' into table load0_mm;
> select count(1) from load0_mm;{code}
> Expected output
> {code:java}
> PREHOOK: query: load data local inpath '../../data/files/kv2.txt' into table
> load0_mm
> PREHOOK: type: LOAD
> A masked pattern was here
> PREHOOK: Output: default@load0_mm
> POSTHOOK: query: load data local inpath '../../data/files/kv2.txt' into table
> load0_mm
> POSTHOOK: type: LOAD
> A masked pattern was here
> POSTHOOK: Output: default@load0_mm
> PREHOOK: query: select count(1) from load0_mm
> PREHOOK: type: QUERY
> PREHOOK: Input: default@load0_mm
> A masked pattern was here
> POSTHOOK: query: select count(1) from load0_mm
> POSTHOOK: type: QUERY
> POSTHOOK: Input: default@load0_mm
> A masked pattern was here
> 1500{code}
> Got:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client Execution
> succeeded but contained differences (error code = 1) after executing
> mm_loaddata.q
> 63c63
> < 1480
> —
> > 1500
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22055) select count gives incorrect result after loading data from text file
[
https://issues.apache.org/jira/browse/HIVE-22055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated HIVE-22055:
-
Attachment: HIVE-22055.2.patch
> select count gives incorrect result after loading data from text file
> -
>
> Key: HIVE-22055
> URL: https://issues.apache.org/jira/browse/HIVE-22055
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22055.2.patch
>
>
> Add one more load to mm_loaddata.q:
> Load data 3 times (both kv1.txt and kv2.txt contains 500 records)
> {code:java}
> create table load0_mm (key string, value string) stored as textfile
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> load data local inpath '../../data/files/kv1.txt' into table load0_mm;
> select count(1) from load0_mm;
> load data local inpath '../../data/files/kv2.txt' into table load0_mm;
> select count(1) from load0_mm;
> load data local inpath '../../data/files/kv2.txt' into table load0_mm;
> select count(1) from load0_mm;{code}
> Expected output
> {code:java}
> PREHOOK: query: load data local inpath '../../data/files/kv2.txt' into table
> load0_mm
> PREHOOK: type: LOAD
> A masked pattern was here
> PREHOOK: Output: default@load0_mm
> POSTHOOK: query: load data local inpath '../../data/files/kv2.txt' into table
> load0_mm
> POSTHOOK: type: LOAD
> A masked pattern was here
> POSTHOOK: Output: default@load0_mm
> PREHOOK: query: select count(1) from load0_mm
> PREHOOK: type: QUERY
> PREHOOK: Input: default@load0_mm
> A masked pattern was here
> POSTHOOK: query: select count(1) from load0_mm
> POSTHOOK: type: QUERY
> POSTHOOK: Input: default@load0_mm
> A masked pattern was here
> 1500{code}
> Got:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client Execution
> succeeded but contained differences (error code = 1) after executing
> mm_loaddata.q
> 63c63
> < 1480
> —
> > 1500
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22055) select count gives incorrect result after loading data from text file
[
https://issues.apache.org/jira/browse/HIVE-22055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated HIVE-22055:
-
Status: Open (was: Patch Available)
> select count gives incorrect result after loading data from text file
> -
>
> Key: HIVE-22055
> URL: https://issues.apache.org/jira/browse/HIVE-22055
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22055.1.patch
>
>
> Add one more load to mm_loaddata.q:
> Load data 3 times (both kv1.txt and kv2.txt contains 500 records)
> {code:java}
> create table load0_mm (key string, value string) stored as textfile
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> load data local inpath '../../data/files/kv1.txt' into table load0_mm;
> select count(1) from load0_mm;
> load data local inpath '../../data/files/kv2.txt' into table load0_mm;
> select count(1) from load0_mm;
> load data local inpath '../../data/files/kv2.txt' into table load0_mm;
> select count(1) from load0_mm;{code}
> Expected output
> {code:java}
> PREHOOK: query: load data local inpath '../../data/files/kv2.txt' into table
> load0_mm
> PREHOOK: type: LOAD
> A masked pattern was here
> PREHOOK: Output: default@load0_mm
> POSTHOOK: query: load data local inpath '../../data/files/kv2.txt' into table
> load0_mm
> POSTHOOK: type: LOAD
> A masked pattern was here
> POSTHOOK: Output: default@load0_mm
> PREHOOK: query: select count(1) from load0_mm
> PREHOOK: type: QUERY
> PREHOOK: Input: default@load0_mm
> A masked pattern was here
> POSTHOOK: query: select count(1) from load0_mm
> POSTHOOK: type: QUERY
> POSTHOOK: Input: default@load0_mm
> A masked pattern was here
> 1500{code}
> Got:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client Execution
> succeeded but contained differences (error code = 1) after executing
> mm_loaddata.q
> 63c63
> < 1480
> —
> > 1500
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-21508) ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
[
https://issues.apache.org/jira/browse/HIVE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925699#comment-16925699
]
Ana Jalba commented on HIVE-21508:
--
Local tests pass for hive-cli on branch-2.3 (with the change made) so I'm not
sure what's happening with that build.
I'll wait before I add another patch. Maybe it's better to upload a patch, wait
for results, merge and then upload another patch. Do you think that's
reasonable, [~pvary]?
> ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
> --
>
> Key: HIVE-21508
> URL: https://issues.apache.org/jira/browse/HIVE-21508
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 3.2.0, 2.3.4
>Reporter: Adar Dembo
>Assignee: Ana Jalba
>Priority: Major
> Fix For: 4.0.0, 2.3.7
>
> Attachments: HIVE-21508.1.branch-2.3.patch, HIVE-21508.patch
>
>
> There's this block of code in {{HiveMetaStoreClient:resolveUris}} (called
> from the constructor) on master:
> {noformat}
> private URI metastoreUris[];
> ...
> if (MetastoreConf.getVar(conf,
> ConfVars.THRIFT_URI_SELECTION).equalsIgnoreCase("RANDOM")) {
> List uriList = Arrays.asList(metastoreUris);
> Collections.shuffle(uriList);
> metastoreUris = (URI[]) uriList.toArray();
> }
> {noformat}
> The cast to {{URI[]}} throws a {{ClassCastException}} beginning with JDK 10,
> possibly with JDK 9 as well. Note that {{THRIFT_URI_SELECTION}} defaults to
> {{RANDOM}} so this should affect anyone who creates a
> {{HiveMetaStoreClient}}. On master this can be overridden with {{SEQUENTIAL}}
> to avoid the broken case; I'm working against 2.3.4 where there's no such
> workaround.
> [Here's|https://stackoverflow.com/questions/51372788/array-cast-java-8-vs-java-9]
> a StackOverflow post that explains the issue in more detail. Interestingly,
> the author described the issue in the context of the HMS; not sure why there
> was no follow up with a Hive bug report.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[
https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925696#comment-16925696
]
Hive QA commented on HIVE-21604:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12979841/files.txt
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18506/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18506/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18506/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-09-09 13:36:25.459
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-18506/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-09-09 13:36:25.461
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
6dbc115..9667f25 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 6dbc115 HIVE-21397: BloomFilter for hive Managed [ACID] table
does not work as expected (Denys Kuzmenko, reviewed by Gopal Vijayaraghavan and
Adam Szita)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 9667f25 HIVE-21942: Remove useless MetastoreType enum from
AbstractCliConfig (Imre Molnar via Laszlo Bodor)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-09-09 13:36:27.016
+ rm -rf ../yetus_PreCommit-HIVE-Build-18506
+ mkdir ../yetus_PreCommit-HIVE-Build-18506
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-18506
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-18506/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: unrecognized input
fatal: unrecognized input
fatal: unrecognized input
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-18506
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12979841 - PreCommit-HIVE-Build
> preCommit job should not be triggered on non-patch attachments
> --
>
> Key: HIVE-21604
> URL: https://issues.apache.org/jira/browse/HIVE-21604
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21604.01.patch, files.txt
>
>
> latest example: HIVE-21604
> https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520
> I think one should be able to upload any kind of attachments (e.g.
> screenshot) without triggering the precommit job
> 2 possible ways:
> 1. strict: enable only .patch (should work)
> (2. lenient: introduce blacklist, .png ...)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925692#comment-16925692
]
Hive QA commented on HIVE-21884:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
51s{color} | {color:blue} standalone-metastore/metastore-common in master has
32 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} common in master has 61 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
54s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
3s{color} | {color:blue} ql in master has 2246 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} itests/util in master has 44 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 1 new + 495 unchanged - 0 fixed = 496 total (was 495) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} common: The patch generated 2 new + 367 unchanged - 0
fixed = 369 total (was 367) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
5s{color} | {color:green} The patch service-rpc passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} The patch metastore passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 18 new + 1250 unchanged - 15 fixed = 1268 total (was 1265) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
33s{color} | {color:red} ql: The patch generated 61 new + 224 unchanged - 0
fixed = 285 total (was 224) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} service: The patch generated 0 new + 38 unchanged -
1 fixed = 38 total (was 39) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
48s{color} | {color:red} root: The patch generated 82 new + 2421 unchanged - 16
fixed = 2503 total (was 2437) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} The patch hcatalog-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch util passed checkstyle {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 17 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
|
[jira] [Commented] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925683#comment-16925683
]
Hive QA commented on HIVE-21884:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12979839/HIVE-21844.06.patch
{color:green}SUCCESS:{color} +1 due to 9 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 691 failed/errored test(s), 16784 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_materialized_view_rewrite_ssb]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timeseries]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz2]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_topn]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_dynamic_partition]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_extractTime]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_floorTime]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_alter]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_ts]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_avro]
(batchId=297)
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_basic]
(batchId=297)
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_csv]
(batchId=297)
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_delimited]
(batchId=297)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[table_nonprintable]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[temp_table_external]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[strict_managed_tables_sysdb]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb_schq]
(batchId=176)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_schq1]
(batchId=102)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=104)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ctas_noperm_loc]
(batchId=104)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[table_nonprintable_negative]
(batchId=104)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[add_part_multiple]
(batchId=145)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[alter_merge_orc]
(batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[alter_merge_stats_orc]
(batchId=131)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[annotate_stats_join]
(batchId=137)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join0]
(batchId=153)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join10]
(batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join11]
(batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join12]
(batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join13]
(batchId=150)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join14]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join15]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join16]
(batchId=131)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join17]
(batchId=151)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join18]
(batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join18_multi_distinct]
(batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join19]
(batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join1]
(batchId=149)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join20]
(batchId=154)
[jira] [Updated] (HIVE-22083) Values of tag order cannot be null, so it can be "byte" instead of "Byte"
[ https://issues.apache.org/jira/browse/HIVE-22083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-22083: --- Attachment: HIVE-22083.4.patch > Values of tag order cannot be null, so it can be "byte" instead of "Byte" > - > > Key: HIVE-22083 > URL: https://issues.apache.org/jira/browse/HIVE-22083 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-22083.1.patch, HIVE-22083.2.patch, > HIVE-22083.3.patch, HIVE-22083.4.patch, HIVE-22083.4.patch, HIVE-22083.4.patch > > > Values of tag order cannot be null, so it can be "byte" instead of "Byte". > Switching between Byte and byte is "cheap" - the Byte objects are cached by > the JVM - but it still costs a bit more memory and CPU usage. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-21942) Remove useless MetastoreType enum from AbstractCliConfig
[ https://issues.apache.org/jira/browse/HIVE-21942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925679#comment-16925679 ] Laszlo Bodor commented on HIVE-21942: - pushed to master, thanks [~imolnar] for the patch! > Remove useless MetastoreType enum from AbstractCliConfig > > > Key: HIVE-21942 > URL: https://issues.apache.org/jira/browse/HIVE-21942 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Assignee: Imre Molnar >Priority: Major > Labels: newbie > Fix For: 4.0.0 > > Attachments: HIVE-21942.1.patch > > > I'm not 100% aware of historical reasons, but this single-value enum seems > totally useless at the moment. > https://github.com/apache/hive/blob/86a15600ae610e1c6017019883ef9344585572ef/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/AbstractCliConfig.java#L50-L52 > According to HIVE-14854, it may have made sense when there was an initiative > of hbase metastore, but it's not in the scope anymore AFAIK. > https://github.com/apache/hive/commit/474425aa62e3f25b119419439373aa684c6c2121 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-21942) Remove useless MetastoreType enum from AbstractCliConfig
[ https://issues.apache.org/jira/browse/HIVE-21942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21942: Resolution: Fixed Status: Resolved (was: Patch Available) > Remove useless MetastoreType enum from AbstractCliConfig > > > Key: HIVE-21942 > URL: https://issues.apache.org/jira/browse/HIVE-21942 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Assignee: Imre Molnar >Priority: Major > Labels: newbie > Fix For: 4.0.0 > > Attachments: HIVE-21942.1.patch > > > I'm not 100% aware of historical reasons, but this single-value enum seems > totally useless at the moment. > https://github.com/apache/hive/blob/86a15600ae610e1c6017019883ef9344585572ef/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/AbstractCliConfig.java#L50-L52 > According to HIVE-14854, it may have made sense when there was an initiative > of hbase metastore, but it's not in the scope anymore AFAIK. > https://github.com/apache/hive/commit/474425aa62e3f25b119419439373aa684c6c2121 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-21397) BloomFilter for hive Managed [ACID] table does not work as expected
[
https://issues.apache.org/jira/browse/HIVE-21397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925633#comment-16925633
]
Denys Kuzmenko commented on HIVE-21397:
---
Thank you, [~szita]!
> BloomFilter for hive Managed [ACID] table does not work as expected
> ---
>
> Key: HIVE-21397
> URL: https://issues.apache.org/jira/browse/HIVE-21397
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Transactions
>Affects Versions: 3.1.1
>Reporter: vaibhav
>Assignee: Denys Kuzmenko
>Priority: Blocker
> Fix For: 4.0.0
>
> Attachments: HIVE-21397.1.patch, HIVE-21397.2.patch,
> HIVE-21397.3.patch, HIVE-21397.4.patch, HIVE-21397.5.patch, HIVE-21397.patch
>
>
> Steps to Reproduce this issue :
> -
> 1. Create a HIveManaged table as below :
> -
> {code:java}
> CREATE TABLE `bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest;
>
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1551206683') {code}
> -
> 2. Insert a few rows.
> -
> -
> 3. Check if bloom filter or active : [ It does not show bloom filters for
> hive managed tables ]
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> [length: 791]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> {code}
> -
> On Another hand: For hive External tables it works :
> -
> {code:java}
> CREATE external TABLE `ext_bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02') {code}
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> [length: 755]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
>
> Stream: column 1 section BLOOM_FILTER_UTF8 start: 41 length 110
> Stream: column 2 section BLOOM_FILTER_UTF8 start: 178 length 114
[jira] [Updated] (HIVE-21397) BloomFilter for hive Managed [ACID] table does not work as expected
[
https://issues.apache.org/jira/browse/HIVE-21397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-21397:
--
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to master, thanks [~dkuzmenko].
> BloomFilter for hive Managed [ACID] table does not work as expected
> ---
>
> Key: HIVE-21397
> URL: https://issues.apache.org/jira/browse/HIVE-21397
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Transactions
>Affects Versions: 3.1.1
>Reporter: vaibhav
>Assignee: Denys Kuzmenko
>Priority: Blocker
> Fix For: 4.0.0
>
> Attachments: HIVE-21397.1.patch, HIVE-21397.2.patch,
> HIVE-21397.3.patch, HIVE-21397.4.patch, HIVE-21397.5.patch, HIVE-21397.patch
>
>
> Steps to Reproduce this issue :
> -
> 1. Create a HIveManaged table as below :
> -
> {code:java}
> CREATE TABLE `bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest;
>
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1551206683') {code}
> -
> 2. Insert a few rows.
> -
> -
> 3. Check if bloom filter or active : [ It does not show bloom filters for
> hive managed tables ]
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> [length: 791]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> {code}
> -
> On Another hand: For hive External tables it works :
> -
> {code:java}
> CREATE external TABLE `ext_bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02') {code}
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> [length: 755]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
>
> Stream: column 1 section BLOOM_FILTER_UTF8 start: 41 length
[jira] [Commented] (HIVE-21508) ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
[
https://issues.apache.org/jira/browse/HIVE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925590#comment-16925590
]
Hive QA commented on HIVE-21508:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12979838/HIVE-21508.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16749 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid2]
(batchId=163)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18504/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18504/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18504/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12979838 - PreCommit-HIVE-Build
> ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
> --
>
> Key: HIVE-21508
> URL: https://issues.apache.org/jira/browse/HIVE-21508
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 3.2.0, 2.3.4
>Reporter: Adar Dembo
>Assignee: Ana Jalba
>Priority: Major
> Fix For: 4.0.0, 2.3.7
>
> Attachments: HIVE-21508.1.branch-2.3.patch, HIVE-21508.patch
>
>
> There's this block of code in {{HiveMetaStoreClient:resolveUris}} (called
> from the constructor) on master:
> {noformat}
> private URI metastoreUris[];
> ...
> if (MetastoreConf.getVar(conf,
> ConfVars.THRIFT_URI_SELECTION).equalsIgnoreCase("RANDOM")) {
> List uriList = Arrays.asList(metastoreUris);
> Collections.shuffle(uriList);
> metastoreUris = (URI[]) uriList.toArray();
> }
> {noformat}
> The cast to {{URI[]}} throws a {{ClassCastException}} beginning with JDK 10,
> possibly with JDK 9 as well. Note that {{THRIFT_URI_SELECTION}} defaults to
> {{RANDOM}} so this should affect anyone who creates a
> {{HiveMetaStoreClient}}. On master this can be overridden with {{SEQUENTIAL}}
> to avoid the broken case; I'm working against 2.3.4 where there's no such
> workaround.
> [Here's|https://stackoverflow.com/questions/51372788/array-cast-java-8-vs-java-9]
> a StackOverflow post that explains the issue in more detail. Interestingly,
> the author described the issue in the context of the HMS; not sure why there
> was no follow up with a Hive bug report.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-21508) ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
[
https://issues.apache.org/jira/browse/HIVE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ana Jalba updated HIVE-21508:
-
Fix Version/s: 2.3.7
> ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
> --
>
> Key: HIVE-21508
> URL: https://issues.apache.org/jira/browse/HIVE-21508
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 3.2.0, 2.3.4
>Reporter: Adar Dembo
>Assignee: Ana Jalba
>Priority: Major
> Fix For: 4.0.0, 2.3.7
>
> Attachments: HIVE-21508.1.branch-2.3.patch, HIVE-21508.patch
>
>
> There's this block of code in {{HiveMetaStoreClient:resolveUris}} (called
> from the constructor) on master:
> {noformat}
> private URI metastoreUris[];
> ...
> if (MetastoreConf.getVar(conf,
> ConfVars.THRIFT_URI_SELECTION).equalsIgnoreCase("RANDOM")) {
> List uriList = Arrays.asList(metastoreUris);
> Collections.shuffle(uriList);
> metastoreUris = (URI[]) uriList.toArray();
> }
> {noformat}
> The cast to {{URI[]}} throws a {{ClassCastException}} beginning with JDK 10,
> possibly with JDK 9 as well. Note that {{THRIFT_URI_SELECTION}} defaults to
> {{RANDOM}} so this should affect anyone who creates a
> {{HiveMetaStoreClient}}. On master this can be overridden with {{SEQUENTIAL}}
> to avoid the broken case; I'm working against 2.3.4 where there's no such
> workaround.
> [Here's|https://stackoverflow.com/questions/51372788/array-cast-java-8-vs-java-9]
> a StackOverflow post that explains the issue in more detail. Interestingly,
> the author described the issue in the context of the HMS; not sure why there
> was no follow up with a Hive bug report.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-21508) ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
[
https://issues.apache.org/jira/browse/HIVE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ana Jalba updated HIVE-21508:
-
Attachment: HIVE-21508.1.branch-2.3.patch
> ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
> --
>
> Key: HIVE-21508
> URL: https://issues.apache.org/jira/browse/HIVE-21508
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 3.2.0, 2.3.4
>Reporter: Adar Dembo
>Assignee: Ana Jalba
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21508.1.branch-2.3.patch, HIVE-21508.patch
>
>
> There's this block of code in {{HiveMetaStoreClient:resolveUris}} (called
> from the constructor) on master:
> {noformat}
> private URI metastoreUris[];
> ...
> if (MetastoreConf.getVar(conf,
> ConfVars.THRIFT_URI_SELECTION).equalsIgnoreCase("RANDOM")) {
> List uriList = Arrays.asList(metastoreUris);
> Collections.shuffle(uriList);
> metastoreUris = (URI[]) uriList.toArray();
> }
> {noformat}
> The cast to {{URI[]}} throws a {{ClassCastException}} beginning with JDK 10,
> possibly with JDK 9 as well. Note that {{THRIFT_URI_SELECTION}} defaults to
> {{RANDOM}} so this should affect anyone who creates a
> {{HiveMetaStoreClient}}. On master this can be overridden with {{SEQUENTIAL}}
> to avoid the broken case; I'm working against 2.3.4 where there's no such
> workaround.
> [Here's|https://stackoverflow.com/questions/51372788/array-cast-java-8-vs-java-9]
> a StackOverflow post that explains the issue in more detail. Interestingly,
> the author described the issue in the context of the HMS; not sure why there
> was no follow up with a Hive bug report.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-21508) ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
[
https://issues.apache.org/jira/browse/HIVE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925568#comment-16925568
]
Ana Jalba commented on HIVE-21508:
--
Looks like the master patch was ok. Can this be merged?
> ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
> --
>
> Key: HIVE-21508
> URL: https://issues.apache.org/jira/browse/HIVE-21508
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 3.2.0, 2.3.4
>Reporter: Adar Dembo
>Assignee: Ana Jalba
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-21508.patch
>
>
> There's this block of code in {{HiveMetaStoreClient:resolveUris}} (called
> from the constructor) on master:
> {noformat}
> private URI metastoreUris[];
> ...
> if (MetastoreConf.getVar(conf,
> ConfVars.THRIFT_URI_SELECTION).equalsIgnoreCase("RANDOM")) {
> List uriList = Arrays.asList(metastoreUris);
> Collections.shuffle(uriList);
> metastoreUris = (URI[]) uriList.toArray();
> }
> {noformat}
> The cast to {{URI[]}} throws a {{ClassCastException}} beginning with JDK 10,
> possibly with JDK 9 as well. Note that {{THRIFT_URI_SELECTION}} defaults to
> {{RANDOM}} so this should affect anyone who creates a
> {{HiveMetaStoreClient}}. On master this can be overridden with {{SEQUENTIAL}}
> to avoid the broken case; I'm working against 2.3.4 where there's no such
> workaround.
> [Here's|https://stackoverflow.com/questions/51372788/array-cast-java-8-vs-java-9]
> a StackOverflow post that explains the issue in more detail. Interestingly,
> the author described the issue in the context of the HMS; not sure why there
> was no follow up with a Hive bug report.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-21508) ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
[
https://issues.apache.org/jira/browse/HIVE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ana Jalba updated HIVE-21508:
-
Fix Version/s: (was: 3.2.0)
4.0.0
> ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
> --
>
> Key: HIVE-21508
> URL: https://issues.apache.org/jira/browse/HIVE-21508
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 3.2.0, 2.3.4
>Reporter: Adar Dembo
>Assignee: Ana Jalba
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21508.patch
>
>
> There's this block of code in {{HiveMetaStoreClient:resolveUris}} (called
> from the constructor) on master:
> {noformat}
> private URI metastoreUris[];
> ...
> if (MetastoreConf.getVar(conf,
> ConfVars.THRIFT_URI_SELECTION).equalsIgnoreCase("RANDOM")) {
> List uriList = Arrays.asList(metastoreUris);
> Collections.shuffle(uriList);
> metastoreUris = (URI[]) uriList.toArray();
> }
> {noformat}
> The cast to {{URI[]}} throws a {{ClassCastException}} beginning with JDK 10,
> possibly with JDK 9 as well. Note that {{THRIFT_URI_SELECTION}} defaults to
> {{RANDOM}} so this should affect anyone who creates a
> {{HiveMetaStoreClient}}. On master this can be overridden with {{SEQUENTIAL}}
> to avoid the broken case; I'm working against 2.3.4 where there's no such
> workaround.
> [Here's|https://stackoverflow.com/questions/51372788/array-cast-java-8-vs-java-9]
> a StackOverflow post that explains the issue in more detail. Interestingly,
> the author described the issue in the context of the HMS; not sure why there
> was no follow up with a Hive bug report.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22178) Parquet FilterPredicate throws CastException after SchemaEvolution.
[
https://issues.apache.org/jira/browse/HIVE-22178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-22178:
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to master!
Thanks [~nareshpr] for the patch!
> Parquet FilterPredicate throws CastException after SchemaEvolution.
> ---
>
> Key: HIVE-22178
> URL: https://issues.apache.org/jira/browse/HIVE-22178
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Naresh P R
>Assignee: Naresh P R
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22178.1.patch, HIVE-22178.2.patch,
> HIVE-22178.3.patch
>
>
> Below are the repro steps.
> {code:java}
> create table parq_test(age int, name string) stored as parquet;
> insert into parq_test values(1, '');
> alter table parq_test change age age string;
> insert into parq_test values('b', '');
> select * from parq_test where age='b';{code}
> Exception thrown after changing column datatype is below
> {code:java}
> Caused by: java.lang.IllegalArgumentException: FilterPredicate column: age's
> declared type (org.apache.parquet.io.api.Binary) does not match the schema
> found in file metadata. Column age is of type: INT32
> Valid types for this column are: [class java.lang.Integer]
> at
> org.apache.parquet.filter2.predicate.ValidTypeMap.assertTypeValid(ValidTypeMap.java:126)
> at
> org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:181)
> at
> org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:151)
> at
> org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:85)
> at
> org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:58)
> at
> org.apache.parquet.filter2.predicate.Operators$Eq.accept(Operators.java:181)
> at
> org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:63)
> at
> org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:92)
> at
> org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:43)
> at
> org.apache.parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:137)
> at
> org.apache.parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:64)
> at
> org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase.getSplit(ParquetRecordReaderBase.java:111)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:147)
> ... 31 more{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22178) Parquet FilterPredicate throws CastException after SchemaEvolution.
[
https://issues.apache.org/jira/browse/HIVE-22178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925551#comment-16925551
]
Sankar Hariappan commented on HIVE-22178:
-
+1, LGTM
> Parquet FilterPredicate throws CastException after SchemaEvolution.
> ---
>
> Key: HIVE-22178
> URL: https://issues.apache.org/jira/browse/HIVE-22178
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Naresh P R
>Assignee: Naresh P R
>Priority: Major
> Attachments: HIVE-22178.1.patch, HIVE-22178.2.patch,
> HIVE-22178.3.patch
>
>
> Below are the repro steps.
> {code:java}
> create table parq_test(age int, name string) stored as parquet;
> insert into parq_test values(1, '');
> alter table parq_test change age age string;
> insert into parq_test values('b', '');
> select * from parq_test where age='b';{code}
> Exception thrown after changing column datatype is below
> {code:java}
> Caused by: java.lang.IllegalArgumentException: FilterPredicate column: age's
> declared type (org.apache.parquet.io.api.Binary) does not match the schema
> found in file metadata. Column age is of type: INT32
> Valid types for this column are: [class java.lang.Integer]
> at
> org.apache.parquet.filter2.predicate.ValidTypeMap.assertTypeValid(ValidTypeMap.java:126)
> at
> org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:181)
> at
> org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:151)
> at
> org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:85)
> at
> org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:58)
> at
> org.apache.parquet.filter2.predicate.Operators$Eq.accept(Operators.java:181)
> at
> org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:63)
> at
> org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:92)
> at
> org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:43)
> at
> org.apache.parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:137)
> at
> org.apache.parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:64)
> at
> org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase.getSplit(ParquetRecordReaderBase.java:111)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:147)
> ... 31 more{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[ https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor reassigned HIVE-21604: --- Assignee: Laszlo Bodor > preCommit job should not be triggered on non-patch attachments > -- > > Key: HIVE-21604 > URL: https://issues.apache.org/jira/browse/HIVE-21604 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-21604.01.patch, files.txt > > > latest example: HIVE-21604 > https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520 > I think one should be able to upload any kind of attachments (e.g. > screenshot) without triggering the precommit job > 2 possible ways: > 1. strict: enable only .patch (should work) > (2. lenient: introduce blacklist, .png ...) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[
https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925542#comment-16925542
]
Laszlo Bodor edited comment on HIVE-21604 at 9/9/19 10:02 AM:
--
tested with attached files.txt
original script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*' files.txt |
grep -v -e 'htm[l]*$' | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*'
{code}
/jira/secure/attachment/12964860/files.txt
/jira/secure/attachment/12964869/01_mvn_out.png
/jira/secure/attachment/12964870/02_hive_log.png
/jira/secure/attachment/12965605/Screen%20Shot%202019-04-11%20at%204.54.41%20PM.png
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
modified script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
files.txt | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
{code}
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
I think the original "grep -v" wasn't satisfying (only filtered html files)
with the current solution, it doesn't seem too strict to force devs to upload
only .patch files for patches
[~vihangk1]: could you please review the patch?
was (Author: abstractdog):
tested with attached files.txt
original script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*' files.txt |
grep -v -e 'htm[l]*$' | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*'
{code}
/jira/secure/attachment/12964860/files.txt
/jira/secure/attachment/12964869/01_mvn_out.png
/jira/secure/attachment/12964870/02_hive_log.png
/jira/secure/attachment/12965605/Screen%20Shot%202019-04-11%20at%204.54.41%20PM.png
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
modified script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
files.txt | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
{code}
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
I think the original "grep -v" wasn't satisfying (only filtered html files), as
it doesn't seem too strict to force devs to upload .patch files
[~vihangk1]: could you please review the patch?
> preCommit job should not be triggered on non-patch attachments
> --
>
> Key: HIVE-21604
> URL: https://issues.apache.org/jira/browse/HIVE-21604
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21604.01.patch, files.txt
>
>
> latest example: HIVE-21604
> https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520
> I think one should be able to upload any kind of attachments (e.g.
> screenshot) without triggering the precommit job
> 2 possible ways:
> 1. strict: enable only .patch (should work)
> (2. lenient: introduce blacklist, .png ...)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[ https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21604: Status: Patch Available (was: Open) > preCommit job should not be triggered on non-patch attachments > -- > > Key: HIVE-21604 > URL: https://issues.apache.org/jira/browse/HIVE-21604 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-21604.01.patch, files.txt > > > latest example: HIVE-21604 > https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520 > I think one should be able to upload any kind of attachments (e.g. > screenshot) without triggering the precommit job > 2 possible ways: > 1. strict: enable only .patch (should work) > (2. lenient: introduce blacklist, .png ...) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[ https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21604: Fix Version/s: 4.0.0 > preCommit job should not be triggered on non-patch attachments > -- > > Key: HIVE-21604 > URL: https://issues.apache.org/jira/browse/HIVE-21604 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21604.01.patch, files.txt > > > latest example: HIVE-21604 > https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520 > I think one should be able to upload any kind of attachments (e.g. > screenshot) without triggering the precommit job > 2 possible ways: > 1. strict: enable only .patch (should work) > (2. lenient: introduce blacklist, .png ...) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-21508) ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
[
https://issues.apache.org/jira/browse/HIVE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925543#comment-16925543
]
Hive QA commented on HIVE-21508:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
51s{color} | {color:blue} standalone-metastore/metastore-common in master has
32 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18504/dev-support/hive-personality.sh
|
| git revision | master / 1b04924 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common U:
standalone-metastore/metastore-common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18504/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
> --
>
> Key: HIVE-21508
> URL: https://issues.apache.org/jira/browse/HIVE-21508
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 3.2.0, 2.3.4
>Reporter: Adar Dembo
>Assignee: Ana Jalba
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-21508.patch
>
>
> There's this block of code in {{HiveMetaStoreClient:resolveUris}} (called
> from the constructor) on master:
> {noformat}
> private URI metastoreUris[];
> ...
> if (MetastoreConf.getVar(conf,
> ConfVars.THRIFT_URI_SELECTION).equalsIgnoreCase("RANDOM")) {
> List uriList = Arrays.asList(metastoreUris);
> Collections.shuffle(uriList);
> metastoreUris = (URI[]) uriList.toArray();
> }
> {noformat}
> The cast to {{URI[]}} throws a {{ClassCastException}} beginning with JDK 10,
> possibly with JDK 9 as well. Note that {{THRIFT_URI_SELECTION}} defaults to
> {{RANDOM}} so this should affect anyone who creates a
> {{HiveMetaStoreClient}}. On master this can be overridden with {{SEQUENTIAL}}
> to avoid the broken case; I'm working against 2.3.4 where there's no such
> workaround.
> [Here's|https://stackoverflow.com/questions/51372788/array-cast-java-8-vs-java-9]
> a StackOverflow post that explains the issue in more detail. Interestingly,
> the author described the issue in the context of the HMS; not sure why
[jira] [Commented] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[
https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925542#comment-16925542
]
Laszlo Bodor commented on HIVE-21604:
-
tested with attached files.txt
original script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*' files.txt |
grep -v -e 'htm[l]*$' | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*'
{code}
/jira/secure/attachment/12964860/files.txt
/jira/secure/attachment/12964869/01_mvn_out.png
/jira/secure/attachment/12964870/02_hive_log.png
/jira/secure/attachment/12965605/Screen%20Shot%202019-04-11%20at%204.54.41%20PM.png
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
modified script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
files.txt | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
{code}
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
I think the original "grep -v" wasn't satisfying (only filtered html files), as
it doesn't seem too strict to force devs to upload .patch files
[~vihangk1]: could you please review the patch
> preCommit job should not be triggered on non-patch attachments
> --
>
> Key: HIVE-21604
> URL: https://issues.apache.org/jira/browse/HIVE-21604
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21604.01.patch, files.txt
>
>
> latest example: HIVE-21604
> https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520
> I think one should be able to upload any kind of attachments (e.g.
> screenshot) without triggering the precommit job
> 2 possible ways:
> 1. strict: enable only .patch (should work)
> (2. lenient: introduce blacklist, .png ...)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[
https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925542#comment-16925542
]
Laszlo Bodor edited comment on HIVE-21604 at 9/9/19 10:00 AM:
--
tested with attached files.txt
original script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*' files.txt |
grep -v -e 'htm[l]*$' | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*'
{code}
/jira/secure/attachment/12964860/files.txt
/jira/secure/attachment/12964869/01_mvn_out.png
/jira/secure/attachment/12964870/02_hive_log.png
/jira/secure/attachment/12965605/Screen%20Shot%202019-04-11%20at%204.54.41%20PM.png
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
modified script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
files.txt | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
{code}
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
I think the original "grep -v" wasn't satisfying (only filtered html files), as
it doesn't seem too strict to force devs to upload .patch files
[~vihangk1]: could you please review the patch?
was (Author: abstractdog):
tested with attached files.txt
original script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*' files.txt |
grep -v -e 'htm[l]*$' | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*'
{code}
/jira/secure/attachment/12964860/files.txt
/jira/secure/attachment/12964869/01_mvn_out.png
/jira/secure/attachment/12964870/02_hive_log.png
/jira/secure/attachment/12965605/Screen%20Shot%202019-04-11%20at%204.54.41%20PM.png
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
modified script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
files.txt | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
{code}
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
I think the original "grep -v" wasn't satisfying (only filtered html files), as
it doesn't seem too strict to force devs to upload .patch files
[~vihangk1]: could you please review the patch
> preCommit job should not be triggered on non-patch attachments
> --
>
> Key: HIVE-21604
> URL: https://issues.apache.org/jira/browse/HIVE-21604
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21604.01.patch, files.txt
>
>
> latest example: HIVE-21604
> https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520
> I think one should be able to upload any kind of attachments (e.g.
> screenshot) without triggering the precommit job
> 2 possible ways:
> 1. strict: enable only .patch (should work)
> (2. lenient: introduce blacklist, .png ...)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[ https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21604: Attachment: HIVE-21604.01.patch > preCommit job should not be triggered on non-patch attachments > -- > > Key: HIVE-21604 > URL: https://issues.apache.org/jira/browse/HIVE-21604 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Priority: Major > Attachments: HIVE-21604.01.patch, files.txt > > > latest example: HIVE-21604 > https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520 > I think one should be able to upload any kind of attachments (e.g. > screenshot) without triggering the precommit job > 2 possible ways: > 1. strict: enable only .patch (should work) > (2. lenient: introduce blacklist, .png ...) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[ https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21604: Attachment: files.txt > preCommit job should not be triggered on non-patch attachments > -- > > Key: HIVE-21604 > URL: https://issues.apache.org/jira/browse/HIVE-21604 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Priority: Major > Attachments: HIVE-21604.01.patch, files.txt > > > latest example: HIVE-21604 > https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520 > I think one should be able to upload any kind of attachments (e.g. > screenshot) without triggering the precommit job > 2 possible ways: > 1. strict: enable only .patch (should work) > (2. lenient: introduce blacklist, .png ...) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[ https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=308712=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308712 ] ASF GitHub Bot logged work on HIVE-20683: - Author: ASF GitHub Bot Created on: 09/Sep/19 09:50 Start Date: 09/Sep/19 09:50 Worklog Time Spent: 10m Work Description: b-slim commented on pull request #723: [HIVE-20683] Add the Ability to push Dynamic Between and Bloom filters to Druid URL: https://github.com/apache/hive/pull/723#discussion_r322155098 ## File path: ql/src/test/queries/clientpositive/druidmini_semijoin_reduction_all_types.q ## @@ -0,0 +1,144 @@ +--! qt:dataset:srcpart +--! qt:dataset:druid_table_alltypesorc +--! qt:dataset:alltypesorc + +set hive.compute.query.using.stats=false; +set hive.mapred.mode=nonstrict; +set hive.explain.user=false; +set hive.optimize.ppd=true; +set hive.ppd.remove.duplicatefilters=true; +set hive.tez.dynamic.partition.pruning=true; +set hive.tez.dynamic.semijoin.reduction=true; +set hive.optimize.metadataonly=false; +set hive.optimize.index.filter=true; +set hive.stats.autogather=true; +set hive.tez.bigtable.minsize.semijoin.reduction=1; +set hive.tez.min.bloom.filter.entries=1; +set hive.stats.fetch.column.stats=true; +set hive.disable.unsafe.external.table.operations=false; +set hive.tez.dynamic.semijoin.reduction.for.mapjoin=true; + +DROP TABLE IF EXISTS alltypesorc_small; +CREATE TABLE alltypesorc_small( +ctinyint TINYINT, +csmallint SMALLINT, +cint INT, +cbigint BIGINT, +cfloat FLOAT, +cdouble DOUBLE, +cstring1 STRING, +cstring2 STRING, +ctimestamp1 TIMESTAMP, +cboolean1 BOOLEAN, +cboolean2 BOOLEAN) +STORED AS ORC; +Insert into table alltypesorc_small +Select ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, cast(`__time` as timestamp), cboolean1, cboolean2 from druid_table_alltypesorc where cstring2 like '%a%' and cstring1 like '%a%'; +Select count(*) from alltypesorc_small; +Select count(*) from druid_table_alltypesorc; + +DESCRIBE druid_table_alltypesorc; +DESCRIBE alltypesorc_small; + +-- Test Joins on all column types one by one +-- String +set hive.disable.unsafe.external.table.operations=false; +EXPLAIN select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cstring1 = druid_table_alltypesorc.cstring1); +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cstring1 = druid_table_alltypesorc.cstring1); +set hive.disable.unsafe.external.table.operations=true; +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cstring1 = druid_table_alltypesorc.cstring1); + +-- tinyint +set hive.disable.unsafe.external.table.operations=false; +EXPLAIN select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.ctinyint = druid_table_alltypesorc.ctinyint); +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.ctinyint = druid_table_alltypesorc.ctinyint); +set hive.disable.unsafe.external.table.operations=true; +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.ctinyint = druid_table_alltypesorc.ctinyint); + +-- smallint +set hive.disable.unsafe.external.table.operations=false; +EXPLAIN select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.csmallint = druid_table_alltypesorc.csmallint); +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.csmallint = druid_table_alltypesorc.csmallint); +set hive.disable.unsafe.external.table.operations=true; +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.csmallint = druid_table_alltypesorc.csmallint); + +-- int +set hive.disable.unsafe.external.table.operations=false; +EXPLAIN select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cint = druid_table_alltypesorc.cint); +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cint = druid_table_alltypesorc.cint); +set hive.disable.unsafe.external.table.operations=true; +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cint = druid_table_alltypesorc.cint); + +-- bigint +set hive.disable.unsafe.external.table.operations=false; +EXPLAIN select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cbigint = druid_table_alltypesorc.cbigint); +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cbigint = druid_table_alltypesorc.cbigint); +set hive.disable.unsafe.external.table.operations=true; +select count(*) from alltypesorc_small join druid_table_alltypesorc on (alltypesorc_small.cbigint = druid_table_alltypesorc.cbigint); + +--
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=308708=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308708
]
ASF GitHub Bot logged work on HIVE-20683:
-
Author: ASF GitHub Bot
Created on: 09/Sep/19 09:40
Start Date: 09/Sep/19 09:40
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #723: [HIVE-20683] Add
the Ability to push Dynamic Between and Bloom filters to Druid
URL: https://github.com/apache/hive/pull/723#discussion_r322150545
##
File path:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java
##
@@ -894,4 +945,255 @@ public static IndexSpec getIndexSpec(Configuration jc) {
ImmutableList aggregatorFactories =
aggregatorFactoryBuilder.build();
return Pair.of(dimensions, aggregatorFactories.toArray(new
AggregatorFactory[0]));
}
+
+ // Druid only supports String,Long,Float,Double selectors
+ private static Set druidSupportedTypeInfos =
ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo, TypeInfoFactory.charTypeInfo,
+ TypeInfoFactory.varcharTypeInfo, TypeInfoFactory.byteTypeInfo,
+ TypeInfoFactory.intTypeInfo, TypeInfoFactory.longTypeInfo,
+ TypeInfoFactory.shortTypeInfo, TypeInfoFactory.doubleTypeInfo
+ );
+
+ private static Set stringTypeInfos = ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo,
+ TypeInfoFactory.charTypeInfo, TypeInfoFactory.varcharTypeInfo
+ );
+
+
+ public static org.apache.druid.query.Query
addDynamicFilters(org.apache.druid.query.Query query,
+ ExprNodeGenericFuncDesc filterExpr, Configuration conf, boolean
resolveDynamicValues
+ ) {
+List virtualColumns = Arrays
+.asList(getVirtualColumns(query).getVirtualColumns());
+org.apache.druid.query.Query rv = query;
+DimFilter joinReductionFilter = toDruidFilter(filterExpr, conf,
virtualColumns,
+resolveDynamicValues
+);
+if(joinReductionFilter != null) {
+ String type = query.getType();
+ DimFilter filter = new AndDimFilter(joinReductionFilter,
query.getFilter());
+ switch (type) {
+ case org.apache.druid.query.Query.TIMESERIES:
+rv = Druids.TimeseriesQueryBuilder.copy((TimeseriesQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.TOPN:
+rv = new TopNQueryBuilder((TopNQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.GROUP_BY:
+rv = new GroupByQuery.Builder((GroupByQuery) query)
+.setDimFilter(filter)
+.setVirtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SCAN:
+rv = ScanQuery.ScanQueryBuilder.copy((ScanQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SELECT:
+rv = Druids.SelectQueryBuilder.copy((SelectQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ default:
+throw new UnsupportedOperationException("Unsupported Query type " +
type);
+ }
+}
+return rv;
+ }
+
+ @Nullable
+ private static DimFilter toDruidFilter(ExprNodeDesc filterExpr,
Configuration configuration,
+ List virtualColumns, boolean resolveDynamicValues
+ ) {
+if(filterExpr == null) {
+ return null;
+}
+Class genericUDFClass =
getGenericUDFClassFromExprDesc(filterExpr);
+if(FunctionRegistry.isOpAnd(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null && !delegates.isEmpty()) {
+return new AndDimFilter(delegates);
+ }
+}
+if(FunctionRegistry.isOpOr(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null) {
+return new OrDimFilter(delegates);
+ }
+} else if(GenericUDFBetween.class
[jira] [Updated] (HIVE-21884) Scheduled query support
[ https://issues.apache.org/jira/browse/HIVE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21884: Attachment: HIVE-21844.06.patch > Scheduled query support > --- > > Key: HIVE-21884 > URL: https://issues.apache.org/jira/browse/HIVE-21884 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch, > HIVE-21844.06.patch, HIVE-21884.01.patch, HIVE-21884.02.patch, > HIVE-21884.03.patch, Scheduled queries2.pdf > > > design document: > https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit# > in case the google doc is not reachable: [^Scheduled queries2.pdf] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-21508) ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
[
https://issues.apache.org/jira/browse/HIVE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ana Jalba updated HIVE-21508:
-
Attachment: HIVE-21508.patch
Fix Version/s: (was: 2.3.7)
(was: 2.4.0)
Status: Patch Available (was: In Progress)
> ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
> --
>
> Key: HIVE-21508
> URL: https://issues.apache.org/jira/browse/HIVE-21508
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 2.3.4, 3.2.0
>Reporter: Adar Dembo
>Assignee: Ana Jalba
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-21508.patch
>
>
> There's this block of code in {{HiveMetaStoreClient:resolveUris}} (called
> from the constructor) on master:
> {noformat}
> private URI metastoreUris[];
> ...
> if (MetastoreConf.getVar(conf,
> ConfVars.THRIFT_URI_SELECTION).equalsIgnoreCase("RANDOM")) {
> List uriList = Arrays.asList(metastoreUris);
> Collections.shuffle(uriList);
> metastoreUris = (URI[]) uriList.toArray();
> }
> {noformat}
> The cast to {{URI[]}} throws a {{ClassCastException}} beginning with JDK 10,
> possibly with JDK 9 as well. Note that {{THRIFT_URI_SELECTION}} defaults to
> {{RANDOM}} so this should affect anyone who creates a
> {{HiveMetaStoreClient}}. On master this can be overridden with {{SEQUENTIAL}}
> to avoid the broken case; I'm working against 2.3.4 where there's no such
> workaround.
> [Here's|https://stackoverflow.com/questions/51372788/array-cast-java-8-vs-java-9]
> a StackOverflow post that explains the issue in more detail. Interestingly,
> the author described the issue in the context of the HMS; not sure why there
> was no follow up with a Hive bug report.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=308707=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308707
]
ASF GitHub Bot logged work on HIVE-20683:
-
Author: ASF GitHub Bot
Created on: 09/Sep/19 09:36
Start Date: 09/Sep/19 09:36
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #723: [HIVE-20683] Add
the Ability to push Dynamic Between and Bloom filters to Druid
URL: https://github.com/apache/hive/pull/723#discussion_r322148915
##
File path:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java
##
@@ -894,4 +945,255 @@ public static IndexSpec getIndexSpec(Configuration jc) {
ImmutableList aggregatorFactories =
aggregatorFactoryBuilder.build();
return Pair.of(dimensions, aggregatorFactories.toArray(new
AggregatorFactory[0]));
}
+
+ // Druid only supports String,Long,Float,Double selectors
+ private static Set druidSupportedTypeInfos =
ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo, TypeInfoFactory.charTypeInfo,
+ TypeInfoFactory.varcharTypeInfo, TypeInfoFactory.byteTypeInfo,
+ TypeInfoFactory.intTypeInfo, TypeInfoFactory.longTypeInfo,
+ TypeInfoFactory.shortTypeInfo, TypeInfoFactory.doubleTypeInfo
+ );
+
+ private static Set stringTypeInfos = ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo,
+ TypeInfoFactory.charTypeInfo, TypeInfoFactory.varcharTypeInfo
+ );
+
+
+ public static org.apache.druid.query.Query
addDynamicFilters(org.apache.druid.query.Query query,
+ ExprNodeGenericFuncDesc filterExpr, Configuration conf, boolean
resolveDynamicValues
+ ) {
+List virtualColumns = Arrays
+.asList(getVirtualColumns(query).getVirtualColumns());
+org.apache.druid.query.Query rv = query;
+DimFilter joinReductionFilter = toDruidFilter(filterExpr, conf,
virtualColumns,
+resolveDynamicValues
+);
+if(joinReductionFilter != null) {
+ String type = query.getType();
+ DimFilter filter = new AndDimFilter(joinReductionFilter,
query.getFilter());
+ switch (type) {
+ case org.apache.druid.query.Query.TIMESERIES:
+rv = Druids.TimeseriesQueryBuilder.copy((TimeseriesQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.TOPN:
+rv = new TopNQueryBuilder((TopNQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.GROUP_BY:
+rv = new GroupByQuery.Builder((GroupByQuery) query)
+.setDimFilter(filter)
+.setVirtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SCAN:
+rv = ScanQuery.ScanQueryBuilder.copy((ScanQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SELECT:
+rv = Druids.SelectQueryBuilder.copy((SelectQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ default:
+throw new UnsupportedOperationException("Unsupported Query type " +
type);
+ }
+}
+return rv;
+ }
+
+ @Nullable
+ private static DimFilter toDruidFilter(ExprNodeDesc filterExpr,
Configuration configuration,
+ List virtualColumns, boolean resolveDynamicValues
+ ) {
+if(filterExpr == null) {
+ return null;
+}
+Class genericUDFClass =
getGenericUDFClassFromExprDesc(filterExpr);
+if(FunctionRegistry.isOpAnd(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null && !delegates.isEmpty()) {
+return new AndDimFilter(delegates);
+ }
+}
+if(FunctionRegistry.isOpOr(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null) {
+return new OrDimFilter(delegates);
+ }
+} else if(GenericUDFBetween.class
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=308706=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308706
]
ASF GitHub Bot logged work on HIVE-20683:
-
Author: ASF GitHub Bot
Created on: 09/Sep/19 09:36
Start Date: 09/Sep/19 09:36
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #723: [HIVE-20683] Add
the Ability to push Dynamic Between and Bloom filters to Druid
URL: https://github.com/apache/hive/pull/723#discussion_r322148627
##
File path:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java
##
@@ -894,4 +945,255 @@ public static IndexSpec getIndexSpec(Configuration jc) {
ImmutableList aggregatorFactories =
aggregatorFactoryBuilder.build();
return Pair.of(dimensions, aggregatorFactories.toArray(new
AggregatorFactory[0]));
}
+
+ // Druid only supports String,Long,Float,Double selectors
+ private static Set druidSupportedTypeInfos =
ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo, TypeInfoFactory.charTypeInfo,
+ TypeInfoFactory.varcharTypeInfo, TypeInfoFactory.byteTypeInfo,
+ TypeInfoFactory.intTypeInfo, TypeInfoFactory.longTypeInfo,
+ TypeInfoFactory.shortTypeInfo, TypeInfoFactory.doubleTypeInfo
+ );
+
+ private static Set stringTypeInfos = ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo,
+ TypeInfoFactory.charTypeInfo, TypeInfoFactory.varcharTypeInfo
+ );
+
+
+ public static org.apache.druid.query.Query
addDynamicFilters(org.apache.druid.query.Query query,
+ ExprNodeGenericFuncDesc filterExpr, Configuration conf, boolean
resolveDynamicValues
+ ) {
+List virtualColumns = Arrays
+.asList(getVirtualColumns(query).getVirtualColumns());
+org.apache.druid.query.Query rv = query;
+DimFilter joinReductionFilter = toDruidFilter(filterExpr, conf,
virtualColumns,
+resolveDynamicValues
+);
+if(joinReductionFilter != null) {
+ String type = query.getType();
+ DimFilter filter = new AndDimFilter(joinReductionFilter,
query.getFilter());
+ switch (type) {
+ case org.apache.druid.query.Query.TIMESERIES:
+rv = Druids.TimeseriesQueryBuilder.copy((TimeseriesQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.TOPN:
+rv = new TopNQueryBuilder((TopNQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.GROUP_BY:
+rv = new GroupByQuery.Builder((GroupByQuery) query)
+.setDimFilter(filter)
+.setVirtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SCAN:
+rv = ScanQuery.ScanQueryBuilder.copy((ScanQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SELECT:
+rv = Druids.SelectQueryBuilder.copy((SelectQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ default:
+throw new UnsupportedOperationException("Unsupported Query type " +
type);
+ }
+}
+return rv;
+ }
+
+ @Nullable
+ private static DimFilter toDruidFilter(ExprNodeDesc filterExpr,
Configuration configuration,
+ List virtualColumns, boolean resolveDynamicValues
+ ) {
+if(filterExpr == null) {
+ return null;
+}
+Class genericUDFClass =
getGenericUDFClassFromExprDesc(filterExpr);
+if(FunctionRegistry.isOpAnd(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null && !delegates.isEmpty()) {
+return new AndDimFilter(delegates);
+ }
+}
+if(FunctionRegistry.isOpOr(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null) {
Review comment:
same here this can not be null.
[jira] [Commented] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925524#comment-16925524
]
Hive QA commented on HIVE-21884:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12979836/HIVE-21844.05.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18502/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18502/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18502/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-09-09 09:33:43.402
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-18502/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-09-09 09:33:43.405
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
5565311..1b04924 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 5565311 HIVE-22143 Break up DDLSemanticAnalyzer - extract
Database related analyzers (Miklos Gergely reviewd by Jesus Camacho Rodriguez)
+ git clean -f -d
Removing ${project.basedir}/
Removing itests/${project.basedir}/
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 3 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 1b04924 HIVE-22106: Remove cross-query synchronization for the
partition-eval (Slim B via Gopal V)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-09-09 09:33:44.823
+ rm -rf ../yetus_PreCommit-HIVE-Build-18502
+ mkdir ../yetus_PreCommit-HIVE-Build-18502
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-18502
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-18502/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:27: trailing whitespace.
# +-- X: p#1
/data/hiveptest/working/scratch/build.patch:249: trailing whitespace.
SELECT 'abc123' AS ORIG_VALUE, CAST('abc123' AS BIGINT) AS CAST_AS_BIGINT, CASE
WHEN CAST('abc123' AS BIGINT) IS NULL THEN 'YES' ELSE 'NO' END AS IS_IT_NULL;
/data/hiveptest/working/scratch/build.patch:815: trailing whitespace.
/data/hiveptest/working/scratch/build.patch:825: trailing whitespace.
/data/hiveptest/working/scratch/build.patch:836: trailing whitespace.
warning: squelched 177 whitespace errors
warning: 182 lines add whitespace errors.
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc5868761099962570501.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc5868761099962570501.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc1133043159592088760.exe, --version]
libprotoc 2.5.0
ANTLR Parser Generator Version 3.5.2
Output file
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=308705=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308705
]
ASF GitHub Bot logged work on HIVE-20683:
-
Author: ASF GitHub Bot
Created on: 09/Sep/19 09:35
Start Date: 09/Sep/19 09:35
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #723: [HIVE-20683] Add
the Ability to push Dynamic Between and Bloom filters to Druid
URL: https://github.com/apache/hive/pull/723#discussion_r322148428
##
File path:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java
##
@@ -894,4 +945,255 @@ public static IndexSpec getIndexSpec(Configuration jc) {
ImmutableList aggregatorFactories =
aggregatorFactoryBuilder.build();
return Pair.of(dimensions, aggregatorFactories.toArray(new
AggregatorFactory[0]));
}
+
+ // Druid only supports String,Long,Float,Double selectors
+ private static Set druidSupportedTypeInfos =
ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo, TypeInfoFactory.charTypeInfo,
+ TypeInfoFactory.varcharTypeInfo, TypeInfoFactory.byteTypeInfo,
+ TypeInfoFactory.intTypeInfo, TypeInfoFactory.longTypeInfo,
+ TypeInfoFactory.shortTypeInfo, TypeInfoFactory.doubleTypeInfo
+ );
+
+ private static Set stringTypeInfos = ImmutableSet.of(
+ TypeInfoFactory.stringTypeInfo,
+ TypeInfoFactory.charTypeInfo, TypeInfoFactory.varcharTypeInfo
+ );
+
+
+ public static org.apache.druid.query.Query
addDynamicFilters(org.apache.druid.query.Query query,
+ ExprNodeGenericFuncDesc filterExpr, Configuration conf, boolean
resolveDynamicValues
+ ) {
+List virtualColumns = Arrays
+.asList(getVirtualColumns(query).getVirtualColumns());
+org.apache.druid.query.Query rv = query;
+DimFilter joinReductionFilter = toDruidFilter(filterExpr, conf,
virtualColumns,
+resolveDynamicValues
+);
+if(joinReductionFilter != null) {
+ String type = query.getType();
+ DimFilter filter = new AndDimFilter(joinReductionFilter,
query.getFilter());
+ switch (type) {
+ case org.apache.druid.query.Query.TIMESERIES:
+rv = Druids.TimeseriesQueryBuilder.copy((TimeseriesQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.TOPN:
+rv = new TopNQueryBuilder((TopNQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.GROUP_BY:
+rv = new GroupByQuery.Builder((GroupByQuery) query)
+.setDimFilter(filter)
+.setVirtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SCAN:
+rv = ScanQuery.ScanQueryBuilder.copy((ScanQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ case org.apache.druid.query.Query.SELECT:
+rv = Druids.SelectQueryBuilder.copy((SelectQuery) query)
+.filters(filter)
+.virtualColumns(VirtualColumns.create(virtualColumns))
+.build();
+break;
+ default:
+throw new UnsupportedOperationException("Unsupported Query type " +
type);
+ }
+}
+return rv;
+ }
+
+ @Nullable
+ private static DimFilter toDruidFilter(ExprNodeDesc filterExpr,
Configuration configuration,
+ List virtualColumns, boolean resolveDynamicValues
+ ) {
+if(filterExpr == null) {
+ return null;
+}
+Class genericUDFClass =
getGenericUDFClassFromExprDesc(filterExpr);
+if(FunctionRegistry.isOpAnd(filterExpr)) {
+ Iterator iterator = filterExpr.getChildren().iterator();
+ List delegates = Lists.newArrayList();
+ while (iterator.hasNext()) {
+DimFilter filter = toDruidFilter(iterator.next(), configuration,
virtualColumns,
+resolveDynamicValues
+);
+if(filter != null) {
+ delegates.add(filter);
+}
+ }
+ if(delegates != null && !delegates.isEmpty()) {
Review comment:
how this can be null ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 308705)
Time Spent: 3h (was: 2h 50m)
> Add the Ability to push Dynamic Between and Bloom filters to Druid
>
[jira] [Work logged] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?focusedWorklogId=308703=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308703
]
ASF GitHub Bot logged work on HIVE-20683:
-
Author: ASF GitHub Bot
Created on: 09/Sep/19 09:34
Start Date: 09/Sep/19 09:34
Worklog Time Spent: 10m
Work Description: b-slim commented on pull request #723: [HIVE-20683] Add
the Ability to push Dynamic Between and Bloom filters to Druid
URL: https://github.com/apache/hive/pull/723#discussion_r322147955
##
File path:
druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java
##
@@ -91,17 +114,37 @@
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.NonSyncByteArrayInputStream;
Review comment:
this is not used.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 308703)
Time Spent: 2h 50m (was: 2h 40m)
> Add the Ability to push Dynamic Between and Bloom filters to Druid
> --
>
> Key: HIVE-20683
> URL: https://issues.apache.org/jira/browse/HIVE-20683
> Project: Hive
> Issue Type: New Feature
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20683.1.patch, HIVE-20683.2.patch,
> HIVE-20683.3.patch, HIVE-20683.4.patch, HIVE-20683.5.patch,
> HIVE-20683.6.patch, HIVE-20683.8.patch, HIVE-20683.patch
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> For optimizing joins, Hive generates BETWEEN filter with min-max and BLOOM
> filter for filtering one side of semi-join.
> Druid 0.13.0 will have support for Bloom filters (Added via
> https://github.com/apache/incubator-druid/pull/6222)
> Implementation details -
> # Hive generates and passes the filters as part of 'filterExpr' in TableScan.
> # DruidQueryBasedRecordReader gets this filter passed as part of the conf.
> # During execution phase, before sending the query to druid in
> DruidQueryBasedRecordReader we will deserialize this filter, translate it
> into a DruidDimFilter and add it to existing DruidQuery. Tez executor
> already ensures that when we start reading results from the record reader,
> all the dynamic values are initialized.
> # Explaining a druid query also prints the query sent to druid as
> {{druid.json.query}}. We also need to make sure to update the druid query
> with the filters. During explain we do not have the actual values for the
> dynamic values, so instead of values we will print the dynamic expression
> itself as part of druid query.
> Note:- This work needs druid to be updated to version 0.13.0
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
