[jira] [Work logged] (HIVE-22360) MultiSerDe returns wrong results in last column when the loaded file has more columns than those in table schema
[
https://issues.apache.org/jira/browse/HIVE-22360?focusedWorklogId=331301=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331301
]
ASF GitHub Bot logged work on HIVE-22360:
-
Author: ASF GitHub Bot
Created on: 21/Oct/19 09:30
Start Date: 21/Oct/19 09:30
Worklog Time Spent: 10m
Work Description: ShubhamChaurasia commented on pull request #823:
HIVE-22360: MultiSerDe returns wrong results in last column when the loaded
file has more columns than those in table schema
URL: https://github.com/apache/hive/pull/823
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331301)
Remaining Estimate: 0h
Time Spent: 10m
> MultiSerDe returns wrong results in last column when the loaded file has more
> columns than those in table schema
>
>
> Key: HIVE-22360
> URL: https://issues.apache.org/jira/browse/HIVE-22360
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Repro steps:
> Input file:
> {code}
> 1^,1^,^,0^,0^,0
> 2^,1^,^,0^,1^,0
> 3^,1^,^,0^,0^,0
> 4^,1^,^,0^,1^,0
> {code}
> Queries:
> {code}
> CREATE TABLE n2(colA int, colB tinyint, colC timestamp, colD smallint, colE
> smallint) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' WITH
> SERDEPROPERTIES ("field.delim"="^,")STORED AS TEXTFILE;
> LOAD DATA LOCAL INPATH '/Users/schaurasia/Documents/input_6_cols.csv'
> OVERWRITE INTO TABLE n2;
> select * from n2;
> // wrong last column results here.
> +--+--+--+--+--+
> | n2.cola | n2.colb | n2.colc | n2.cold | n2.cole |
> +--+--+--+--+--+
> | 1| 1| NULL | 0| NULL |
> | 2| 1| NULL | 0| NULL |
> | 3| 1| NULL | 0| NULL |
> | 4| 1| NULL | 0| NULL |
> +--+--+--+--+--+
> {code}
> Cause:
> In multi-serde parsing, the total length calculation here:
> https://github.com/apache/hive/blob/rel/release-3.1.2/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java#L308
> does not take extra fields into account.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22360) MultiSerDe returns wrong results in last column when the loaded file has more columns than those in table schema
[
https://issues.apache.org/jira/browse/HIVE-22360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-22360:
--
Labels: pull-request-available (was: )
> MultiSerDe returns wrong results in last column when the loaded file has more
> columns than those in table schema
>
>
> Key: HIVE-22360
> URL: https://issues.apache.org/jira/browse/HIVE-22360
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
>
> Repro steps:
> Input file:
> {code}
> 1^,1^,^,0^,0^,0
> 2^,1^,^,0^,1^,0
> 3^,1^,^,0^,0^,0
> 4^,1^,^,0^,1^,0
> {code}
> Queries:
> {code}
> CREATE TABLE n2(colA int, colB tinyint, colC timestamp, colD smallint, colE
> smallint) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' WITH
> SERDEPROPERTIES ("field.delim"="^,")STORED AS TEXTFILE;
> LOAD DATA LOCAL INPATH '/Users/schaurasia/Documents/input_6_cols.csv'
> OVERWRITE INTO TABLE n2;
> select * from n2;
> // wrong last column results here.
> +--+--+--+--+--+
> | n2.cola | n2.colb | n2.colc | n2.cold | n2.cole |
> +--+--+--+--+--+
> | 1| 1| NULL | 0| NULL |
> | 2| 1| NULL | 0| NULL |
> | 3| 1| NULL | 0| NULL |
> | 4| 1| NULL | 0| NULL |
> +--+--+--+--+--+
> {code}
> Cause:
> In multi-serde parsing, the total length calculation here:
> https://github.com/apache/hive/blob/rel/release-3.1.2/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java#L308
> does not take extra fields into account.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-19653) Incorrect predicate pushdown for groupby with grouping sets
[
https://issues.apache.org/jira/browse/HIVE-19653?focusedWorklogId=331291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331291
]
ASF GitHub Bot logged work on HIVE-19653:
-
Author: ASF GitHub Bot
Created on: 21/Oct/19 09:00
Start Date: 21/Oct/19 09:00
Worklog Time Spent: 10m
Work Description: fan624009652 commented on issue #354: HIVE-19653:
Incorrect predicate pushdown for groupby with grouping sets
URL: https://github.com/apache/hive/pull/354#issuecomment-544419803
Now I'm facing this problem and I wonder why this pull request is still
unmerged.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331291)
Remaining Estimate: 0h
Time Spent: 10m
> Incorrect predicate pushdown for groupby with grouping sets
> ---
>
> Key: HIVE-19653
> URL: https://issues.apache.org/jira/browse/HIVE-19653
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Zhang Li
>Assignee: Zhang Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-19653.1.patch, HIVE-19653.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Consider the following query:
> {code:java}
> CREATE TABLE T1(a STRING, b STRING, s BIGINT);
> INSERT OVERWRITE TABLE T1 VALUES ('', '', 123456);
> SELECT * FROM (
> SELECT a, b, sum(s)
> FROM T1
> GROUP BY a, b GROUPING SETS ((), (a), (b), (a, b))
> ) t WHERE a IS NOT NULL;
> {code}
> When hive.optimize.ppd is enabled (and hive.cbo.enable=false), the query will
> output:
> {code:java}
> NULL NULL123456
> NULL 123456
> NULL123456
> 123456
> {code}
> We can see the predicate "a IS NOT NULL" takes no effect, which is incorrect.
> When performing PPD optimization for a GBY operator, we should make sure all
> grouping sets contains the processing expr before pushdown. otherwise the
> expr value after GBY is changed and the result is wrong.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (HIVE-21226) Exclude read-only transactions from ValidTxnList
[ https://issues.apache.org/jira/browse/HIVE-21226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-21226: -- Comment: was deleted (was: [~vgumashta], [~pvary] do you know if this is still relevant? My understanding was that we are now using ValidTxnWriteIdList, so there won't be anything related to ReadOnly transactions. Do I miss something? Is there some particular scenario where we would benefit from this optimization? ) > Exclude read-only transactions from ValidTxnList > > > Key: HIVE-21226 > URL: https://issues.apache.org/jira/browse/HIVE-21226 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Denys Kuzmenko >Priority: Major > > Once HIVE-21114 is done, we should make sure that ValidTxnList doesn't > contain any read-only txns in the exceptions list since by definition there > is no data tagged with such txnid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22240) Function percentile_cont fails when array parameter passed
[
https://issues.apache.org/jira/browse/HIVE-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22240:
--
Status: Open (was: Patch Available)
> Function percentile_cont fails when array parameter passed
> --
>
> Key: HIVE-22240
> URL: https://issues.apache.org/jira/browse/HIVE-22240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22240.1.patch, HIVE-22240.2.patch,
> HIVE-22240.3.patch, HIVE-22240.3.patch, HIVE-22240.4.patch,
> HIVE-22240.4.patch, HIVE-22240.4.patch, HIVE-22240.4.patch, HIVE-22240.4.patch
>
>
> {code}
> SELECT
> percentile_cont(array(0.2, 0.5, 0.9)) WITHIN GROUP (ORDER BY value)
> FROM t_test;
> {code}
> hive.log:
> {code}
> 2019-09-24T21:00:43,203 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:793)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> ... 11 more
> Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast
> to org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileCont$PercentileContEvaluator.iterate(GenericUDAFPercentileCont.java:259)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:214)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:639)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:814)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:720)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:788)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22240) Function percentile_cont fails when array parameter passed
[
https://issues.apache.org/jira/browse/HIVE-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22240:
--
Status: Patch Available (was: Open)
> Function percentile_cont fails when array parameter passed
> --
>
> Key: HIVE-22240
> URL: https://issues.apache.org/jira/browse/HIVE-22240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22240.1.patch, HIVE-22240.2.patch,
> HIVE-22240.3.patch, HIVE-22240.3.patch, HIVE-22240.4.patch,
> HIVE-22240.4.patch, HIVE-22240.4.patch, HIVE-22240.4.patch, HIVE-22240.4.patch
>
>
> {code}
> SELECT
> percentile_cont(array(0.2, 0.5, 0.9)) WITHIN GROUP (ORDER BY value)
> FROM t_test;
> {code}
> hive.log:
> {code}
> 2019-09-24T21:00:43,203 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:793)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> ... 11 more
> Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast
> to org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileCont$PercentileContEvaluator.iterate(GenericUDAFPercentileCont.java:259)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:214)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:639)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:814)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:720)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:788)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22240) Function percentile_cont fails when array parameter passed
[
https://issues.apache.org/jira/browse/HIVE-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22240:
--
Attachment: HIVE-22240.4.patch
> Function percentile_cont fails when array parameter passed
> --
>
> Key: HIVE-22240
> URL: https://issues.apache.org/jira/browse/HIVE-22240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22240.1.patch, HIVE-22240.2.patch,
> HIVE-22240.3.patch, HIVE-22240.3.patch, HIVE-22240.4.patch,
> HIVE-22240.4.patch, HIVE-22240.4.patch, HIVE-22240.4.patch, HIVE-22240.4.patch
>
>
> {code}
> SELECT
> percentile_cont(array(0.2, 0.5, 0.9)) WITHIN GROUP (ORDER BY value)
> FROM t_test;
> {code}
> hive.log:
> {code}
> 2019-09-24T21:00:43,203 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:793)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> ... 11 more
> Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast
> to org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileCont$PercentileContEvaluator.iterate(GenericUDAFPercentileCont.java:259)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:214)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:639)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:814)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:720)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:788)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22363) ReduceDeduplication may leave an invalid GroupByOperator behind in some cases
[
https://issues.apache.org/jira/browse/HIVE-22363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-22363:
Attachment: HIVE-22363.02.patch
> ReduceDeduplication may leave an invalid GroupByOperator behind in some cases
> -
>
> Key: HIVE-22363
> URL: https://issues.apache.org/jira/browse/HIVE-22363
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 3.1.2
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22363.01.patch, HIVE-22363.02.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> since HIVE-11387 reducededup may traverse {{GroupByOperators}} [as
> well|https://github.com/apache/hive/blob/c6626edb65c2cd00576647e54db1995628fe64da/ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationUtilities.java#L244]
> But the removal logic only removes the first parent; so if there is some
> other operator (a FIL in this case) between the sink and the gby - the
> removal may not happen
> [here|https://github.com/apache/hive/blob/c6626edb65c2cd00576647e54db1995628fe64da/ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationUtilities.java#L458]
> {code}
> set hive.cbo.enable=false;
> drop table if exists xl1;
> create table xl1 as
> select '1' as mdl_yr_desc, 2 as seq_no,'3' as opt_desc1,4 as opt_desc,1 as
> row_num;
> explain
> select trim(base.mdl_yr_desc) mdl_yr_desc, trim(base.opt_desc) opt_desc
> from
> (
> SELECT trim(mdl_yr_desc) mdl_yr_desc, concat_ws(' ',
> collect_set(trim(opt_desc1))) AS opt_desc
> from
> (
> select t14304.*
> from
> (
> select * from xl1
> ) t14304
> where row_num = 1
> order by trim(mdl_yr_desc), cast(seq_no as int) asc
> ) x
> group by trim(mdl_yr_desc)
> ) base
> inner join
> (
> select 1 as v
> ) dedup
> on trim(base.mdl_yr_desc) != dedup.v
> group by trim(base.mdl_yr_desc), trim(base.opt_desc) ;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22346) Yetus is failing rat check
[
https://issues.apache.org/jira/browse/HIVE-22346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-22346:
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
pushed to master.Thank you [~kkasa]
> Yetus is failing rat check
> --
>
> Key: HIVE-22346
> URL: https://issues.apache.org/jira/browse/HIVE-22346
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22346.1.patch, HIVE-22346.2.patch,
> HIVE-22346.2.patch, HIVE-22346.2.patch, HIVE-22346.2.patch, HIVE-22346.2.patch
>
>
> {code:java}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header:
> !?
> /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18996/standalone-metastore/metastore-server/src/test/resources/ldap/ad.example.com.ldif
> !?
> /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18996/standalone-metastore/metastore-server/src/test/resources/ldap/example.com.ldif
> !?
> /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18996/standalone-metastore/metastore-server/src/test/resources/ldap/microsoft.schema.ldif
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22376) Cancelled query still prints exception if it was stuck in waiting for lock
[
https://issues.apache.org/jira/browse/HIVE-22376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary reassigned HIVE-22376:
-
Assignee: Laszlo Pinter (was: Peter Vary)
> Cancelled query still prints exception if it was stuck in waiting for lock
> --
>
> Key: HIVE-22376
> URL: https://issues.apache.org/jira/browse/HIVE-22376
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Peter Vary
>Assignee: Laszlo Pinter
>Priority: Major
>
> The query waits for locks, then cancelled.
> It prints this to the logs, which is unnecessary and missleading:
> {code}
> apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:326)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:344)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: NoSuchLockException(message:No such lock lockid:272)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$check_lock_result$check_lock_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$check_lock_result$check_lock_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$check_lock_result.read(ThriftHiveMetastore.java)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_check_lock(ThriftHiveMetastore.java:5730)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.check_lock(ThriftHiveMetastore.java:5717)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.checkLock(HiveMetaStoreClient.java:3128)
> at sun.reflect.GeneratedMethodAccessor351.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> at com.sun.proxy.$Proxy59.checkLock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor351.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:)
> at com.sun.proxy.$Proxy59.checkLock(Unknown Source)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbLockManager.lock(DbLockManager.java:115)
> ... 25 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22346) Yetus is failing rat check
[
https://issues.apache.org/jira/browse/HIVE-22346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955870#comment-16955870
]
Hive QA commented on HIVE-22346:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12983568/HIVE-22346.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17545 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19083/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19083/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19083/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12983568 - PreCommit-HIVE-Build
> Yetus is failing rat check
> --
>
> Key: HIVE-22346
> URL: https://issues.apache.org/jira/browse/HIVE-22346
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22346.1.patch, HIVE-22346.2.patch,
> HIVE-22346.2.patch, HIVE-22346.2.patch, HIVE-22346.2.patch, HIVE-22346.2.patch
>
>
> {code:java}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header:
> !?
> /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18996/standalone-metastore/metastore-server/src/test/resources/ldap/ad.example.com.ldif
> !?
> /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18996/standalone-metastore/metastore-server/src/test/resources/ldap/example.com.ldif
> !?
> /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18996/standalone-metastore/metastore-server/src/test/resources/ldap/microsoft.schema.ldif
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22375) ObjectStore.lockNotificationSequenceForUpdate is leaking query in case of error
[
https://issues.apache.org/jira/browse/HIVE-22375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marta Kuczora updated HIVE-22375:
-
Status: Patch Available (was: Open)
> ObjectStore.lockNotificationSequenceForUpdate is leaking query in case of
> error
> ---
>
> Key: HIVE-22375
> URL: https://issues.apache.org/jira/browse/HIVE-22375
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Minor
> Attachments: HIVE-22375.1.patch
>
>
> In the ObjectStore.lockNotificationSequenceForUpdate method, the query
> doesn't get closed if an error occur:
> {noformat}
> private void lockNotificationSequenceForUpdate() throws MetaException {
> if (sqlGenerator.getDbProduct() == DatabaseProduct.DERBY && directSql !=
> null) {
> // Derby doesn't allow FOR UPDATE to lock the row being selected (See
> https://db.apache
> // .org/derby/docs/10.1/ref/rrefsqlj31783.html) . So lock the whole
> table. Since there's
> // only one row in the table, this shouldn't cause any performance
> degradation.
> new RetryingExecutor(conf, () -> {
> directSql.lockDbTable("NOTIFICATION_SEQUENCE");
> }).run();
> } else {
> String selectQuery = "select \"NEXT_EVENT_ID\" from
> \"NOTIFICATION_SEQUENCE\"";
> String lockingQuery = sqlGenerator.addForUpdateClause(selectQuery);
> new RetryingExecutor(conf, () -> {
> prepareQuotes();
> Query query = pm.newQuery("javax.jdo.query.SQL", lockingQuery);
> query.setUnique(true);
> // only need to execute it to get db Lock
> query.execute();
> query.closeAll();
> }).run();
> }
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22375) ObjectStore.lockNotificationSequenceForUpdate is leaking query in case of error
[
https://issues.apache.org/jira/browse/HIVE-22375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marta Kuczora updated HIVE-22375:
-
Attachment: HIVE-22375.1.patch
> ObjectStore.lockNotificationSequenceForUpdate is leaking query in case of
> error
> ---
>
> Key: HIVE-22375
> URL: https://issues.apache.org/jira/browse/HIVE-22375
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Minor
> Attachments: HIVE-22375.1.patch
>
>
> In the ObjectStore.lockNotificationSequenceForUpdate method, the query
> doesn't get closed if an error occur:
> {noformat}
> private void lockNotificationSequenceForUpdate() throws MetaException {
> if (sqlGenerator.getDbProduct() == DatabaseProduct.DERBY && directSql !=
> null) {
> // Derby doesn't allow FOR UPDATE to lock the row being selected (See
> https://db.apache
> // .org/derby/docs/10.1/ref/rrefsqlj31783.html) . So lock the whole
> table. Since there's
> // only one row in the table, this shouldn't cause any performance
> degradation.
> new RetryingExecutor(conf, () -> {
> directSql.lockDbTable("NOTIFICATION_SEQUENCE");
> }).run();
> } else {
> String selectQuery = "select \"NEXT_EVENT_ID\" from
> \"NOTIFICATION_SEQUENCE\"";
> String lockingQuery = sqlGenerator.addForUpdateClause(selectQuery);
> new RetryingExecutor(conf, () -> {
> prepareQuotes();
> Query query = pm.newQuery("javax.jdo.query.SQL", lockingQuery);
> query.setUnique(true);
> // only need to execute it to get db Lock
> query.execute();
> query.closeAll();
> }).run();
> }
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22376) Cancelled query still prints exception if it was stuck in waiting for lock
[
https://issues.apache.org/jira/browse/HIVE-22376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary reassigned HIVE-22376:
-
> Cancelled query still prints exception if it was stuck in waiting for lock
> --
>
> Key: HIVE-22376
> URL: https://issues.apache.org/jira/browse/HIVE-22376
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
>
> The query waits for locks, then cancelled.
> It prints this to the logs, which is unnecessary and missleading:
> {code}
> apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:326)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:344)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: NoSuchLockException(message:No such lock lockid:272)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$check_lock_result$check_lock_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$check_lock_result$check_lock_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$check_lock_result.read(ThriftHiveMetastore.java)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_check_lock(ThriftHiveMetastore.java:5730)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.check_lock(ThriftHiveMetastore.java:5717)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.checkLock(HiveMetaStoreClient.java:3128)
> at sun.reflect.GeneratedMethodAccessor351.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> at com.sun.proxy.$Proxy59.checkLock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor351.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:)
> at com.sun.proxy.$Proxy59.checkLock(Unknown Source)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbLockManager.lock(DbLockManager.java:115)
> ... 25 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22315) Support Decimal64 column division with decimal64 scalar
[ https://issues.apache.org/jira/browse/HIVE-22315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22315: Status: Open (was: Patch Available) > Support Decimal64 column division with decimal64 scalar > --- > > Key: HIVE-22315 > URL: https://issues.apache.org/jira/browse/HIVE-22315 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22315.1.patch, HIVE-22315.2.patch, > HIVE-22315.3.patch, HIVE-22315.4.patch, HIVE-22315.6.patch, HIVE-22315.7.patch > > > Currently division operation is not supported for Decimal64 column. This Jira > will take care of supporting decimal64 column division with a decimal64 > scalar. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22315) Support Decimal64 column division with decimal64 scalar
[ https://issues.apache.org/jira/browse/HIVE-22315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22315: Attachment: HIVE-22315.7.patch Status: Patch Available (was: Open) > Support Decimal64 column division with decimal64 scalar > --- > > Key: HIVE-22315 > URL: https://issues.apache.org/jira/browse/HIVE-22315 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22315.1.patch, HIVE-22315.2.patch, > HIVE-22315.3.patch, HIVE-22315.4.patch, HIVE-22315.6.patch, HIVE-22315.7.patch > > > Currently division operation is not supported for Decimal64 column. This Jira > will take care of supporting decimal64 column division with a decimal64 > scalar. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22375) ObjectStore.lockNotificationSequenceForUpdate is leaking query in case of error
[
https://issues.apache.org/jira/browse/HIVE-22375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marta Kuczora reassigned HIVE-22375:
> ObjectStore.lockNotificationSequenceForUpdate is leaking query in case of
> error
> ---
>
> Key: HIVE-22375
> URL: https://issues.apache.org/jira/browse/HIVE-22375
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Minor
>
> In the ObjectStore.lockNotificationSequenceForUpdate method, the query
> doesn't get closed if an error occur:
> {noformat}
> private void lockNotificationSequenceForUpdate() throws MetaException {
> if (sqlGenerator.getDbProduct() == DatabaseProduct.DERBY && directSql !=
> null) {
> new RetryingExecutor(conf, () -> {
> directSql.lockDbTable("NOTIFICATION_SEQUENCE");
> }).run();
> } else {
> String selectQuery = "select \"NEXT_EVENT_ID\" from
> \"NOTIFICATION_SEQUENCE\"";
> String lockingQuery = sqlGenerator.addForUpdateClause(selectQuery);
> new RetryingExecutor(conf, () -> {
> prepareQuotes();
> Query query = pm.newQuery("javax.jdo.query.SQL", lockingQuery);
> query.setUnique(true);
> // only need to execute it to get db Lock
> query.execute();
> query.closeAll();
> }).run();
> }
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22375) ObjectStore.lockNotificationSequenceForUpdate is leaking query in case of error
[
https://issues.apache.org/jira/browse/HIVE-22375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marta Kuczora updated HIVE-22375:
-
Description:
In the ObjectStore.lockNotificationSequenceForUpdate method, the query doesn't
get closed if an error occur:
{noformat}
private void lockNotificationSequenceForUpdate() throws MetaException {
if (sqlGenerator.getDbProduct() == DatabaseProduct.DERBY && directSql !=
null) {
// Derby doesn't allow FOR UPDATE to lock the row being selected (See
https://db.apache
// .org/derby/docs/10.1/ref/rrefsqlj31783.html) . So lock the whole
table. Since there's
// only one row in the table, this shouldn't cause any performance
degradation.
new RetryingExecutor(conf, () -> {
directSql.lockDbTable("NOTIFICATION_SEQUENCE");
}).run();
} else {
String selectQuery = "select \"NEXT_EVENT_ID\" from
\"NOTIFICATION_SEQUENCE\"";
String lockingQuery = sqlGenerator.addForUpdateClause(selectQuery);
new RetryingExecutor(conf, () -> {
prepareQuotes();
Query query = pm.newQuery("javax.jdo.query.SQL", lockingQuery);
query.setUnique(true);
// only need to execute it to get db Lock
query.execute();
query.closeAll();
}).run();
}
}
{noformat}
was:
In the ObjectStore.lockNotificationSequenceForUpdate method, the query doesn't
get closed if an error occur:
{noformat}
private void lockNotificationSequenceForUpdate() throws MetaException {
if (sqlGenerator.getDbProduct() == DatabaseProduct.DERBY && directSql != null)
{
new RetryingExecutor(conf, () -> {
directSql.lockDbTable("NOTIFICATION_SEQUENCE");
}).run();
} else {
String selectQuery = "select \"NEXT_EVENT_ID\" from \"NOTIFICATION_SEQUENCE\"";
String lockingQuery = sqlGenerator.addForUpdateClause(selectQuery);
new RetryingExecutor(conf, () -> {
prepareQuotes();
Query query = pm.newQuery("javax.jdo.query.SQL", lockingQuery);
query.setUnique(true);
// only need to execute it to get db Lock
query.execute();
query.closeAll();
}).run();
}
}
{noformat}
> ObjectStore.lockNotificationSequenceForUpdate is leaking query in case of
> error
> ---
>
> Key: HIVE-22375
> URL: https://issues.apache.org/jira/browse/HIVE-22375
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Minor
>
> In the ObjectStore.lockNotificationSequenceForUpdate method, the query
> doesn't get closed if an error occur:
> {noformat}
> private void lockNotificationSequenceForUpdate() throws MetaException {
> if (sqlGenerator.getDbProduct() == DatabaseProduct.DERBY && directSql !=
> null) {
> // Derby doesn't allow FOR UPDATE to lock the row being selected (See
> https://db.apache
> // .org/derby/docs/10.1/ref/rrefsqlj31783.html) . So lock the whole
> table. Since there's
> // only one row in the table, this shouldn't cause any performance
> degradation.
> new RetryingExecutor(conf, () -> {
> directSql.lockDbTable("NOTIFICATION_SEQUENCE");
> }).run();
> } else {
> String selectQuery = "select \"NEXT_EVENT_ID\" from
> \"NOTIFICATION_SEQUENCE\"";
> String lockingQuery = sqlGenerator.addForUpdateClause(selectQuery);
> new RetryingExecutor(conf, () -> {
> prepareQuotes();
> Query query = pm.newQuery("javax.jdo.query.SQL", lockingQuery);
> query.setUnique(true);
> // only need to execute it to get db Lock
> query.execute();
> query.closeAll();
> }).run();
> }
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22346) Yetus is failing rat check
[
https://issues.apache.org/jira/browse/HIVE-22346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955854#comment-16955854
]
Hive QA commented on HIVE-22346:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19083/dev-support/hive-personality.sh
|
| git revision | master / c1e5fa3 |
| Default Java | 1.8.0_111 |
| modules | C: standalone-metastore/metastore-server . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19083/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Yetus is failing rat check
> --
>
> Key: HIVE-22346
> URL: https://issues.apache.org/jira/browse/HIVE-22346
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22346.1.patch, HIVE-22346.2.patch,
> HIVE-22346.2.patch, HIVE-22346.2.patch, HIVE-22346.2.patch, HIVE-22346.2.patch
>
>
> {code:java}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header:
> !?
> /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18996/standalone-metastore/metastore-server/src/test/resources/ldap/ad.example.com.ldif
> !?
> /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18996/standalone-metastore/metastore-server/src/test/resources/ldap/example.com.ldif
> !?
> /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18996/standalone-metastore/metastore-server/src/test/resources/ldap/microsoft.schema.ldif
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java
[
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955823#comment-16955823
]
Peter Vary commented on HIVE-21246:
---
[~belugabehr]: Aren't you a committer? You can commit the change when you got
+1 from another committer :)
> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21246.1.patch, HIVE-21246.1.patch,
> HIVE-21246.2.patch
>
>
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
> make it easier to get rid of later. It's currently buried in
> {{PlanUtils.java}}.
> A SerDe and a boolean flag gets passed into these methods. If the flag is
> set to true, the specified SerDe is overwritten and assigned to
> {{DelimitedJSONSerDe}}. This is not documented anywhere and it's a weird
> thing to do, just pass in the required SerDe from the start instead of
> sending the wrong SerDe and a flag to overwrite it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-15079) Hive cannot read Parquet string timetamps as TIMESTAMP data type
[
https://issues.apache.org/jira/browse/HIVE-15079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955820#comment-16955820
]
Ganesha Shreedhara commented on HIVE-15079:
---
I have an another instance similar to this where the data is in long format and
table schema has column type as timestamp.
This works with ORC but throws ClassCastException when parquet is used. Long
type data can be converted to timestamp (Eg: new Timestamp(longValue)).
Do we have any plans to support automatic type conversion for parquet file
formats in hive?
> Hive cannot read Parquet string timetamps as TIMESTAMP data type
>
>
> Key: HIVE-15079
> URL: https://issues.apache.org/jira/browse/HIVE-15079
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Sergio Peña
>Priority: Major
>
> The Hive Wiki for timestamps specifies that strings timestamps can be read by
> Hive.
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-Timestamps
> {noformat}
> Supported conversions:
> Integer numeric types: Interpreted as UNIX timestamp in seconds
> Floating point numeric types: Interpreted as UNIX timestamp in seconds with
> decimal precision
> Strings: JDBC compliant java.sql.Timestamp format "-MM-DD
> HH:MM:SS.f" (9 decimal place precision)
> {noformat}
> This works fine with Text table formats, but when Parquet is used, then it
> throws the following exception:
> {noformat}
> java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to
> org.apache.hadoop.hive.serde2.io.TimestampWritable
> {noformat}
> How to reproduce
> {noformat}
> > create table t1 (id int, time string) stored as parquet;
> > insert into table t1 values (1,'2016-07-17 14:42:18');
> > alter table t1 replace columns (id int, time timestamp);
> > select * from t1
> {noformat}
> The above example will run fine if you use a TEXT format instead of PARQUET.
> This issue was raised on PARQUET-723
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22240) Function percentile_cont fails when array parameter passed
[
https://issues.apache.org/jira/browse/HIVE-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955819#comment-16955819
]
Hive QA commented on HIVE-22240:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12983569/HIVE-22240.4.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17545 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.testCliDriver[spark_submit_negative_executor_cores]
(batchId=301)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19082/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19082/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19082/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12983569 - PreCommit-HIVE-Build
> Function percentile_cont fails when array parameter passed
> --
>
> Key: HIVE-22240
> URL: https://issues.apache.org/jira/browse/HIVE-22240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22240.1.patch, HIVE-22240.2.patch,
> HIVE-22240.3.patch, HIVE-22240.3.patch, HIVE-22240.4.patch,
> HIVE-22240.4.patch, HIVE-22240.4.patch, HIVE-22240.4.patch
>
>
> {code}
> SELECT
> percentile_cont(array(0.2, 0.5, 0.9)) WITHIN GROUP (ORDER BY value)
> FROM t_test;
> {code}
> hive.log:
> {code}
> 2019-09-24T21:00:43,203 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:793)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> ... 11 more
> Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast
> to org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileCont$PercentileContEvaluator.iterate(GenericUDAFPercentileCont.java:259)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:214)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:639)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:814)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:720)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:788)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
[jira] [Updated] (HIVE-22360) MultiSerDe returns wrong results in last column when the loaded file has more columns than those in table schema
[
https://issues.apache.org/jira/browse/HIVE-22360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shubham Chaurasia updated HIVE-22360:
-
Affects Version/s: (was: 3.1.2)
4.0.0
> MultiSerDe returns wrong results in last column when the loaded file has more
> columns than those in table schema
>
>
> Key: HIVE-22360
> URL: https://issues.apache.org/jira/browse/HIVE-22360
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
>
> Repro steps:
> Input file:
> {code}
> 1^,1^,^,0^,0^,0
> 2^,1^,^,0^,1^,0
> 3^,1^,^,0^,0^,0
> 4^,1^,^,0^,1^,0
> {code}
> Queries:
> {code}
> CREATE TABLE n2(colA int, colB tinyint, colC timestamp, colD smallint, colE
> smallint) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' WITH
> SERDEPROPERTIES ("field.delim"="^,")STORED AS TEXTFILE;
> LOAD DATA LOCAL INPATH '/Users/schaurasia/Documents/input_6_cols.csv'
> OVERWRITE INTO TABLE n2;
> select * from n2;
> // wrong last column results here.
> +--+--+--+--+--+
> | n2.cola | n2.colb | n2.colc | n2.cold | n2.cole |
> +--+--+--+--+--+
> | 1| 1| NULL | 0| NULL |
> | 2| 1| NULL | 0| NULL |
> | 3| 1| NULL | 0| NULL |
> | 4| 1| NULL | 0| NULL |
> +--+--+--+--+--+
> {code}
> Cause:
> In multi-serde parsing, the total length calculation here:
> https://github.com/apache/hive/blob/rel/release-3.1.2/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java#L308
> does not take extra fields into account.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22240) Function percentile_cont fails when array parameter passed
[
https://issues.apache.org/jira/browse/HIVE-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955784#comment-16955784
]
Hive QA commented on HIVE-22240:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 1547 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19082/dev-support/hive-personality.sh
|
| git revision | master / c1e5fa3 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19082/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19082/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Function percentile_cont fails when array parameter passed
> --
>
> Key: HIVE-22240
> URL: https://issues.apache.org/jira/browse/HIVE-22240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22240.1.patch, HIVE-22240.2.patch,
> HIVE-22240.3.patch, HIVE-22240.3.patch, HIVE-22240.4.patch,
> HIVE-22240.4.patch, HIVE-22240.4.patch, HIVE-22240.4.patch
>
>
> {code}
> SELECT
> percentile_cont(array(0.2, 0.5, 0.9)) WITHIN GROUP (ORDER BY value)
> FROM t_test;
> {code}
> hive.log:
> {code}
> 2019-09-24T21:00:43,203 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
