[jira] [Resolved] (HIVE-25443) Arrow SerDe Cannot serialize/deserialize complex data types When there are more than 1024 values
[
https://issues.apache.org/jira/browse/HIVE-25443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich resolved HIVE-25443.
-
Resolution: Fixed
merged into master. Thank you [~srahman]!
> Arrow SerDe Cannot serialize/deserialize complex data types When there are
> more than 1024 values
>
>
> Key: HIVE-25443
> URL: https://issues.apache.org/jira/browse/HIVE-25443
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.1.0, 3.0.0, 3.1.1, 3.1.2
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Complex data types like MAP, STRUCT cannot be serialized/deserialzed using
> Arrow SerDe when there are more than 1024 values. This happens due to
> ColumnVector always being initialized with a size of 1024.
> Issue #1 :
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/ArrowColumnarBatchSerDe.java#L213
> Issue #2 :
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/ArrowColumnarBatchSerDe.java#L215
> Sample unit test to reproduce the case in TestArrowColumnarBatchSerDe :
> {code:java}
> @Test
>public void testListBooleanWithMoreThan1024Values() throws SerDeException {
> String[][] schema = {
> {"boolean_list", "array"},
> };
>
> Object[][] rows = new Object[1025][1];
> for (int i = 0; i < 1025; i++) {
>rows[i][0] = new BooleanWritable(true);
> }
>
> initAndSerializeAndDeserialize(schema, toList(rows));
>}
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25443) Arrow SerDe Cannot serialize/deserialize complex data types When there are more than 1024 values
[
https://issues.apache.org/jira/browse/HIVE-25443?focusedWorklogId=683127=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-683127
]
ASF GitHub Bot logged work on HIVE-25443:
-
Author: ASF GitHub Bot
Created on: 18/Nov/21 07:30
Start Date: 18/Nov/21 07:30
Worklog Time Spent: 10m
Work Description: kgyrtkirk merged pull request #2581:
URL: https://github.com/apache/hive/pull/2581
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 683127)
Time Spent: 40m (was: 0.5h)
> Arrow SerDe Cannot serialize/deserialize complex data types When there are
> more than 1024 values
>
>
> Key: HIVE-25443
> URL: https://issues.apache.org/jira/browse/HIVE-25443
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.1.0, 3.0.0, 3.1.1, 3.1.2
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Complex data types like MAP, STRUCT cannot be serialized/deserialzed using
> Arrow SerDe when there are more than 1024 values. This happens due to
> ColumnVector always being initialized with a size of 1024.
> Issue #1 :
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/ArrowColumnarBatchSerDe.java#L213
> Issue #2 :
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/ArrowColumnarBatchSerDe.java#L215
> Sample unit test to reproduce the case in TestArrowColumnarBatchSerDe :
> {code:java}
> @Test
>public void testListBooleanWithMoreThan1024Values() throws SerDeException {
> String[][] schema = {
> {"boolean_list", "array"},
> };
>
> Object[][] rows = new Object[1025][1];
> for (int i = 0; i < 1025; i++) {
>rows[i][0] = new BooleanWritable(true);
> }
>
> initAndSerializeAndDeserialize(schema, toList(rows));
>}
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-25714) Some tests are flaky because docker is not able to start in 5 seconds
[
https://issues.apache.org/jira/browse/HIVE-25714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445693#comment-17445693
]
Zoltan Haindrich commented on HIVE-25714:
-
ok; I'm not a big fan of that - but of course we can try that...actually;
adding some log lines could also help to know how much stuff we are reading
from docker logs
> Some tests are flaky because docker is not able to start in 5 seconds
> -
>
> Key: HIVE-25714
> URL: https://issues.apache.org/jira/browse/HIVE-25714
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> there are some testruns failing with; and on the test site multiple pods are
> running in parallel - its not an ideal environment for tight deadlines
> {code}
> Unexpected exception java.lang.RuntimeException: Process docker failed to run
> in 5 seconds
> at
> org.apache.hadoop.hive.ql.externalDB.AbstractExternalDB.runCmd(AbstractExternalDB.java:92)
> at
> org.apache.hadoop.hive.ql.externalDB.AbstractExternalDB.launchDockerContainer(AbstractExternalDB.java:123)
> at
> org.apache.hadoop.hive.ql.qoption.QTestDatabaseHandler.beforeTest(QTestDatabaseHandler.java:111)
> at
> org.apache.hadoop.hive.ql.qoption.QTestOptionDispatcher.beforeTest(QTestOptionDispatcher.java:79)
> {code}
> http://ci.hive.apache.org/job/hive-precommit/job/PR-1674/4/testReport/junit/org.apache.hadoop.hive.cli.split19/TestMiniLlapLocalCliDriver/Testing___split_14___PostProcess___testCliDriver_qt_database_all_/
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25095) Beeline/hive -e command can't deal with query with trailing quote
[
https://issues.apache.org/jira/browse/HIVE-25095?focusedWorklogId=683122=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-683122
]
ASF GitHub Bot logged work on HIVE-25095:
-
Author: ASF GitHub Bot
Created on: 18/Nov/21 07:22
Start Date: 18/Nov/21 07:22
Worklog Time Spent: 10m
Work Description: kgyrtkirk merged pull request #2526:
URL: https://github.com/apache/hive/pull/2526
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 683122)
Time Spent: 1h 20m (was: 1h 10m)
> Beeline/hive -e command can't deal with query with trailing quote
> -
>
> Key: HIVE-25095
> URL: https://issues.apache.org/jira/browse/HIVE-25095
> Project: Hive
> Issue Type: Bug
>Reporter: Robbie Zhang
>Assignee: Robbie Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The command
> {code:java}
> hive -e 'select "hive"'{code}
> and
> {code:java}
> beeline -e 'select "hive"'{code}
> fail with such error:
> {code:java}
> Error: Error while compiling statement: FAILED: ParseException line 1:12
> character '' not supported here (state=42000,code=4){code}
> The reason is that org.apache.commons.cli.Util.stripLeadingAndTrailingQuotes
> in commons-cli-1.2.jar strips the trailing quote so the query string is
> changed to
> {code:java}
> select "hive{code}
> This bug is fixed in commons-cli-1.3.1 and commons-cli-1.4.jar. The
> workaround is to overwrite commons-cli-1.2.jar with commons-cli-1.3.1 or
> commons-cli-1.4.jar.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25095) Beeline/hive -e command can't deal with query with trailing quote
[
https://issues.apache.org/jira/browse/HIVE-25095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich resolved HIVE-25095.
-
Fix Version/s: 4.0.0
Resolution: Fixed
merged into master. Thank you [~robbiezhang]!
> Beeline/hive -e command can't deal with query with trailing quote
> -
>
> Key: HIVE-25095
> URL: https://issues.apache.org/jira/browse/HIVE-25095
> Project: Hive
> Issue Type: Bug
>Reporter: Robbie Zhang
>Assignee: Robbie Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The command
> {code:java}
> hive -e 'select "hive"'{code}
> and
> {code:java}
> beeline -e 'select "hive"'{code}
> fail with such error:
> {code:java}
> Error: Error while compiling statement: FAILED: ParseException line 1:12
> character '' not supported here (state=42000,code=4){code}
> The reason is that org.apache.commons.cli.Util.stripLeadingAndTrailingQuotes
> in commons-cli-1.2.jar strips the trailing quote so the query string is
> changed to
> {code:java}
> select "hive{code}
> This bug is fixed in commons-cli-1.3.1 and commons-cli-1.4.jar. The
> workaround is to overwrite commons-cli-1.2.jar with commons-cli-1.3.1 or
> commons-cli-1.4.jar.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-25722) Compaction fails if there are empty buckets.
[ https://issues.apache.org/jira/browse/HIVE-25722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arko Sharma reassigned HIVE-25722: -- > Compaction fails if there are empty buckets. > > > Key: HIVE-25722 > URL: https://issues.apache.org/jira/browse/HIVE-25722 > Project: Hive > Issue Type: Bug >Reporter: Arko Sharma >Assignee: Arko Sharma >Priority: Major > > Compaction fails if there are empty buckets. > This patch detects and deletes empty buckets before launching compaction in > order to protect it from failing. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25721) Outer join result is wrong
[
https://issues.apache.org/jira/browse/HIVE-25721?focusedWorklogId=683113=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-683113
]
ASF GitHub Bot logged work on HIVE-25721:
-
Author: ASF GitHub Bot
Created on: 18/Nov/21 06:39
Start Date: 18/Nov/21 06:39
Worklog Time Spent: 10m
Work Description: SparksFyz commented on pull request #2798:
URL: https://github.com/apache/hive/pull/2798#issuecomment-972581163
@maheshk114 @zabetak Could you please take a look when you have a chance?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 683113)
Time Spent: 20m (was: 10m)
> Outer join result is wrong
> --
>
> Key: HIVE-25721
> URL: https://issues.apache.org/jira/browse/HIVE-25721
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: All Versions
>Reporter: Yizhen Fan
>Assignee: Yizhen Fan
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-25721.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Outer Join results is wrong, here is a left join case.
> select b.fields from a left join b on a.key=b.key and a.filter=xxx
> there are some necessary condition to produce this problem:
> # `select` clause only contains right table fields
> # `on` clause contains left table condition, and this condition can filter
> records
> h3. cause:
> candidateStorage[tag].addRow(value); // CommonMergeJoinOperator.process
> row of left table cannot be add into row container because tblDesc of left
> table is null, while left table data can not be ignored in this case.
> h3. Reproducible steps are mentioned below.
>
> set hive.auto.convert.join=false;
> create table t_smj_left (key string, value int);
> insert into t_smj_left values
> ('key1', 1),
> ('key1', 2);
> create table t_smj_right (key string, value int);
> insert into t_smj_right values
> ('key1', 1);
> select
> t2.value
> from t_smj_left t1
> left join t_smj_right t2 on t1.key=t2.key and t1.value=2;
>
> Result:
> +
> NULL
> NULL
> +
> Expected Output:
> +
> 1
> NULL
> +
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25721) Outer join result is wrong
[
https://issues.apache.org/jira/browse/HIVE-25721?focusedWorklogId=683112=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-683112
]
ASF GitHub Bot logged work on HIVE-25721:
-
Author: ASF GitHub Bot
Created on: 18/Nov/21 06:37
Start Date: 18/Nov/21 06:37
Worklog Time Spent: 10m
Work Description: SparksFyz opened a new pull request #2798:
URL: https://github.com/apache/hive/pull/2798
For CommonMergeJoinOperator, the tableDesc will be null in the case that all
columns in that table is not used. The tableDesc is null lead to using a dummy
row to denote all rows in that table instead of adding row into rowContainer.
However, rows in that table cannot be ignored when it contains filter in on
clause.
Reproduced steps are commented in the following ticket.
https://issues.apache.org/jira/browse/HIVE-25721
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 683112)
Remaining Estimate: 0h
Time Spent: 10m
> Outer join result is wrong
> --
>
> Key: HIVE-25721
> URL: https://issues.apache.org/jira/browse/HIVE-25721
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: All Versions
>Reporter: Yizhen Fan
>Assignee: Yizhen Fan
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-25721.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Outer Join results is wrong, here is a left join case.
> select b.fields from a left join b on a.key=b.key and a.filter=xxx
> there are some necessary condition to produce this problem:
> # `select` clause only contains right table fields
> # `on` clause contains left table condition, and this condition can filter
> records
> h3. cause:
> candidateStorage[tag].addRow(value); // CommonMergeJoinOperator.process
> row of left table cannot be add into row container because tblDesc of left
> table is null, while left table data can not be ignored in this case.
> h3. Reproducible steps are mentioned below.
>
> set hive.auto.convert.join=false;
> create table t_smj_left (key string, value int);
> insert into t_smj_left values
> ('key1', 1),
> ('key1', 2);
> create table t_smj_right (key string, value int);
> insert into t_smj_right values
> ('key1', 1);
> select
> t2.value
> from t_smj_left t1
> left join t_smj_right t2 on t1.key=t2.key and t1.value=2;
>
> Result:
> +
> NULL
> NULL
> +
> Expected Output:
> +
> 1
> NULL
> +
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25721) Outer join result is wrong
[
https://issues.apache.org/jira/browse/HIVE-25721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-25721:
--
Labels: pull-request-available (was: )
> Outer join result is wrong
> --
>
> Key: HIVE-25721
> URL: https://issues.apache.org/jira/browse/HIVE-25721
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: All Versions
>Reporter: Yizhen Fan
>Assignee: Yizhen Fan
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-25721.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Outer Join results is wrong, here is a left join case.
> select b.fields from a left join b on a.key=b.key and a.filter=xxx
> there are some necessary condition to produce this problem:
> # `select` clause only contains right table fields
> # `on` clause contains left table condition, and this condition can filter
> records
> h3. cause:
> candidateStorage[tag].addRow(value); // CommonMergeJoinOperator.process
> row of left table cannot be add into row container because tblDesc of left
> table is null, while left table data can not be ignored in this case.
> h3. Reproducible steps are mentioned below.
>
> set hive.auto.convert.join=false;
> create table t_smj_left (key string, value int);
> insert into t_smj_left values
> ('key1', 1),
> ('key1', 2);
> create table t_smj_right (key string, value int);
> insert into t_smj_right values
> ('key1', 1);
> select
> t2.value
> from t_smj_left t1
> left join t_smj_right t2 on t1.key=t2.key and t1.value=2;
>
> Result:
> +
> NULL
> NULL
> +
> Expected Output:
> +
> 1
> NULL
> +
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work started] (HIVE-25721) Outer join result is wrong
[
https://issues.apache.org/jira/browse/HIVE-25721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-25721 started by Yizhen Fan.
-
> Outer join result is wrong
> --
>
> Key: HIVE-25721
> URL: https://issues.apache.org/jira/browse/HIVE-25721
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: All Versions
>Reporter: Yizhen Fan
>Assignee: Yizhen Fan
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-25721.patch
>
>
> Outer Join results is wrong, here is a left join case.
> select b.fields from a left join b on a.key=b.key and a.filter=xxx
> there are some necessary condition to produce this problem:
> # `select` clause only contains right table fields
> # `on` clause contains left table condition, and this condition can filter
> records
> h3. cause:
> candidateStorage[tag].addRow(value); // CommonMergeJoinOperator.process
> row of left table cannot be add into row container because tblDesc of left
> table is null, while left table data can not be ignored in this case.
> h3. Reproducible steps are mentioned below.
>
> set hive.auto.convert.join=false;
> create table t_smj_left (key string, value int);
> insert into t_smj_left values
> ('key1', 1),
> ('key1', 2);
> create table t_smj_right (key string, value int);
> insert into t_smj_right values
> ('key1', 1);
> select
> t2.value
> from t_smj_left t1
> left join t_smj_right t2 on t1.key=t2.key and t1.value=2;
>
> Result:
> +
> NULL
> NULL
> +
> Expected Output:
> +
> 1
> NULL
> +
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25721) Outer join result is wrong
[
https://issues.apache.org/jira/browse/HIVE-25721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
范宜臻 updated HIVE-25721:
---
Attachment: HIVE-25721.patch
> Outer join result is wrong
> --
>
> Key: HIVE-25721
> URL: https://issues.apache.org/jira/browse/HIVE-25721
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: All Versions
>Reporter: 范宜臻
>Assignee: 范宜臻
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-25721.patch
>
>
> Outer Join results is wrong, here is a left join case.
> select b.fields from a left join b on a.key=b.key and a.filter=xxx
> there are some necessary condition to produce this problem:
> # `select` clause only contains right table fields
> # `on` clause contains left table condition, and this condition can filter
> records
> h3. cause:
> candidateStorage[tag].addRow(value); // CommonMergeJoinOperator.process
> row of left table cannot be add into row container because tblDesc of left
> table is null, while left table data can not be ignored in this case.
> h3. Reproducible steps are mentioned below.
>
> set hive.auto.convert.join=false;
> create table t_smj_left (key string, value int);
> insert into t_smj_left values
> ('key1', 1),
> ('key1', 2);
> create table t_smj_right (key string, value int);
> insert into t_smj_right values
> ('key1', 1);
> select
> t2.value
> from t_smj_left t1
> left join t_smj_right t2 on t1.key=t2.key and t1.value=2;
>
> Result:
> +
> NULL
> NULL
> +
> Expected Output:
> +
> 1
> NULL
> +
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25721) Outer join result is wrong
[
https://issues.apache.org/jira/browse/HIVE-25721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
范宜臻 updated HIVE-25721:
---
Attachment: (was: HIVE-25721.patch)
> Outer join result is wrong
> --
>
> Key: HIVE-25721
> URL: https://issues.apache.org/jira/browse/HIVE-25721
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: All Versions
>Reporter: 范宜臻
>Assignee: 范宜臻
>Priority: Major
> Fix For: 4.0.0
>
>
> Outer Join results is wrong, here is a left join case.
> select b.fields from a left join b on a.key=b.key and a.filter=xxx
> there are some necessary condition to produce this problem:
> # `select` clause only contains right table fields
> # `on` clause contains left table condition, and this condition can filter
> records
> h3. cause:
> candidateStorage[tag].addRow(value); // CommonMergeJoinOperator.process
> row of left table cannot be add into row container because tblDesc of left
> table is null, while left table data can not be ignored in this case.
> h3. Reproducible steps are mentioned below.
>
> set hive.auto.convert.join=false;
> create table t_smj_left (key string, value int);
> insert into t_smj_left values
> ('key1', 1),
> ('key1', 2);
> create table t_smj_right (key string, value int);
> insert into t_smj_right values
> ('key1', 1);
> select
> t2.value
> from t_smj_left t1
> left join t_smj_right t2 on t1.key=t2.key and t1.value=2;
>
> Result:
> +
> NULL
> NULL
> +
> Expected Output:
> +
> 1
> NULL
> +
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25721) Outer join result is wrong
[
https://issues.apache.org/jira/browse/HIVE-25721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
范宜臻 updated HIVE-25721:
---
Attachment: HIVE-25721.patch
> Outer join result is wrong
> --
>
> Key: HIVE-25721
> URL: https://issues.apache.org/jira/browse/HIVE-25721
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: All Versions
>Reporter: 范宜臻
>Assignee: 范宜臻
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-25721.patch
>
>
> Outer Join results is wrong, here is a left join case.
> select b.fields from a left join b on a.key=b.key and a.filter=xxx
> there are some necessary condition to produce this problem:
> # `select` clause only contains right table fields
> # `on` clause contains left table condition, and this condition can filter
> records
> h3. cause:
> candidateStorage[tag].addRow(value); // CommonMergeJoinOperator.process
> row of left table cannot be add into row container because tblDesc of left
> table is null, while left table data can not be ignored in this case.
> h3. Reproducible steps are mentioned below.
>
> set hive.auto.convert.join=false;
> create table t_smj_left (key string, value int);
> insert into t_smj_left values
> ('key1', 1),
> ('key1', 2);
> create table t_smj_right (key string, value int);
> insert into t_smj_right values
> ('key1', 1);
> select
> t2.value
> from t_smj_left t1
> left join t_smj_right t2 on t1.key=t2.key and t1.value=2;
>
> Result:
> +
> NULL
> NULL
> +
> Expected Output:
> +
> 1
> NULL
> +
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25721) Outer join result is wrong
[
https://issues.apache.org/jira/browse/HIVE-25721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
范宜臻 updated HIVE-25721:
---
Description:
Outer Join results is wrong, here is a left join case.
select b.fields from a left join b on a.key=b.key and a.filter=xxx
there are some necessary condition to produce this problem:
# `select` clause only contains right table fields
# `on` clause contains left table condition, and this condition can filter
records
h3. cause:
candidateStorage[tag].addRow(value); // CommonMergeJoinOperator.process
row of left table cannot be add into row container because tblDesc of left
table is null, while left table data can not be ignored in this case.
h3. Reproducible steps are mentioned below.
set hive.auto.convert.join=false;
create table t_smj_left (key string, value int);
insert into t_smj_left values
('key1', 1),
('key1', 2);
create table t_smj_right (key string, value int);
insert into t_smj_right values
('key1', 1);
select
t2.value
from t_smj_left t1
left join t_smj_right t2 on t1.key=t2.key and t1.value=2;
Result:
+
NULL
NULL
+
Expected Output:
+
1
NULL
+
was:
Outer Join results is wrong, here is a left join case.
select b.fields from a left join b on a.key=b.key and a.filter=xxx
there are some necessary condition to produce this problem:
# `select` clause only contains right table fields
# `on` clause contains left table condition, and this condition can filter
records
Reproducible steps are mentioned below.
set hive.auto.convert.join=false;
create table t_smj_left (key string, value int);
insert into t_smj_left values
('key1', 1),
('key1', 2);
create table t_smj_right (key string, value int);
insert into t_smj_right values
('key1', 1);
select
t2.value
from t_smj_left t1
left join t_smj_right t2 on t1.key=t2.key and t1.value=2;
Result:
+
NULL
NULL
+
Expected Output:
+
1
NULL
+
> Outer join result is wrong
> --
>
> Key: HIVE-25721
> URL: https://issues.apache.org/jira/browse/HIVE-25721
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: All Versions
>Reporter: 范宜臻
>Assignee: 范宜臻
>Priority: Major
> Fix For: 4.0.0
>
>
> Outer Join results is wrong, here is a left join case.
> select b.fields from a left join b on a.key=b.key and a.filter=xxx
> there are some necessary condition to produce this problem:
> # `select` clause only contains right table fields
> # `on` clause contains left table condition, and this condition can filter
> records
> h3. cause:
> candidateStorage[tag].addRow(value); // CommonMergeJoinOperator.process
> row of left table cannot be add into row container because tblDesc of left
> table is null, while left table data can not be ignored in this case.
> h3. Reproducible steps are mentioned below.
>
> set hive.auto.convert.join=false;
> create table t_smj_left (key string, value int);
> insert into t_smj_left values
> ('key1', 1),
> ('key1', 2);
> create table t_smj_right (key string, value int);
> insert into t_smj_right values
> ('key1', 1);
> select
> t2.value
> from t_smj_left t1
> left join t_smj_right t2 on t1.key=t2.key and t1.value=2;
>
> Result:
> +
> NULL
> NULL
> +
> Expected Output:
> +
> 1
> NULL
> +
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25721) Outer join result is wrong
[
https://issues.apache.org/jira/browse/HIVE-25721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
范宜臻 updated HIVE-25721:
---
Description:
Outer Join results is wrong, here is a left join case.
select b.fields from a left join b on a.key=b.key and a.filter=xxx
there are some necessary condition to produce this problem:
# `select` clause only contains right table fields
# `on` clause contains left table condition, and this condition can filter
records
Reproducible steps are mentioned below.
set hive.auto.convert.join=false;
create table t_smj_left (key string, value int);
insert into t_smj_left values
('key1', 1),
('key1', 2);
create table t_smj_right (key string, value int);
insert into t_smj_right values
('key1', 1);
select
t2.value
from t_smj_left t1
left join t_smj_right t2 on t1.key=t2.key and t1.value=2;
Result:
+
NULL
NULL
+
Expected Output:
+
1
NULL
+
> Outer join result is wrong
> --
>
> Key: HIVE-25721
> URL: https://issues.apache.org/jira/browse/HIVE-25721
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: All Versions
>Reporter: 范宜臻
>Assignee: 范宜臻
>Priority: Major
> Fix For: 4.0.0
>
>
> Outer Join results is wrong, here is a left join case.
> select b.fields from a left join b on a.key=b.key and a.filter=xxx
> there are some necessary condition to produce this problem:
> # `select` clause only contains right table fields
> # `on` clause contains left table condition, and this condition can filter
> records
> Reproducible steps are mentioned below.
>
> set hive.auto.convert.join=false;
> create table t_smj_left (key string, value int);
> insert into t_smj_left values
> ('key1', 1),
> ('key1', 2);
> create table t_smj_right (key string, value int);
> insert into t_smj_right values
> ('key1', 1);
> select
> t2.value
> from t_smj_left t1
> left join t_smj_right t2 on t1.key=t2.key and t1.value=2;
>
> Result:
> +
> NULL
> NULL
> +
> Expected Output:
> +
> 1
> NULL
> +
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25721) Outer join result is wrong
[ https://issues.apache.org/jira/browse/HIVE-25721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 范宜臻 updated HIVE-25721: --- Summary: Outer join result is wrong (was: Out join result is wrong) > Outer join result is wrong > -- > > Key: HIVE-25721 > URL: https://issues.apache.org/jira/browse/HIVE-25721 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: All Versions >Reporter: 范宜臻 >Assignee: 范宜臻 >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25721) Out join result is wrong
[ https://issues.apache.org/jira/browse/HIVE-25721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 范宜臻 reassigned HIVE-25721: -- > Out join result is wrong > > > Key: HIVE-25721 > URL: https://issues.apache.org/jira/browse/HIVE-25721 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: All Versions >Reporter: 范宜臻 >Assignee: 范宜臻 >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25443) Arrow SerDe Cannot serialize/deserialize complex data types When there are more than 1024 values
[
https://issues.apache.org/jira/browse/HIVE-25443?focusedWorklogId=683102=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-683102
]
ASF GitHub Bot logged work on HIVE-25443:
-
Author: ASF GitHub Bot
Created on: 18/Nov/21 05:20
Start Date: 18/Nov/21 05:20
Worklog Time Spent: 10m
Work Description: shameersss1 commented on pull request #2581:
URL: https://github.com/apache/hive/pull/2581#issuecomment-972548623
> @shameersss1 there are some commits with your
[r@amazon.com](mailto:r@amazon.com) address - could you add associate
that email address with your github account? without that the committer email
address will be changed to some
[...@users.noreply.github.com](mailto:...@users.noreply.github.com)
@kgyrtkirk - I have linked that email address. Please take it forward and
thanks for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 683102)
Time Spent: 0.5h (was: 20m)
> Arrow SerDe Cannot serialize/deserialize complex data types When there are
> more than 1024 values
>
>
> Key: HIVE-25443
> URL: https://issues.apache.org/jira/browse/HIVE-25443
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.1.0, 3.0.0, 3.1.1, 3.1.2
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Complex data types like MAP, STRUCT cannot be serialized/deserialzed using
> Arrow SerDe when there are more than 1024 values. This happens due to
> ColumnVector always being initialized with a size of 1024.
> Issue #1 :
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/ArrowColumnarBatchSerDe.java#L213
> Issue #2 :
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/ArrowColumnarBatchSerDe.java#L215
> Sample unit test to reproduce the case in TestArrowColumnarBatchSerDe :
> {code:java}
> @Test
>public void testListBooleanWithMoreThan1024Values() throws SerDeException {
> String[][] schema = {
> {"boolean_list", "array"},
> };
>
> Object[][] rows = new Object[1025][1];
> for (int i = 0; i < 1025; i++) {
>rows[i][0] = new BooleanWritable(true);
> }
>
> initAndSerializeAndDeserialize(schema, toList(rows));
>}
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25582) Empty result when using offset limit with MR
[ https://issues.apache.org/jira/browse/HIVE-25582?focusedWorklogId=683092=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-683092 ] ASF GitHub Bot logged work on HIVE-25582: - Author: ASF GitHub Bot Created on: 18/Nov/21 04:56 Start Date: 18/Nov/21 04:56 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #2693: URL: https://github.com/apache/hive/pull/2693 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 683092) Time Spent: 2h (was: 1h 50m) > Empty result when using offset limit with MR > > > Key: HIVE-25582 > URL: https://issues.apache.org/jira/browse/HIVE-25582 > Project: Hive > Issue Type: Bug > Components: Operators >Affects Versions: 4.0.0 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > The _mr.ObjectCache_ caches nothing, every time when the limit [retrieving > global counter from the > cache|https://github.com/apache/hive/blob/7b3ecf617a6d46f48a3b6f77e0339fd4ad95a420/ql/src/java/org/apache/hadoop/hive/ql/exec/LimitOperator.java#L150-L161], > a new AtomicInteger will be returned. This make offset _<= > currentCountForAllTasksInt_ always be evaluated to false, as _offset > 0_, > the operator will skip all rows. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25582) Empty result when using offset limit with MR
[ https://issues.apache.org/jira/browse/HIVE-25582?focusedWorklogId=683091=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-683091 ] ASF GitHub Bot logged work on HIVE-25582: - Author: ASF GitHub Bot Created on: 18/Nov/21 04:55 Start Date: 18/Nov/21 04:55 Worklog Time Spent: 10m Work Description: dengzhhu653 closed pull request #2693: URL: https://github.com/apache/hive/pull/2693 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 683091) Time Spent: 1h 50m (was: 1h 40m) > Empty result when using offset limit with MR > > > Key: HIVE-25582 > URL: https://issues.apache.org/jira/browse/HIVE-25582 > Project: Hive > Issue Type: Bug > Components: Operators >Affects Versions: 4.0.0 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > The _mr.ObjectCache_ caches nothing, every time when the limit [retrieving > global counter from the > cache|https://github.com/apache/hive/blob/7b3ecf617a6d46f48a3b6f77e0339fd4ad95a420/ql/src/java/org/apache/hadoop/hive/ql/exec/LimitOperator.java#L150-L161], > a new AtomicInteger will be returned. This make offset _<= > currentCountForAllTasksInt_ always be evaluated to false, as _offset > 0_, > the operator will skip all rows. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25095) Beeline/hive -e command can't deal with query with trailing quote
[
https://issues.apache.org/jira/browse/HIVE-25095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445564#comment-17445564
]
Robbie Zhang commented on HIVE-25095:
-
[~kgyrtkirk] , thanks for your offering :D
> Beeline/hive -e command can't deal with query with trailing quote
> -
>
> Key: HIVE-25095
> URL: https://issues.apache.org/jira/browse/HIVE-25095
> Project: Hive
> Issue Type: Bug
>Reporter: Robbie Zhang
>Assignee: Robbie Zhang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The command
> {code:java}
> hive -e 'select "hive"'{code}
> and
> {code:java}
> beeline -e 'select "hive"'{code}
> fail with such error:
> {code:java}
> Error: Error while compiling statement: FAILED: ParseException line 1:12
> character '' not supported here (state=42000,code=4){code}
> The reason is that org.apache.commons.cli.Util.stripLeadingAndTrailingQuotes
> in commons-cli-1.2.jar strips the trailing quote so the query string is
> changed to
> {code:java}
> select "hive{code}
> This bug is fixed in commons-cli-1.3.1 and commons-cli-1.4.jar. The
> workaround is to overwrite commons-cli-1.2.jar with commons-cli-1.3.1 or
> commons-cli-1.4.jar.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-25720) Fix flaky test TestScheduledReplicationScenarios
[
https://issues.apache.org/jira/browse/HIVE-25720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445400#comment-17445400
]
Zoltan Haindrich commented on HIVE-25720:
-
2 testcases are already disabled in this class by HIVE-23395 ; in recent runs
I've seen issues with the other 2 testmethods...
> Fix flaky test TestScheduledReplicationScenarios
>
>
> Key: HIVE-25720
> URL: https://issues.apache.org/jira/browse/HIVE-25720
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Priority: Major
>
> failed at the first attempt; the issue happened during
> {code}
> drop scheduled query repl_load_p2
> {code}
> which is in a finally block ; so this exception may be shadowing another
> exception
> http://ci.hive.apache.org/job/hive-flaky-check/463/
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-25718) ORDER BY query on external MSSQL table fails
[
https://issues.apache.org/jira/browse/HIVE-25718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445364#comment-17445364
]
Stamatis Zampetakis commented on HIVE-25718:
Most likely the problem is on CBO which seems to generate a query that cannot
be executed on MSSQL. This may require a fix in Apache Calcite.
> ORDER BY query on external MSSQL table fails
>
>
> Key: HIVE-25718
> URL: https://issues.apache.org/jira/browse/HIVE-25718
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Priority: Major
> Attachments: jdbc_table_orderby_mssql.q
>
>
> +Microsoft SQLServer+
> {code:sql}
> CREATE TABLE country (id int, name varchar(20));
> insert into country values (1, 'India');
> insert into country values (2, 'Russia');
> insert into country values (3, 'USA');
> {code}
> +Hive+
> {code:sql}
> CREATE EXTERNAL TABLE country (id int, name varchar(20))
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "MSSQL",
> "hive.sql.jdbc.driver" = "com.microsoft.sqlserver.jdbc.SQLServerDriver",
> "hive.sql.jdbc.url" = "jdbc:sqlserver://localhost:1433;",
> "hive.sql.dbcp.username" = "sa",
> "hive.sql.dbcp.password" = "Its-a-s3cret",
> "hive.sql.table" = "country");
> SELECT * FROM country ORDER BY id;
> {code}
> The query fails with the following stacktrace:
> {noformat}
> com.microsoft.sqlserver.jdbc.SQLServerException: The ORDER BY clause is
> invalid in views, inline functions, derived tables, subqueries, and common
> table expressions, unless TOP, OFFSET or FOR XML is also specified.
> at
> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:258)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1535)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:467)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:409)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7151)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2478)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:219)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:199)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:331)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
> ~[commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
> ~[commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getRecordIterator(GenericJdbcDatabaseAccessor.java:180)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:58)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:35)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:589)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:529)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.Driver.getFetchingTableResults(Driver.java:716)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:668)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:241)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:277)
> [hive-cli-4.0.0-SNAPSHOT.jar:?]
> at
>
[jira] [Updated] (HIVE-25718) ORDER BY query on external MSSQL table fails
[
https://issues.apache.org/jira/browse/HIVE-25718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated HIVE-25718:
---
Attachment: jdbc_table_orderby_mssql.q
> ORDER BY query on external MSSQL table fails
>
>
> Key: HIVE-25718
> URL: https://issues.apache.org/jira/browse/HIVE-25718
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Priority: Major
> Attachments: jdbc_table_orderby_mssql.q
>
>
> +Microsoft SQLServer+
> {code:sql}
> CREATE TABLE country (id int, name varchar(20));
> insert into country values (1, 'India');
> insert into country values (2, 'Russia');
> insert into country values (3, 'USA');
> {code}
> +Hive+
> {code:sql}
> CREATE EXTERNAL TABLE country (id int, name varchar(20))
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "MSSQL",
> "hive.sql.jdbc.driver" = "com.microsoft.sqlserver.jdbc.SQLServerDriver",
> "hive.sql.jdbc.url" = "jdbc:sqlserver://localhost:1433;",
> "hive.sql.dbcp.username" = "sa",
> "hive.sql.dbcp.password" = "Its-a-s3cret",
> "hive.sql.table" = "country");
> SELECT * FROM country ORDER BY id;
> {code}
> The query fails with the following stacktrace:
> {noformat}
> com.microsoft.sqlserver.jdbc.SQLServerException: The ORDER BY clause is
> invalid in views, inline functions, derived tables, subqueries, and common
> table expressions, unless TOP, OFFSET or FOR XML is also specified.
> at
> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:258)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1535)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:467)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:409)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7151)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2478)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:219)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:199)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:331)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
> ~[commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
> ~[commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getRecordIterator(GenericJdbcDatabaseAccessor.java:180)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:58)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:35)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:589)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:529)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.Driver.getFetchingTableResults(Driver.java:716)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:668)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:241)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:277)
> [hive-cli-4.0.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
> [hive-cli-4.0.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
>
[jira] [Commented] (HIVE-25718) ORDER BY query on external MSSQL table fails
[
https://issues.apache.org/jira/browse/HIVE-25718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445362#comment-17445362
]
Stamatis Zampetakis commented on HIVE-25718:
To reproduce the problem run:
{code:bash}
mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=jdbc_table_orderby_mssql.q
-Dtest.output.overwrite
{code}
> ORDER BY query on external MSSQL table fails
>
>
> Key: HIVE-25718
> URL: https://issues.apache.org/jira/browse/HIVE-25718
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Priority: Major
> Attachments: jdbc_table_orderby_mssql.q
>
>
> +Microsoft SQLServer+
> {code:sql}
> CREATE TABLE country (id int, name varchar(20));
> insert into country values (1, 'India');
> insert into country values (2, 'Russia');
> insert into country values (3, 'USA');
> {code}
> +Hive+
> {code:sql}
> CREATE EXTERNAL TABLE country (id int, name varchar(20))
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "MSSQL",
> "hive.sql.jdbc.driver" = "com.microsoft.sqlserver.jdbc.SQLServerDriver",
> "hive.sql.jdbc.url" = "jdbc:sqlserver://localhost:1433;",
> "hive.sql.dbcp.username" = "sa",
> "hive.sql.dbcp.password" = "Its-a-s3cret",
> "hive.sql.table" = "country");
> SELECT * FROM country ORDER BY id;
> {code}
> The query fails with the following stacktrace:
> {noformat}
> com.microsoft.sqlserver.jdbc.SQLServerException: The ORDER BY clause is
> invalid in views, inline functions, derived tables, subqueries, and common
> table expressions, unless TOP, OFFSET or FOR XML is also specified.
> at
> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:258)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1535)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:467)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:409)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7151)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2478)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:219)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:199)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:331)
> ~[mssql-jdbc-6.2.1.jre8.jar:?]
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
> ~[commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
> ~[commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getRecordIterator(GenericJdbcDatabaseAccessor.java:180)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:58)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:35)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:589)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:529)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.Driver.getFetchingTableResults(Driver.java:716)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:668)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:241)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:277)
> [hive-cli-4.0.0-SNAPSHOT.jar:?]
> at
>
[jira] [Work logged] (HIVE-25443) Arrow SerDe Cannot serialize/deserialize complex data types When there are more than 1024 values
[
https://issues.apache.org/jira/browse/HIVE-25443?focusedWorklogId=682738=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-682738
]
ASF GitHub Bot logged work on HIVE-25443:
-
Author: ASF GitHub Bot
Created on: 17/Nov/21 16:27
Start Date: 17/Nov/21 16:27
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #2581:
URL: https://github.com/apache/hive/pull/2581#issuecomment-971744573
@shameersss1 there are some commits with your r@amazon.com address -
could you add associate that email address with your github account?
without that the committer email address will be changed to some
...@users.noreply.github.com
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 682738)
Time Spent: 20m (was: 10m)
> Arrow SerDe Cannot serialize/deserialize complex data types When there are
> more than 1024 values
>
>
> Key: HIVE-25443
> URL: https://issues.apache.org/jira/browse/HIVE-25443
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.1.0, 3.0.0, 3.1.1, 3.1.2
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Complex data types like MAP, STRUCT cannot be serialized/deserialzed using
> Arrow SerDe when there are more than 1024 values. This happens due to
> ColumnVector always being initialized with a size of 1024.
> Issue #1 :
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/ArrowColumnarBatchSerDe.java#L213
> Issue #2 :
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/ArrowColumnarBatchSerDe.java#L215
> Sample unit test to reproduce the case in TestArrowColumnarBatchSerDe :
> {code:java}
> @Test
>public void testListBooleanWithMoreThan1024Values() throws SerDeException {
> String[][] schema = {
> {"boolean_list", "array"},
> };
>
> Object[][] rows = new Object[1025][1];
> for (int i = 0; i < 1025; i++) {
>rows[i][0] = new BooleanWritable(true);
> }
>
> initAndSerializeAndDeserialize(schema, toList(rows));
>}
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25670) Avoid getTable() calls for foreign key tables not used in a query
[ https://issues.apache.org/jira/browse/HIVE-25670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich resolved HIVE-25670. - Fix Version/s: 4.0.0 Resolution: Fixed merged into master. Thank you [~scarlin] ! > Avoid getTable() calls for foreign key tables not used in a query > - > > Key: HIVE-25670 > URL: https://issues.apache.org/jira/browse/HIVE-25670 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Steve Carlin >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > In RelOptHiveTable, we generate the referential constraints for the table. In > this process, we make a metastore call to fetch these tables. This is used > later on for potential gain on joins done on the key. > However, there is no need to fetch these constraints if the table is not used > in the query. If we can get this information up front, we can save a bit on > compilation time. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25670) Avoid getTable() calls for foreign key tables not used in a query
[ https://issues.apache.org/jira/browse/HIVE-25670?focusedWorklogId=682734=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-682734 ] ASF GitHub Bot logged work on HIVE-25670: - Author: ASF GitHub Bot Created on: 17/Nov/21 16:23 Start Date: 17/Nov/21 16:23 Worklog Time Spent: 10m Work Description: kgyrtkirk merged pull request #2763: URL: https://github.com/apache/hive/pull/2763 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 682734) Time Spent: 1h 20m (was: 1h 10m) > Avoid getTable() calls for foreign key tables not used in a query > - > > Key: HIVE-25670 > URL: https://issues.apache.org/jira/browse/HIVE-25670 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Steve Carlin >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > In RelOptHiveTable, we generate the referential constraints for the table. In > this process, we make a metastore call to fetch these tables. This is used > later on for potential gain on joins done on the key. > However, there is no need to fetch these constraints if the table is not used > in the query. If we can get this information up front, we can save a bit on > compilation time. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25700) Prevent deletion of Notification Events post restarts
[
https://issues.apache.org/jira/browse/HIVE-25700?focusedWorklogId=682692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-682692
]
ASF GitHub Bot logged work on HIVE-25700:
-
Author: ASF GitHub Bot
Created on: 17/Nov/21 15:27
Start Date: 17/Nov/21 15:27
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #2789:
URL: https://github.com/apache/hive/pull/2789#discussion_r751353413
##
File path:
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java
##
@@ -1417,10 +1426,27 @@ private void process(NotificationEvent event,
ListenerEvent listenerEvent) throw
setTimeToLive(MetastoreConf.getTimeVar(conf, ttlConf, TimeUnit.SECONDS));
setCleanupInterval(
MetastoreConf.getTimeVar(conf,
ConfVars.EVENT_DB_LISTENER_CLEAN_INTERVAL, TimeUnit.MILLISECONDS));
+ setWaitInterval(MetastoreConf
+ .getTimeVar(conf, EVENT_DB_LISTENER_CLEAN_STARTUP_WAIT_INTERVAL,
TimeUnit.MILLISECONDS));
}
@Override
public void run() {
+ LOG.info("Wait interval is {}", waitInterval);
+ if (waitInterval > 0) {
+try {
+ LOG.info("Cleaner Thread Restarted and {} or {} is configured. So
cleaner thread will startup post waiting "
+ + "{} ms", EVENT_DB_LISTENER_CLEAN_STARTUP_WAIT_INTERVAL,
+ EVENT_DB_LISTENER_CLEAN_STARTUP_WAIT_INTERVAL.getHiveName(),
waitInterval);
+ Thread.sleep(waitInterval);
+} catch (InterruptedException e) {
+ LOG.error("Failed during the initial wait before start.", e);
+ Thread.currentThread().interrupt();
Review comment:
As discussed, Have wrapped this isTest check
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##
@@ -752,6 +752,9 @@ public static ConfVars getMetaConf(String name) {
EVENT_DB_LISTENER_CLEAN_INTERVAL("metastore.event.db.listener.clean.interval",
"hive.metastore.event.db.listener.clean.interval", 7200,
TimeUnit.SECONDS,
"sleep interval between each run for cleanup of events from the
database listener queue"),
+
EVENT_DB_LISTENER_CLEAN_STARTUP_WAIT_INTERVAL("metastore.event.db.listener.clean.startup.wait.interval",
+"hive.metastore.event.db.listener.clean.startup.wait.interval", 1,
TimeUnit.DAYS,
+"wait interval post start of Cleaner thread after which the cleaner
thread starts to work"),
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 682692)
Time Spent: 40m (was: 0.5h)
> Prevent deletion of Notification Events post restarts
> -
>
> Key: HIVE-25700
> URL: https://issues.apache.org/jira/browse/HIVE-25700
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> In case of DR scenarios, when Hive service goes down, Prevent deletion of
> entries in the Notification Log immediately, Give time for ADMINs to
> reconfigure properties to handle further Replication process.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-25717) INSERT INTO on external MariaDB/MySQL table fails silently
[
https://issues.apache.org/jira/browse/HIVE-25717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445288#comment-17445288
]
Stamatis Zampetakis commented on HIVE-25717:
It seems that this was caused as a side effect of fixing HIVE-22706.
> INSERT INTO on external MariaDB/MySQL table fails silently
> --
>
> Key: HIVE-25717
> URL: https://issues.apache.org/jira/browse/HIVE-25717
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: jdbc_table_dml_mysql.q
>
>
> +MariaDB/MySQL+
> {code:sql}
> CREATE TABLE country (id int, name varchar(20));
> insert into country values (1, 'India');
> insert into country values (2, 'Russia');
> insert into country values (3, 'USA');
> {code}
> +Hive+
> {code:sql}
> CREATE EXTERNAL TABLE country (id int, name varchar(20))
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "MYSQL",
> "hive.sql.jdbc.driver" = "com.mysql.jdbc.Driver",
> "hive.sql.jdbc.url" = "jdbc:mysql://localhost:3306/qtestDB",
> "hive.sql.dbcp.username" = "root",
> "hive.sql.dbcp.password" = "qtestpassword",
> "hive.sql.table" = "country"
> );
> INSERT INTO country VALUES (8, 'Hungary');
> SELECT * FROM country;
> {code}
> +Expected results+
> ||ID||NAME||
> |1| India|
> |2| Russia|
> |3| USA|
> |8| Hungary|
> +Actual results+
> ||ID||NAME||
> |1| India|
> |2| Russia|
> |3| USA|
> The {{INSERT INTO}} statement finishes without showing any kind of problem in
> the logs but the row is not inserted in the table.
> Running the test it comes back green although the following exception is
> printed in the System.err (not in the logs).
> {noformat}
> java.sql.SQLException: Parameter metadata not available for the given
> statement
> at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
> at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
> at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
> at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
> at
> com.mysql.cj.jdbc.MysqlParameterMetadata.checkAvailable(MysqlParameterMetadata.java:86)
> at
> com.mysql.cj.jdbc.MysqlParameterMetadata.getParameterType(MysqlParameterMetadata.java:138)
> at
> org.apache.hive.storage.jdbc.DBRecordWritable.write(DBRecordWritable.java:67)
> at
> org.apache.hadoop.mapreduce.lib.db.DBOutputFormat$DBRecordWriter.write(DBOutputFormat.java:122)
> at
> org.apache.hive.storage.jdbc.JdbcRecordWriter.write(JdbcRecordWriter.java:47)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:1160)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:94)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133)
> at
> org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFInline.process(GenericUDTFInline.java:64)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:94)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:173)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:154)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:101)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:83)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:414)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:311)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:277)
> at
>
[jira] [Commented] (HIVE-25717) INSERT INTO on external MariaDB/MySQL table fails silently
[
https://issues.apache.org/jira/browse/HIVE-25717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445284#comment-17445284
]
Stamatis Zampetakis commented on HIVE-25717:
The problem can be reproduced by running:
{code:bash}
mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=jdbc_table_dml_mysql.q
-Dtest.output.overwrite
{code}
> INSERT INTO on external MariaDB/MySQL table fails silently
> --
>
> Key: HIVE-25717
> URL: https://issues.apache.org/jira/browse/HIVE-25717
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: jdbc_table_dml_mysql.q
>
>
> +MariaDB/MySQL+
> {code:sql}
> CREATE TABLE country (id int, name varchar(20));
> insert into country values (1, 'India');
> insert into country values (2, 'Russia');
> insert into country values (3, 'USA');
> {code}
> +Hive+
> {code:sql}
> CREATE EXTERNAL TABLE country (id int, name varchar(20))
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "MYSQL",
> "hive.sql.jdbc.driver" = "com.mysql.jdbc.Driver",
> "hive.sql.jdbc.url" = "jdbc:mysql://localhost:3306/qtestDB",
> "hive.sql.dbcp.username" = "root",
> "hive.sql.dbcp.password" = "qtestpassword",
> "hive.sql.table" = "country"
> );
> INSERT INTO country VALUES (8, 'Hungary');
> SELECT * FROM country;
> {code}
> +Expected results+
> ||ID||NAME||
> |1| India|
> |2| Russia|
> |3| USA|
> |8| Hungary|
> +Actual results+
> ||ID||NAME||
> |1| India|
> |2| Russia|
> |3| USA|
> The {{INSERT INTO}} statement finishes without showing any kind of problem in
> the logs but the row is not inserted in the table.
> Running the test it comes back green although the following exception is
> printed in the System.err (not in the logs).
> {noformat}
> java.sql.SQLException: Parameter metadata not available for the given
> statement
> at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
> at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
> at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
> at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
> at
> com.mysql.cj.jdbc.MysqlParameterMetadata.checkAvailable(MysqlParameterMetadata.java:86)
> at
> com.mysql.cj.jdbc.MysqlParameterMetadata.getParameterType(MysqlParameterMetadata.java:138)
> at
> org.apache.hive.storage.jdbc.DBRecordWritable.write(DBRecordWritable.java:67)
> at
> org.apache.hadoop.mapreduce.lib.db.DBOutputFormat$DBRecordWriter.write(DBOutputFormat.java:122)
> at
> org.apache.hive.storage.jdbc.JdbcRecordWriter.write(JdbcRecordWriter.java:47)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:1160)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:94)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133)
> at
> org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFInline.process(GenericUDTFInline.java:64)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:94)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:173)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:154)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:101)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:83)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:414)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:311)
> at
>
[jira] [Updated] (HIVE-25717) INSERT INTO on external MariaDB/MySQL table fails silently
[
https://issues.apache.org/jira/browse/HIVE-25717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated HIVE-25717:
---
Attachment: jdbc_table_dml_mysql.q
> INSERT INTO on external MariaDB/MySQL table fails silently
> --
>
> Key: HIVE-25717
> URL: https://issues.apache.org/jira/browse/HIVE-25717
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: jdbc_table_dml_mysql.q
>
>
> +MariaDB/MySQL+
> {code:sql}
> CREATE TABLE country (id int, name varchar(20));
> insert into country values (1, 'India');
> insert into country values (2, 'Russia');
> insert into country values (3, 'USA');
> {code}
> +Hive+
> {code:sql}
> CREATE EXTERNAL TABLE country (id int, name varchar(20))
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "MYSQL",
> "hive.sql.jdbc.driver" = "com.mysql.jdbc.Driver",
> "hive.sql.jdbc.url" = "jdbc:mysql://localhost:3306/qtestDB",
> "hive.sql.dbcp.username" = "root",
> "hive.sql.dbcp.password" = "qtestpassword",
> "hive.sql.table" = "country"
> );
> INSERT INTO country VALUES (8, 'Hungary');
> SELECT * FROM country;
> {code}
> +Expected results+
> ||ID||NAME||
> |1| India|
> |2| Russia|
> |3| USA|
> |8| Hungary|
> +Actual results+
> ||ID||NAME||
> |1| India|
> |2| Russia|
> |3| USA|
> The {{INSERT INTO}} statement finishes without showing any kind of problem in
> the logs but the row is not inserted in the table.
> Running the test it comes back green although the following exception is
> printed in the System.err (not in the logs).
> {noformat}
> java.sql.SQLException: Parameter metadata not available for the given
> statement
> at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
> at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
> at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
> at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
> at
> com.mysql.cj.jdbc.MysqlParameterMetadata.checkAvailable(MysqlParameterMetadata.java:86)
> at
> com.mysql.cj.jdbc.MysqlParameterMetadata.getParameterType(MysqlParameterMetadata.java:138)
> at
> org.apache.hive.storage.jdbc.DBRecordWritable.write(DBRecordWritable.java:67)
> at
> org.apache.hadoop.mapreduce.lib.db.DBOutputFormat$DBRecordWriter.write(DBOutputFormat.java:122)
> at
> org.apache.hive.storage.jdbc.JdbcRecordWriter.write(JdbcRecordWriter.java:47)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:1160)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:94)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133)
> at
> org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFInline.process(GenericUDTFInline.java:64)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:94)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:173)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:154)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:101)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:83)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:414)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:311)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:277)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381)
> at
>
[jira] [Assigned] (HIVE-25717) INSERT INTO on external MariaDB/MySQL table fails silently
[
https://issues.apache.org/jira/browse/HIVE-25717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis reassigned HIVE-25717:
--
> INSERT INTO on external MariaDB/MySQL table fails silently
> --
>
> Key: HIVE-25717
> URL: https://issues.apache.org/jira/browse/HIVE-25717
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>
> +MariaDB/MySQL+
> {code:sql}
> CREATE TABLE country (id int, name varchar(20));
> insert into country values (1, 'India');
> insert into country values (2, 'Russia');
> insert into country values (3, 'USA');
> {code}
> +Hive+
> {code:sql}
> CREATE EXTERNAL TABLE country (id int, name varchar(20))
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "MYSQL",
> "hive.sql.jdbc.driver" = "com.mysql.jdbc.Driver",
> "hive.sql.jdbc.url" = "jdbc:mysql://localhost:3306/qtestDB",
> "hive.sql.dbcp.username" = "root",
> "hive.sql.dbcp.password" = "qtestpassword",
> "hive.sql.table" = "country"
> );
> INSERT INTO country VALUES (8, 'Hungary');
> SELECT * FROM country;
> {code}
> +Expected results+
> ||ID||NAME||
> |1| India|
> |2| Russia|
> |3| USA|
> |8| Hungary|
> +Actual results+
> ||ID||NAME||
> |1| India|
> |2| Russia|
> |3| USA|
> The {{INSERT INTO}} statement finishes without showing any kind of problem in
> the logs but the row is not inserted in the table.
> Running the test it comes back green although the following exception is
> printed in the System.err (not in the logs).
> {noformat}
> java.sql.SQLException: Parameter metadata not available for the given
> statement
> at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
> at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
> at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
> at
> com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
> at
> com.mysql.cj.jdbc.MysqlParameterMetadata.checkAvailable(MysqlParameterMetadata.java:86)
> at
> com.mysql.cj.jdbc.MysqlParameterMetadata.getParameterType(MysqlParameterMetadata.java:138)
> at
> org.apache.hive.storage.jdbc.DBRecordWritable.write(DBRecordWritable.java:67)
> at
> org.apache.hadoop.mapreduce.lib.db.DBOutputFormat$DBRecordWriter.write(DBOutputFormat.java:122)
> at
> org.apache.hive.storage.jdbc.JdbcRecordWriter.write(JdbcRecordWriter.java:47)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:1160)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:94)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133)
> at
> org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFInline.process(GenericUDTFInline.java:64)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:94)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:173)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:154)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:101)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:83)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:414)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:311)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:277)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:82)
> at
>
[jira] [Updated] (HIVE-25716) Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
[
https://issues.apache.org/jira/browse/HIVE-25716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-25716:
-
Labels: flaky-test (was: )
> Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
> --
>
> Key: HIVE-25716
> URL: https://issues.apache.org/jira/browse/HIVE-25716
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Priority: Major
> Labels: flaky-test
>
> Flaky check failed on run #59:
> [http://ci.hive.apache.org/job/hive-flaky-check/467/|http://ci.hive.apache.org/job/hive-flaky-check/467/]
> {code:java}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hive.ql.txn.compactor.TestCompactionMetrics.testOldestReadyForCleaningAge(TestCompactionMetrics.java:214)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24390) Spelling
[ https://issues.apache.org/jira/browse/HIVE-24390?focusedWorklogId=682655=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-682655 ] ASF GitHub Bot logged work on HIVE-24390: - Author: ASF GitHub Bot Created on: 17/Nov/21 14:34 Start Date: 17/Nov/21 14:34 Worklog Time Spent: 10m Work Description: jsoref commented on pull request #1674: URL: https://github.com/apache/hive/pull/1674#issuecomment-971639430 Yes. That's my plan. This is sadly a tiny slice. I think I'll do another PR of about the same size. When we get to the end, I'll see about a PR for the action-workflow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 682655) Time Spent: 2h 20m (was: 2h 10m) > Spelling > > > Key: HIVE-24390 > URL: https://issues.apache.org/jira/browse/HIVE-24390 > Project: Hive > Issue Type: Bug >Reporter: Josh Soref >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25690) Fix column reorder detection for Iceberg schema evolution
[ https://issues.apache.org/jira/browse/HIVE-25690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445157#comment-17445157 ] Marton Bod commented on HIVE-25690: --- Pushed to master. Thanks [~szita] for reviewing! > Fix column reorder detection for Iceberg schema evolution > - > > Key: HIVE-25690 > URL: https://issues.apache.org/jira/browse/HIVE-25690 > Project: Hive > Issue Type: Bug >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Current algorithm for detecting schema differences between HMS and Iceberg > schema is broken when it comes to column reorders. This patch should fix that > up and add more extensive testing. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25690) Fix column reorder detection for Iceberg schema evolution
[ https://issues.apache.org/jira/browse/HIVE-25690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod resolved HIVE-25690. --- Resolution: Fixed > Fix column reorder detection for Iceberg schema evolution > - > > Key: HIVE-25690 > URL: https://issues.apache.org/jira/browse/HIVE-25690 > Project: Hive > Issue Type: Bug >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Current algorithm for detecting schema differences between HMS and Iceberg > schema is broken when it comes to column reorders. This patch should fix that > up and add more extensive testing. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25690) Fix column reorder detection for Iceberg schema evolution
[ https://issues.apache.org/jira/browse/HIVE-25690?focusedWorklogId=682605=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-682605 ] ASF GitHub Bot logged work on HIVE-25690: - Author: ASF GitHub Bot Created on: 17/Nov/21 13:19 Start Date: 17/Nov/21 13:19 Worklog Time Spent: 10m Work Description: marton-bod merged pull request #2779: URL: https://github.com/apache/hive/pull/2779 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 682605) Time Spent: 50m (was: 40m) > Fix column reorder detection for Iceberg schema evolution > - > > Key: HIVE-25690 > URL: https://issues.apache.org/jira/browse/HIVE-25690 > Project: Hive > Issue Type: Bug >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Current algorithm for detecting schema differences between HMS and Iceberg > schema is broken when it comes to column reorders. This patch should fix that > up and add more extensive testing. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25582) Empty result when using offset limit with MR
[ https://issues.apache.org/jira/browse/HIVE-25582?focusedWorklogId=682595=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-682595 ] ASF GitHub Bot logged work on HIVE-25582: - Author: ASF GitHub Bot Created on: 17/Nov/21 13:11 Start Date: 17/Nov/21 13:11 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #2693: URL: https://github.com/apache/hive/pull/2693#issuecomment-971565843 The failed test seems unrelated... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 682595) Time Spent: 1h 40m (was: 1.5h) > Empty result when using offset limit with MR > > > Key: HIVE-25582 > URL: https://issues.apache.org/jira/browse/HIVE-25582 > Project: Hive > Issue Type: Bug > Components: Operators >Affects Versions: 4.0.0 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > The _mr.ObjectCache_ caches nothing, every time when the limit [retrieving > global counter from the > cache|https://github.com/apache/hive/blob/7b3ecf617a6d46f48a3b6f77e0339fd4ad95a420/ql/src/java/org/apache/hadoop/hive/ql/exec/LimitOperator.java#L150-L161], > a new AtomicInteger will be returned. This make offset _<= > currentCountForAllTasksInt_ always be evaluated to false, as _offset > 0_, > the operator will skip all rows. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25701) Declare JDBC drivers as runtime & optional dependencies
[ https://issues.apache.org/jira/browse/HIVE-25701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis resolved HIVE-25701. Fix Version/s: 4.0.0 Resolution: Fixed Fixed in [34158d8531725ab778a6c6112391370b44669bd8|https://github.com/apache/hive/commit/34158d8531725ab778a6c6112391370b44669bd8]. Thanks for the review [~kgyrtkirk]! > Declare JDBC drivers as runtime & optional dependencies > --- > > Key: HIVE-25701 > URL: https://issues.apache.org/jira/browse/HIVE-25701 > Project: Hive > Issue Type: Task > Components: Standalone Metastore, Testing Infrastructure >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently, we are using the following JDBC drivers in various Hive modules: > * MariaDB > * MySQL > * Oracle > * Postgres > * MSSQL > * Derby > MariaDB, MySQL, and Oracle licenses are not compatible with Apache License 2 > ([Category-X |https://www.apache.org/legal/resolved.html#category-x]) and in > the past we used various ways to circumvent licensing problems (see > HIVE-23284). Now, some of them appear as test scope dependency which is OKish > but in the near future may lead again to licensing problems. > JDBC drivers are only needed at runtime so they could all be declared at > runtime scope. Moreover, Hive does not require a specific JDBC driver in > order to operate so they are all optional. > The goal of this issue is to declare every JDBC driver at runtime scope and > mark it as optional > ([ASF-optional|https://www.apache.org/legal/resolved.html#optional], > [maven-optional|https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html]). > > This has the following advantages: > * Eliminates the risk to write code which needs JDBC driver classes in order > to compile and potentially violate AL2. > * Unifies the declaration of JDBC drivers making easier to add/remove some if > necessary. > * Removes the need to use download-maven-plugin and other similar workarounds > to avoid licensing problems. > * Simplifies the execution of tests using these drivers since now they are > added in the runtime classpath automatically by maven. > * Projects with dependencies depending on Hive will not inherit any JDBC > driver by default. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25701) Declare JDBC drivers as runtime & optional dependencies
[ https://issues.apache.org/jira/browse/HIVE-25701?focusedWorklogId=682571=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-682571 ] ASF GitHub Bot logged work on HIVE-25701: - Author: ASF GitHub Bot Created on: 17/Nov/21 12:28 Start Date: 17/Nov/21 12:28 Worklog Time Spent: 10m Work Description: zabetak closed pull request #2790: URL: https://github.com/apache/hive/pull/2790 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 682571) Time Spent: 0.5h (was: 20m) > Declare JDBC drivers as runtime & optional dependencies > --- > > Key: HIVE-25701 > URL: https://issues.apache.org/jira/browse/HIVE-25701 > Project: Hive > Issue Type: Task > Components: Standalone Metastore, Testing Infrastructure >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently, we are using the following JDBC drivers in various Hive modules: > * MariaDB > * MySQL > * Oracle > * Postgres > * MSSQL > * Derby > MariaDB, MySQL, and Oracle licenses are not compatible with Apache License 2 > ([Category-X |https://www.apache.org/legal/resolved.html#category-x]) and in > the past we used various ways to circumvent licensing problems (see > HIVE-23284). Now, some of them appear as test scope dependency which is OKish > but in the near future may lead again to licensing problems. > JDBC drivers are only needed at runtime so they could all be declared at > runtime scope. Moreover, Hive does not require a specific JDBC driver in > order to operate so they are all optional. > The goal of this issue is to declare every JDBC driver at runtime scope and > mark it as optional > ([ASF-optional|https://www.apache.org/legal/resolved.html#optional], > [maven-optional|https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html]). > > This has the following advantages: > * Eliminates the risk to write code which needs JDBC driver classes in order > to compile and potentially violate AL2. > * Unifies the declaration of JDBC drivers making easier to add/remove some if > necessary. > * Removes the need to use download-maven-plugin and other similar workarounds > to avoid licensing problems. > * Simplifies the execution of tests using these drivers since now they are > added in the runtime classpath automatically by maven. > * Projects with dependencies depending on Hive will not inherit any JDBC > driver by default. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25656) Get materialized view state based on number of affected rows of transactions
[
https://issues.apache.org/jira/browse/HIVE-25656?focusedWorklogId=682568=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-682568
]
ASF GitHub Bot logged work on HIVE-25656:
-
Author: ASF GitHub Bot
Created on: 17/Nov/21 12:27
Start Date: 17/Nov/21 12:27
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #2756:
URL: https://github.com/apache/hive/pull/2756#discussion_r751191780
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
##
@@ -12847,13 +12848,12 @@ protected void saveViewDefinition() throws
SemanticException {
createVwDesc.setViewExpandedText(expandedText);
}
- private Set getTablesUsed(ParseContext parseCtx) {
-Set tablesUsed = new HashSet<>();
+ private Set getTablesUsed(ParseContext parseCtx) {
Review comment:
Try to move this method to `ParseContext`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 682568)
Time Spent: 2h 10m (was: 2h)
> Get materialized view state based on number of affected rows of transactions
>
>
> Key: HIVE-25656
> URL: https://issues.apache.org/jira/browse/HIVE-25656
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views, Transactions
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> To enable the faster incremental rebuild of materialized views presence of
> update/delete operations on the source tables of the view since the last
> rebuild must be checked. Based on the outcome different plan is generated for
> scenarios in presence of update/delete and insert only operations.
> Currently this is done by querying the COMPLETED_TXN_COMPONENTS table however
> the records from this table is cleaned when MV source tables are compacted.
> This reduces the chances of incremental MV rebuild.
> The goal of this patch is to find an alternative way to store and retrieve
> this information.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25531) Remove the core classified hive-exec artifact
[ https://issues.apache.org/jira/browse/HIVE-25531?focusedWorklogId=682565=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-682565 ] ASF GitHub Bot logged work on HIVE-25531: - Author: ASF GitHub Bot Created on: 17/Nov/21 12:22 Start Date: 17/Nov/21 12:22 Worklog Time Spent: 10m Work Description: zabetak commented on a change in pull request #2648: URL: https://github.com/apache/hive/pull/2648#discussion_r751178653 ## File path: ql/pom.xml ## @@ -1036,26 +1036,6 @@ test-jar - - -core-jar -package - - jar - - - core - - Review comment: nit: Remove also line 1085/1065. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 682565) Time Spent: 20m (was: 10m) > Remove the core classified hive-exec artifact > - > > Key: HIVE-25531 > URL: https://issues.apache.org/jira/browse/HIVE-25531 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > * this artifact was introduced in HIVE-7423 > * loading this artifact and the shaded hive-exec (along with the jdbc driver) > could create interesting classpath problems > * if other projects have issues with the shaded hive-exec artifact we must > start fix those problems -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25714) Some tests are flaky because docker is not able to start in 5 seconds
[
https://issues.apache.org/jira/browse/HIVE-25714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445118#comment-17445118
]
Stamatis Zampetakis commented on HIVE-25714:
The good news is that the flaky checker is still running which means that the
problem may not be that often :D Maybe we can just increase the timeout till a
better solution is found.
> Some tests are flaky because docker is not able to start in 5 seconds
> -
>
> Key: HIVE-25714
> URL: https://issues.apache.org/jira/browse/HIVE-25714
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> there are some testruns failing with; and on the test site multiple pods are
> running in parallel - its not an ideal environment for tight deadlines
> {code}
> Unexpected exception java.lang.RuntimeException: Process docker failed to run
> in 5 seconds
> at
> org.apache.hadoop.hive.ql.externalDB.AbstractExternalDB.runCmd(AbstractExternalDB.java:92)
> at
> org.apache.hadoop.hive.ql.externalDB.AbstractExternalDB.launchDockerContainer(AbstractExternalDB.java:123)
> at
> org.apache.hadoop.hive.ql.qoption.QTestDatabaseHandler.beforeTest(QTestDatabaseHandler.java:111)
> at
> org.apache.hadoop.hive.ql.qoption.QTestOptionDispatcher.beforeTest(QTestOptionDispatcher.java:79)
> {code}
> http://ci.hive.apache.org/job/hive-precommit/job/PR-1674/4/testReport/junit/org.apache.hadoop.hive.cli.split19/TestMiniLlapLocalCliDriver/Testing___split_14___PostProcess___testCliDriver_qt_database_all_/
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-25715) Provide nightly builds
[ https://issues.apache.org/jira/browse/HIVE-25715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445119#comment-17445119 ] Zoltan Haindrich commented on HIVE-25715: - there are different levels I could think of: * ugly-duckling: ** just do a dist build and take packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/ as the nightly build ** this will work; however its version will be 4.0.0-SNAPSHOT * twisted: ** run a versions:set on the project which will change the versions of the packages "connected" to the root pom ** this will have more proper versions on the outside; but internally storage api will run with X-SNAPSHOT * frankeinstein: ** reconnect the disconnected pom.xml-s to the root and then run a versions:set ** this approach will make storage-api pick up the hive version in these builds ** it might have its own problems because in the original project the pom-s are disconnected * gentlemen: ** go into disconnected modules and update the versions for those as well (storage-api,standalone-metastore,?) ** this will need some script magic since maven cant do these updates correctly outside I think I'll go with the "twisted" approach for now - doing the proper one could probably also reused as some kind of level release automation...the twisted approach is simple enough to start with and build the infra around it ; and we can move forward to "gentlemen" later if there is interest in it > Provide nightly builds > -- > > Key: HIVE-25715 > URL: https://issues.apache.org/jira/browse/HIVE-25715 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > provide nightly builds for the master branch -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25405) Implement Connector Provider for Amazon Redshift
[
https://issues.apache.org/jira/browse/HIVE-25405?focusedWorklogId=682553=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-682553
]
ASF GitHub Bot logged work on HIVE-25405:
-
Author: ASF GitHub Bot
Created on: 17/Nov/21 11:55
Start Date: 17/Nov/21 11:55
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #2546:
URL: https://github.com/apache/hive/pull/2546#discussion_r751148777
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/RedshiftConnectorProvider.java
##
@@ -0,0 +1,86 @@
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.Table;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class RedshiftConnectorProvider extends AbstractJDBCConnectorProvider {
+private static Logger LOG =
LoggerFactory.getLogger(RedshiftConnectorProvider.class);
+
+private static final String DRIVER_CLASS =
"com.amazon.redshift.jdbc42.Driver".intern();
+
+public RedshiftConnectorProvider(String dbName, DataConnector dataConn) {
+super(dbName, dataConn, DRIVER_CLASS);
+}
+
+protected String getDataType(String dbDataType, int size) {
+String mappedType = super.getDataType(dbDataType, size);
+
+// The VOID type points to the corresponding datatype not existing in
hive. These datatypes are datastore
+// specific. They need special handling. An example would be the
Geometric type that is not supported in Hive.
+// The other cases where a datatype in redshift is resolved to a VOID
type are during the use of aliases like
+// float8, int8 etc. These can be mapped to existing hive types and
are done below.
+if (!mappedType.equalsIgnoreCase(ColumnType.VOID_TYPE_NAME)) {
+return mappedType;
+}
+
+// map any db specific types here.
Review comment:
The Intention is clear, comment is redundant.
```suggestion
```
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/RedshiftConnectorProvider.java
##
@@ -0,0 +1,86 @@
+package org.apache.hadoop.hive.metastore.dataconnector.jdbc;
+
+import org.apache.hadoop.hive.metastore.ColumnType;
+import org.apache.hadoop.hive.metastore.api.DataConnector;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.Table;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class RedshiftConnectorProvider extends AbstractJDBCConnectorProvider {
+private static Logger LOG =
LoggerFactory.getLogger(RedshiftConnectorProvider.class);
+
+private static final String DRIVER_CLASS =
"com.amazon.redshift.jdbc42.Driver".intern();
Review comment:
There is no benefit at all calling intern() on static final field.
Actually, I don't see a good reason to have field declaration here. We could
inline the string to the constructor.
##
File path: ql/src/test/queries/clientpositive/redshift_data_connector.q
##
@@ -0,0 +1,65 @@
+-- SORT_QUERY_RESULTS
+SHOW CONNECTORS;
+
+-- CREATE with comment
+CREATE CONNECTOR redshift_test
+TYPE 'redshift'
+URL
'jdbc:redshift://redshift-cluster-1.c1gffkxfot1v.us-east-2.redshift.amazonaws.com:5439/dev'
Review comment:
Agree with Zoltan, I don't think we can add this. Very likely now the
Redhshift instance specified here may not be available and the test will fail.
Since Postgres and Redshift have many similarities I would suggest adding at
least CONNECTOR test with Postgres. An easy way to do it would be using
`--!qt:database:postgres:` option.
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/AbstractJDBCConnectorProvider.java
##
@@ -185,9 +185,18 @@ protected Connection getConnection() {
return null;
}
- protected abstract ResultSet fetchTableMetadata(String tableName) throws
MetaException;
+ protected ResultSet fetchTableMetadata(String tableName) throws
MetaException {
+try {
+ return fetchTablesViaDBMetaData(tableName);
+}
+catch (SQLException sqle) {
+ throw new MetaException("Error while trying to access the table names in
the database" + sqle);
+}
+ }
- protected abstract ResultSet fetchTableNames() throws MetaException;
+ protected ResultSet fetchTableNames() throws MetaException {
+return fetchTableMetadata(null);
+ }
[jira] [Work logged] (HIVE-25656) Get materialized view state based on number of affected rows of transactions
[
https://issues.apache.org/jira/browse/HIVE-25656?focusedWorklogId=682537=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-682537
]
ASF GitHub Bot logged work on HIVE-25656:
-
Author: ASF GitHub Bot
Created on: 17/Nov/21 11:26
Start Date: 17/Nov/21 11:26
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #2756:
URL: https://github.com/apache/hive/pull/2756#discussion_r751148742
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
##
@@ -12847,13 +12849,21 @@ protected void saveViewDefinition() throws
SemanticException {
createVwDesc.setViewExpandedText(expandedText);
}
- private Set getTablesUsed(ParseContext parseCtx) {
-Set tablesUsed = new HashSet<>();
+ private Set getTablesUsed(ParseContext parseCtx) {
+Set tablesUsed = new HashSet<>();
for (TableScanOperator topOp : parseCtx.getTopOps().values()) {
Table table = topOp.getConf().getTableMetadata();
if (!table.isMaterializedTable() && !table.isView()) {
// Add to signature
-tablesUsed.add(table.getFullyQualifiedName());
+SourceTable sourceTable = new SourceTable();
Review comment:
Refactored how `SourceTable` instances are created and in some cases
like the one you mentioned `TableName` is a better choice because the stats
contained by `SourceTable` is not needed here.
Added factory methods to `ql.metadata.Table`
* `SourceTable createSourceTable()` - to create a `SourceTable` instance in
an initial state referencing this table instance.
* `TableName getFullTableName()` - to create a `TableName` using this table
instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 682537)
Time Spent: 2h (was: 1h 50m)
> Get materialized view state based on number of affected rows of transactions
>
>
> Key: HIVE-25656
> URL: https://issues.apache.org/jira/browse/HIVE-25656
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views, Transactions
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> To enable the faster incremental rebuild of materialized views presence of
> update/delete operations on the source tables of the view since the last
> rebuild must be checked. Based on the outcome different plan is generated for
> scenarios in presence of update/delete and insert only operations.
> Currently this is done by querying the COMPLETED_TXN_COMPONENTS table however
> the records from this table is cleaned when MV source tables are compacted.
> This reduces the chances of incremental MV rebuild.
> The goal of this patch is to find an alternative way to store and retrieve
> this information.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-25715) Provide nightly builds
[ https://issues.apache.org/jira/browse/HIVE-25715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-25715: --- > Provide nightly builds > -- > > Key: HIVE-25715 > URL: https://issues.apache.org/jira/browse/HIVE-25715 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > provide nightly builds for the master branch -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-22527) Hive on Tez : Job of merging small files will be submitted into another queue (default queue)
[ https://issues.apache.org/jira/browse/HIVE-22527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22527: Summary: Hive on Tez : Job of merging small files will be submitted into another queue (default queue) (was: Hive on Tez : Job of merging samll files will be submitted into another queue (default queue)) > Hive on Tez : Job of merging small files will be submitted into another queue > (default queue) > - > > Key: HIVE-22527 > URL: https://issues.apache.org/jira/browse/HIVE-22527 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhangbutao >Assignee: zhangbutao >Priority: Blocker > Fix For: 4.0.0 > > Attachments: HIVE-22527-branch-3.1.0.patch, HIVE-22527.01.patch, > HIVE-22527.02.patch, explain with merge files.png, file merge job.png, hive > logs.png > > > Hive on Tez. We enable small file merge configuration with set > *hive.merge.tezfiles=true*. So , There will be another job launched for > merging files after sql job. However, the merge file job is submitted into > another yarn queue, not the queue of current beeline client session. It seems > that the merging files job start a new tez session with new conf which is > different the current session conf, leading to the merging file job goes into > default queue. > > Attachment *hive logs.png* shows that current session queue is > *root.bdoc.production* ( String queueName = session.getQueueName();) incoming > queue name is *null* ( String confQueueName = > conf.get(TezConfiguration.TEZ_QUEUE_NAME);). In fact, we log in to the same > beeline client with *set tez.queue.name=* *root.bdoc.production,* and all > jobs should be submitted into the same queue including file merge job. > [https://github.com/apache/hive/blob/bcc7df95824831a8d2f1524e4048dfc23ab98c19/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L445] > [https://github.com/apache/hive/blob/bcc7df95824831a8d2f1524e4048dfc23ab98c19/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L446] > > Attachment *explain with merge files.png* shows that ** the stage-4 is > individual merge file job which is submitted into another yarn queue（default > queue）， not the queue root.bdoc.production. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25692) ExceptionHandler may mask checked exceptions
[
https://issues.apache.org/jira/browse/HIVE-25692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich resolved HIVE-25692.
-
Fix Version/s: 4.0.0
Resolution: Fixed
merged into master. Thank you [~kkasa] and [~dengzh] for reviewing the changes!
> ExceptionHandler may mask checked exceptions
>
>
> Key: HIVE-25692
> URL: https://issues.apache.org/jira/browse/HIVE-25692
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> HIVE-25055 have changed the way exceptions as rethrowed - but one of the
> methods may let checked exception out without them being declared on the
> method (and avoid the compile time error for it)
> testcase for:
> org.apache.hadoop.hive.metastore.TestExceptionHandler
> {code}
> @Test
> public void testInvalid() throws MetaException {
> try {
> throw new IOException("IOException test");
> } catch (Exception e) {
> throw handleException(e).throwIfInstance(AccessControlException.class,
> IOException.class).defaultMetaException();
> }
> }
> {code}
> this testcase should not compile - as it may throw IOException or
> AccessControlException as well
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25634) Eclipse compiler bumps into AIOBE during ObjectStore compilation
[ https://issues.apache.org/jira/browse/HIVE-25634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich resolved HIVE-25634. - Fix Version/s: 4.0.0 Resolution: Fixed merged into master. Thank you [~dengzh]! > Eclipse compiler bumps into AIOBE during ObjectStore compilation > > > Key: HIVE-25634 > URL: https://issues.apache.org/jira/browse/HIVE-25634 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > this issue seem to have started appearing after HIVE-23633 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-23095) NDV might be overestimated for a table with ~70 value
[ https://issues.apache.org/jira/browse/HIVE-23095?focusedWorklogId=682447=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-682447 ] ASF GitHub Bot logged work on HIVE-23095: - Author: ASF GitHub Bot Created on: 17/Nov/21 09:03 Start Date: 17/Nov/21 09:03 Worklog Time Spent: 10m Work Description: kgyrtkirk commented on pull request #2783: URL: https://github.com/apache/hive/pull/2783#issuecomment-971374080 @kasakrisz could you please take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 682447) Time Spent: 2h 10m (was: 2h) > NDV might be overestimated for a table with ~70 value > - > > Key: HIVE-23095 > URL: https://issues.apache.org/jira/browse/HIVE-23095 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23095.01.patch, HIVE-23095.02.patch, > HIVE-23095.03.patch, HIVE-23095.04.patch, HIVE-23095.04.patch, > HIVE-23095.04.patch, HIVE-23095.05.patch, hll-bench.md > > Time Spent: 2h 10m > Remaining Estimate: 0h > > uncovered during looking into HIVE-23082 > https://issues.apache.org/jira/browse/HIVE-23082?focusedCommentId=17067773=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17067773 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25692) ExceptionHandler may mask checked exceptions
[
https://issues.apache.org/jira/browse/HIVE-25692?focusedWorklogId=682445=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-682445
]
ASF GitHub Bot logged work on HIVE-25692:
-
Author: ASF GitHub Bot
Created on: 17/Nov/21 08:58
Start Date: 17/Nov/21 08:58
Worklog Time Spent: 10m
Work Description: kgyrtkirk merged pull request #2782:
URL: https://github.com/apache/hive/pull/2782
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 682445)
Time Spent: 1h (was: 50m)
> ExceptionHandler may mask checked exceptions
>
>
> Key: HIVE-25692
> URL: https://issues.apache.org/jira/browse/HIVE-25692
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> HIVE-25055 have changed the way exceptions as rethrowed - but one of the
> methods may let checked exception out without them being declared on the
> method (and avoid the compile time error for it)
> testcase for:
> org.apache.hadoop.hive.metastore.TestExceptionHandler
> {code}
> @Test
> public void testInvalid() throws MetaException {
> try {
> throw new IOException("IOException test");
> } catch (Exception e) {
> throw handleException(e).throwIfInstance(AccessControlException.class,
> IOException.class).defaultMetaException();
> }
> }
> {code}
> this testcase should not compile - as it may throw IOException or
> AccessControlException as well
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25531) Remove the core classified hive-exec artifact
[ https://issues.apache.org/jira/browse/HIVE-25531?focusedWorklogId=682435=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-682435 ] ASF GitHub Bot logged work on HIVE-25531: - Author: ASF GitHub Bot Created on: 17/Nov/21 08:41 Start Date: 17/Nov/21 08:41 Worklog Time Spent: 10m Work Description: kgyrtkirk commented on pull request #2648: URL: https://github.com/apache/hive/pull/2648#issuecomment-971357845 @zabetak could you please take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 682435) Remaining Estimate: 0h Time Spent: 10m > Remove the core classified hive-exec artifact > - > > Key: HIVE-25531 > URL: https://issues.apache.org/jira/browse/HIVE-25531 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > * this artifact was introduced in HIVE-7423 > * loading this artifact and the shaded hive-exec (along with the jdbc driver) > could create interesting classpath problems > * if other projects have issues with the shaded hive-exec artifact we must > start fix those problems -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25531) Remove the core classified hive-exec artifact
[ https://issues.apache.org/jira/browse/HIVE-25531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25531: -- Labels: pull-request-available (was: ) > Remove the core classified hive-exec artifact > - > > Key: HIVE-25531 > URL: https://issues.apache.org/jira/browse/HIVE-25531 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > * this artifact was introduced in HIVE-7423 > * loading this artifact and the shaded hive-exec (along with the jdbc driver) > could create interesting classpath problems > * if other projects have issues with the shaded hive-exec artifact we must > start fix those problems -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25714) Some tests are flaky because docker is not able to start in 5 seconds
[
https://issues.apache.org/jira/browse/HIVE-25714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445006#comment-17445006
]
Zoltan Haindrich commented on HIVE-25714:
-
the testrun also contains some info - it seems like the docker log command for
oracle took over a second every time it was executed; maybe it has a lot of
logs?
fyi: [~zabetak]
> Some tests are flaky because docker is not able to start in 5 seconds
> -
>
> Key: HIVE-25714
> URL: https://issues.apache.org/jira/browse/HIVE-25714
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> there are some testruns failing with; and on the test site multiple pods are
> running in parallel - its not an ideal environment for tight deadlines
> {code}
> Unexpected exception java.lang.RuntimeException: Process docker failed to run
> in 5 seconds
> at
> org.apache.hadoop.hive.ql.externalDB.AbstractExternalDB.runCmd(AbstractExternalDB.java:92)
> at
> org.apache.hadoop.hive.ql.externalDB.AbstractExternalDB.launchDockerContainer(AbstractExternalDB.java:123)
> at
> org.apache.hadoop.hive.ql.qoption.QTestDatabaseHandler.beforeTest(QTestDatabaseHandler.java:111)
> at
> org.apache.hadoop.hive.ql.qoption.QTestOptionDispatcher.beforeTest(QTestOptionDispatcher.java:79)
> {code}
> http://ci.hive.apache.org/job/hive-precommit/job/PR-1674/4/testReport/junit/org.apache.hadoop.hive.cli.split19/TestMiniLlapLocalCliDriver/Testing___split_14___PostProcess___testCliDriver_qt_database_all_/
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-25714) Some tests are flaky because docker is not able to start in 5 seconds
[
https://issues.apache.org/jira/browse/HIVE-25714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445005#comment-17445005
]
Zoltan Haindrich commented on HIVE-25714:
-
apparently; the docker log command times out after 5 seconds
https://github.com/apache/hive/blob/dbec7744f04a9389e1bb168e85b5e8f73d357011/itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java#L123
launched flaky-check for it ; I doubt that it will give more info - but it will
have the full hive.log and seeing how long earlier `docker log` cmds took might
help
http://ci.hive.apache.org/job/hive-flaky-check/466/
> Some tests are flaky because docker is not able to start in 5 seconds
> -
>
> Key: HIVE-25714
> URL: https://issues.apache.org/jira/browse/HIVE-25714
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> there are some testruns failing with; and on the test site multiple pods are
> running in parallel - its not an ideal environment for tight deadlines
> {code}
> Unexpected exception java.lang.RuntimeException: Process docker failed to run
> in 5 seconds
> at
> org.apache.hadoop.hive.ql.externalDB.AbstractExternalDB.runCmd(AbstractExternalDB.java:92)
> at
> org.apache.hadoop.hive.ql.externalDB.AbstractExternalDB.launchDockerContainer(AbstractExternalDB.java:123)
> at
> org.apache.hadoop.hive.ql.qoption.QTestDatabaseHandler.beforeTest(QTestDatabaseHandler.java:111)
> at
> org.apache.hadoop.hive.ql.qoption.QTestOptionDispatcher.beforeTest(QTestOptionDispatcher.java:79)
> {code}
> http://ci.hive.apache.org/job/hive-precommit/job/PR-1674/4/testReport/junit/org.apache.hadoop.hive.cli.split19/TestMiniLlapLocalCliDriver/Testing___split_14___PostProcess___testCliDriver_qt_database_all_/
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-25714) Some tests are flaky because docker is not able to start in 5 seconds
[
https://issues.apache.org/jira/browse/HIVE-25714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich reassigned HIVE-25714:
---
> Some tests are flaky because docker is not able to start in 5 seconds
> -
>
> Key: HIVE-25714
> URL: https://issues.apache.org/jira/browse/HIVE-25714
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> there are some testruns failing with; and on the test site multiple pods are
> running in parallel - its not an ideal environment for tight deadlines
> {code}
> Unexpected exception java.lang.RuntimeException: Process docker failed to run
> in 5 seconds
> at
> org.apache.hadoop.hive.ql.externalDB.AbstractExternalDB.runCmd(AbstractExternalDB.java:92)
> at
> org.apache.hadoop.hive.ql.externalDB.AbstractExternalDB.launchDockerContainer(AbstractExternalDB.java:123)
> at
> org.apache.hadoop.hive.ql.qoption.QTestDatabaseHandler.beforeTest(QTestDatabaseHandler.java:111)
> at
> org.apache.hadoop.hive.ql.qoption.QTestOptionDispatcher.beforeTest(QTestOptionDispatcher.java:79)
> {code}
> http://ci.hive.apache.org/job/hive-precommit/job/PR-1674/4/testReport/junit/org.apache.hadoop.hive.cli.split19/TestMiniLlapLocalCliDriver/Testing___split_14___PostProcess___testCliDriver_qt_database_all_/
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
