[jira] [Work logged] (HIVE-25321) [HMS] Advance write Id during AlterTableDropPartition
[ https://issues.apache.org/jira/browse/HIVE-25321?focusedWorklogId=628167=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628167 ] ASF GitHub Bot logged work on HIVE-25321: - Author: ASF GitHub Bot Created on: 27/Jul/21 05:58 Start Date: 27/Jul/21 05:58 Worklog Time Spent: 10m Work Description: kishendas commented on pull request #2465: URL: https://github.com/apache/hive/pull/2465#issuecomment-887232500 @nrg4878 Got a green build -> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-2465/13/pipeline/ . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 628167) Time Spent: 1h 10m (was: 1h) > [HMS] Advance write Id during AlterTableDropPartition > - > > Key: HIVE-25321 > URL: https://issues.apache.org/jira/browse/HIVE-25321 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Kishen Das >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > All DDLs should advance the write ID, so that we can provide consistent data > from the cache, based on the validWriteIds. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25192) Nullify summary for non-native tables
[
https://issues.apache.org/jira/browse/HIVE-25192?focusedWorklogId=628154=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628154
]
ASF GitHub Bot logged work on HIVE-25192:
-
Author: ASF GitHub Bot
Created on: 27/Jul/21 05:29
Start Date: 27/Jul/21 05:29
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2473:
URL: https://github.com/apache/hive/pull/2473#discussion_r677129392
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
##
@@ -2572,18 +2569,24 @@ public static ContentSummary getInputSummary(final
Context ctx, MapWork work, Pa
int numExecutors = getMaxExecutorsForInputListing(ctx.getConf(),
pathNeedProcess.size());
if (numExecutors > 1) {
Review comment:
I am not sure that the problem is with our threads, and not with threads
calling this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 628154)
Time Spent: 1h 40m (was: 1.5h)
> Nullify summary for non-native tables
> -
>
> Key: HIVE-25192
> URL: https://issues.apache.org/jira/browse/HIVE-25192
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> When creating non-native tables like kudu, hbase and so on, we create a
> warehouse location for these tables, though these tables may not use the
> location to store data or for job plan, we should skip getting the input
> summary of non-native tables when optimising joins, as which may cause oom
> problem when the non-native table is on the build side.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25334) Refactor UDF CAST( as TIMESTAMP)
[
https://issues.apache.org/jira/browse/HIVE-25334?focusedWorklogId=628151=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628151
]
ASF GitHub Bot logged work on HIVE-25334:
-
Author: ASF GitHub Bot
Created on: 27/Jul/21 05:27
Start Date: 27/Jul/21 05:27
Worklog Time Spent: 10m
Work Description: ashish-kumar-sharma commented on a change in pull
request #2482:
URL: https://github.com/apache/hive/pull/2482#discussion_r677128701
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java
##
@@ -63,61 +71,44 @@
* otherwise, it's interpreted as timestamp in seconds.
*/
private boolean intToTimestampInSeconds = false;
+ private boolean strict = true;
@Override
public ObjectInspector initialize(ObjectInspector[] arguments) throws
UDFArgumentException {
-if (arguments.length < 1) {
- throw new UDFArgumentLengthException(
- "The function TIMESTAMP requires at least one argument, got "
- + arguments.length);
-}
-
-SessionState ss = SessionState.get();
-if (ss != null) {
- intToTimestampInSeconds =
ss.getConf().getBoolVar(ConfVars.HIVE_INT_TIMESTAMP_CONVERSION_IN_SECONDS);
-}
+checkArgsSize(arguments, 1, 1);
+checkArgPrimitive(arguments, 0);
+checkArgGroups(arguments, 0, tsInputTypes, STRING_GROUP, DATE_GROUP,
NUMERIC_GROUP, VOID_GROUP, BOOLEAN_GROUP);
-try {
- argumentOI = (PrimitiveObjectInspector) arguments[0];
-} catch (ClassCastException e) {
- throw new UDFArgumentException(
- "The function TIMESTAMP takes only primitive types");
-}
+strict = SessionState.get() != null ? SessionState.get().getConf()
+.getBoolVar(ConfVars.HIVE_STRICT_TIMESTAMP_CONVERSION) : new HiveConf()
+.getBoolVar(ConfVars.HIVE_STRICT_TIMESTAMP_CONVERSION);
+intToTimestampInSeconds = SessionState.get() != null ?
SessionState.get().getConf()
+.getBoolVar(ConfVars.HIVE_INT_TIMESTAMP_CONVERSION_IN_SECONDS) : new
HiveConf()
+.getBoolVar(ConfVars.HIVE_INT_TIMESTAMP_CONVERSION_IN_SECONDS);
-if (ss != null &&
ss.getConf().getBoolVar(ConfVars.HIVE_STRICT_TIMESTAMP_CONVERSION)) {
- PrimitiveCategory category = argumentOI.getPrimitiveCategory();
- PrimitiveGrouping group =
PrimitiveObjectInspectorUtils.getPrimitiveGrouping(category);
- if (group == PrimitiveGrouping.NUMERIC_GROUP) {
+if (strict) {
Review comment:
@adesh-rao this is right sequence of validation check. All methods
called with in the initialize() methods throw exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 628151)
Time Spent: 40m (was: 0.5h)
> Refactor UDF CAST( as TIMESTAMP)
> -
>
> Key: HIVE-25334
> URL: https://issues.apache.org/jira/browse/HIVE-25334
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashish Sharma
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Description
> Refactor GenericUDFTimestamp.class
> DOD
> Refactor
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25192) Nullify summary for non-native tables
[
https://issues.apache.org/jira/browse/HIVE-25192?focusedWorklogId=628149=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628149
]
ASF GitHub Bot logged work on HIVE-25192:
-
Author: ASF GitHub Bot
Created on: 27/Jul/21 05:25
Start Date: 27/Jul/21 05:25
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2473:
URL: https://github.com/apache/hive/pull/2473#discussion_r677128092
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
##
@@ -2680,9 +2683,9 @@ public void run() {
total += estimator.estimate(jobConf, scanOp,
-1).getTotalLength();
}
recordSummary(path, new ContentSummary(total, -1, -1));
- } else {
-// todo: should nullify summary for non-native tables,
-// not to be selected as a mapjoin target
+ } else if (handler == null) {
Review comment:
This seems unrelated for me. Maybe a different PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 628149)
Time Spent: 1.5h (was: 1h 20m)
> Nullify summary for non-native tables
> -
>
> Key: HIVE-25192
> URL: https://issues.apache.org/jira/browse/HIVE-25192
> Project: Hive
> Issue Type: Bug
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When creating non-native tables like kudu, hbase and so on, we create a
> warehouse location for these tables, though these tables may not use the
> location to store data or for job plan, we should skip getting the input
> summary of non-native tables when optimising joins, as which may cause oom
> problem when the non-native table is on the build side.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25334) Refactor UDF CAST( as TIMESTAMP)
[
https://issues.apache.org/jira/browse/HIVE-25334?focusedWorklogId=628147=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628147
]
ASF GitHub Bot logged work on HIVE-25334:
-
Author: ASF GitHub Bot
Created on: 27/Jul/21 05:23
Start Date: 27/Jul/21 05:23
Worklog Time Spent: 10m
Work Description: adesh-rao commented on a change in pull request #2482:
URL: https://github.com/apache/hive/pull/2482#discussion_r677126600
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java
##
@@ -63,61 +71,44 @@
* otherwise, it's interpreted as timestamp in seconds.
*/
private boolean intToTimestampInSeconds = false;
+ private boolean strict = true;
@Override
public ObjectInspector initialize(ObjectInspector[] arguments) throws
UDFArgumentException {
-if (arguments.length < 1) {
- throw new UDFArgumentLengthException(
- "The function TIMESTAMP requires at least one argument, got "
- + arguments.length);
-}
-
-SessionState ss = SessionState.get();
-if (ss != null) {
- intToTimestampInSeconds =
ss.getConf().getBoolVar(ConfVars.HIVE_INT_TIMESTAMP_CONVERSION_IN_SECONDS);
-}
+checkArgsSize(arguments, 1, 1);
+checkArgPrimitive(arguments, 0);
+checkArgGroups(arguments, 0, tsInputTypes, STRING_GROUP, DATE_GROUP,
NUMERIC_GROUP, VOID_GROUP, BOOLEAN_GROUP);
-try {
- argumentOI = (PrimitiveObjectInspector) arguments[0];
-} catch (ClassCastException e) {
- throw new UDFArgumentException(
- "The function TIMESTAMP takes only primitive types");
-}
+strict = SessionState.get() != null ? SessionState.get().getConf()
+.getBoolVar(ConfVars.HIVE_STRICT_TIMESTAMP_CONVERSION) : new HiveConf()
+.getBoolVar(ConfVars.HIVE_STRICT_TIMESTAMP_CONVERSION);
+intToTimestampInSeconds = SessionState.get() != null ?
SessionState.get().getConf()
+.getBoolVar(ConfVars.HIVE_INT_TIMESTAMP_CONVERSION_IN_SECONDS) : new
HiveConf()
+.getBoolVar(ConfVars.HIVE_INT_TIMESTAMP_CONVERSION_IN_SECONDS);
-if (ss != null &&
ss.getConf().getBoolVar(ConfVars.HIVE_STRICT_TIMESTAMP_CONVERSION)) {
- PrimitiveCategory category = argumentOI.getPrimitiveCategory();
- PrimitiveGrouping group =
PrimitiveObjectInspectorUtils.getPrimitiveGrouping(category);
- if (group == PrimitiveGrouping.NUMERIC_GROUP) {
+if (strict) {
Review comment:
nit: Move this condition above intToTimestampInSeconds initialization?
If we have to throw exception, let's do it without computing anything else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 628147)
Time Spent: 0.5h (was: 20m)
> Refactor UDF CAST( as TIMESTAMP)
> -
>
> Key: HIVE-25334
> URL: https://issues.apache.org/jira/browse/HIVE-25334
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashish Sharma
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Description
> Refactor GenericUDFTimestamp.class
> DOD
> Refactor
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25334) Refactor UDF CAST( as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-25334?focusedWorklogId=628138=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628138 ] ASF GitHub Bot logged work on HIVE-25334: - Author: ASF GitHub Bot Created on: 27/Jul/21 05:07 Start Date: 27/Jul/21 05:07 Worklog Time Spent: 10m Work Description: ashish-kumar-sharma commented on pull request #2482: URL: https://github.com/apache/hive/pull/2482#issuecomment-887212423 @zabetak @mattmccline-microsoft @adesh-rao Could you please review the PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 628138) Time Spent: 20m (was: 10m) > Refactor UDF CAST( as TIMESTAMP) > - > > Key: HIVE-25334 > URL: https://issues.apache.org/jira/browse/HIVE-25334 > Project: Hive > Issue Type: Sub-task >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Description > Refactor GenericUDFTimestamp.class > DOD > Refactor -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25387) Fix TestMiniLlapLocalCliDriver#replication_metrics_ingest.q
[ https://issues.apache.org/jira/browse/HIVE-25387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Sinha reassigned HIVE-25387: --- Assignee: Haymant Mangla > Fix TestMiniLlapLocalCliDriver#replication_metrics_ingest.q > - > > Key: HIVE-25387 > URL: https://issues.apache.org/jira/browse/HIVE-25387 > Project: Hive > Issue Type: Test > Components: repl, Test >Reporter: Peter Vary >Assignee: Haymant Mangla >Priority: Major > > The test is flaky we need to fix it: > http://ci.hive.apache.org/job/hive-flaky-check/344/ > CC: [~aasha] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25261) RetryingHMSHandler should wrap the MetaException with short description of the target
[
https://issues.apache.org/jira/browse/HIVE-25261?focusedWorklogId=628127=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628127
]
ASF GitHub Bot logged work on HIVE-25261:
-
Author: ASF GitHub Bot
Created on: 27/Jul/21 03:45
Start Date: 27/Jul/21 03:45
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on pull request #2405:
URL: https://github.com/apache/hive/pull/2405#issuecomment-887184438
Hi @zabetak, cloud this fix go in? thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 628127)
Time Spent: 40m (was: 0.5h)
> RetryingHMSHandler should wrap the MetaException with short description of
> the target
> -
>
> Key: HIVE-25261
> URL: https://issues.apache.org/jira/browse/HIVE-25261
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> [RetryingMetaStoreClient|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java#L267-L276]
> relies on the message of MetaException to make decision on retrying the
> current operation when failed. However the RetryingHMSHandler only wraps the
> message into MetaException, which may cause the client unable to retry with
> other metastore instances.
> For example, if we got exception:
> {noformat}
> Caused by: javax.jdo.JDOFatalUserException: Persistence Manager has been
> closed
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.assertIsOpen(JDOPersistenceManager.java:2235)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.evictAll(JDOPersistenceManager.java:481)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:635)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1415)
> at sun.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498){noformat}
> RetryingHMSHandler will throw MetaException with message 'Persistence Manager
> has been closed', which not in the recoverable pattern defined in client.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25014) Optimise ObjectStore::updateTableColumnStatistics
[ https://issues.apache.org/jira/browse/HIVE-25014?focusedWorklogId=628102=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628102 ] ASF GitHub Bot logged work on HIVE-25014: - Author: ASF GitHub Bot Created on: 27/Jul/21 02:00 Start Date: 27/Jul/21 02:00 Worklog Time Spent: 10m Work Description: rbalamohan opened a new pull request #2533: URL: https://github.com/apache/hive/pull/2533 ### Why are the changes needed? https://issues.apache.org/jira/browse/HIVE-25014 Table details were fetched multiple times within column stats updation. Depending on the width of the table, this was called multiple times. Datanuc* moves from persistent -> detached state on commit. https://www.datanucleus.org/products/accessplatform_4_2/jpa/object_lifecycle.html. Since we commit at the end of all processing, fetching table details once is good enough. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? small internal cluster -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 628102) Remaining Estimate: 0h Time Spent: 10m > Optimise ObjectStore::updateTableColumnStatistics > - > > Key: HIVE-25014 > URL: https://issues.apache.org/jira/browse/HIVE-25014 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > "ObjectStore::updateTableColumnStatistics" fetches table level details > multiple times. Need to consider reducing the getTable lookups to avoid > pressure on HMS. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25014) Optimise ObjectStore::updateTableColumnStatistics
[ https://issues.apache.org/jira/browse/HIVE-25014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25014: -- Labels: pull-request-available (was: ) > Optimise ObjectStore::updateTableColumnStatistics > - > > Key: HIVE-25014 > URL: https://issues.apache.org/jira/browse/HIVE-25014 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > "ObjectStore::updateTableColumnStatistics" fetches table level details > multiple times. Need to consider reducing the getTable lookups to avoid > pressure on HMS. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25342) Optimize set_aggr_stats_for for mergeColStats path.
[ https://issues.apache.org/jira/browse/HIVE-25342?focusedWorklogId=628099=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628099 ] ASF GitHub Bot logged work on HIVE-25342: - Author: ASF GitHub Bot Created on: 27/Jul/21 01:42 Start Date: 27/Jul/21 01:42 Worklog Time Spent: 10m Work Description: maheshk114 merged pull request #2489: URL: https://github.com/apache/hive/pull/2489 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 628099) Time Spent: 0.5h (was: 20m) > Optimize set_aggr_stats_for for mergeColStats path. > > > Key: HIVE-25342 > URL: https://issues.apache.org/jira/browse/HIVE-25342 > Project: Hive > Issue Type: Sub-task >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The optimisation used for normal path to use direct sql can also be used for > mergeColStats > path. The stats to be updated can be accumulated in a temp list and that list > can be used to update the stats in a batch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25192) Nullify summary for non-native tables
[ https://issues.apache.org/jira/browse/HIVE-25192?focusedWorklogId=628096=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628096 ] ASF GitHub Bot logged work on HIVE-25192: - Author: ASF GitHub Bot Created on: 27/Jul/21 01:31 Start Date: 27/Jul/21 01:31 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #2473: URL: https://github.com/apache/hive/pull/2473#issuecomment-887140117 Hi @pvary, @belugabehr, @sahilTakiar cloud you take a look if have secs? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 628096) Time Spent: 1h 20m (was: 1h 10m) > Nullify summary for non-native tables > - > > Key: HIVE-25192 > URL: https://issues.apache.org/jira/browse/HIVE-25192 > Project: Hive > Issue Type: Bug >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > When creating non-native tables like kudu, hbase and so on, we create a > warehouse location for these tables, though these tables may not use the > location to store data or for job plan, we should skip getting the input > summary of non-native tables when optimising joins, as which may cause oom > problem when the non-native table is on the build side. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25136) Remove MetaExceptions From RawStore First Cut
[ https://issues.apache.org/jira/browse/HIVE-25136?focusedWorklogId=628083=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628083 ] ASF GitHub Bot logged work on HIVE-25136: - Author: ASF GitHub Bot Created on: 27/Jul/21 00:09 Start Date: 27/Jul/21 00:09 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #2290: URL: https://github.com/apache/hive/pull/2290 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 628083) Time Spent: 1h (was: 50m) > Remove MetaExceptions From RawStore First Cut > - > > Key: HIVE-25136 > URL: https://issues.apache.org/jira/browse/HIVE-25136 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24936) Fix file name parsing and copy file move.
[ https://issues.apache.org/jira/browse/HIVE-24936?focusedWorklogId=628084=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628084 ] ASF GitHub Bot logged work on HIVE-24936: - Author: ASF GitHub Bot Created on: 27/Jul/21 00:09 Start Date: 27/Jul/21 00:09 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #2120: URL: https://github.com/apache/hive/pull/2120#issuecomment-887111691 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 628084) Time Spent: 1h 10m (was: 1h) > Fix file name parsing and copy file move. > - > > Key: HIVE-24936 > URL: https://issues.apache.org/jira/browse/HIVE-24936 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish JP >Assignee: Harish JP >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > The taskId and taskAttemptId is not extracted correctly for copy files > (1_02_copy_3) and when doing a move file of an incompatible copy file the > rename utility generates wrong file names. Ex: 1_02_copy_3 is renamed to > 1_02_copy_3_1 if 1_02_copy_3 already exists, ideally it should be > 1_02_copy_N. > > Incompatible files should be always renamed using the current task or it can > get deleted if the file name conflicts with another task output file. Ex: if > the input file name for a task is 5_01 and is incompatible then if we > move this file, it will be treated as an output file for task id 5, attempt 1 > which if exists will try to generate the same file and fail and another > attempt will be made. There will be 2 files 5_01, 5_02, the deduping > code will remove 5_01 resulting in data loss. There are other scenarios > where the same can happen. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21075) Metastore: Drop partition performance downgrade with Postgres DB
[ https://issues.apache.org/jira/browse/HIVE-21075?focusedWorklogId=628080=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628080 ] ASF GitHub Bot logged work on HIVE-21075: - Author: ASF GitHub Bot Created on: 27/Jul/21 00:09 Start Date: 27/Jul/21 00:09 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #2323: URL: https://github.com/apache/hive/pull/2323#issuecomment-887111645 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 628080) Time Spent: 3h 10m (was: 3h) > Metastore: Drop partition performance downgrade with Postgres DB > > > Key: HIVE-21075 > URL: https://issues.apache.org/jira/browse/HIVE-21075 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Yongzhi Chen >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21075.2.patch > > Time Spent: 3h 10m > Remaining Estimate: 0h > > In order to workaround oracle not supporting limit statement caused > performance issue, HIVE-9447 makes all the backend DB run select count(1) > from SDS where SDS.CD_ID=? to check if the specific CD_ID is referenced in > SDS table before drop a partition. This select count(1) statement does not > scale well in Postgres, and there is no index for CD_ID column in SDS table. > For a SDS table with with 1.5 million rows, select count(1) has average 700ms > without index, while in 10-20ms with index. But the statement before > HIVE-9447( SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1) uses > less than 10ms . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24532) Reduce sink vectorization mixes column types
[
https://issues.apache.org/jira/browse/HIVE-24532?focusedWorklogId=628082=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628082
]
ASF GitHub Bot logged work on HIVE-24532:
-
Author: ASF GitHub Bot
Created on: 27/Jul/21 00:09
Start Date: 27/Jul/21 00:09
Worklog Time Spent: 10m
Work Description: github-actions[bot] closed pull request #2284:
URL: https://github.com/apache/hive/pull/2284
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 628082)
Time Spent: 0.5h (was: 20m)
> Reduce sink vectorization mixes column types
>
>
> Key: HIVE-24532
> URL: https://issues.apache.org/jira/browse/HIVE-24532
> Project: Hive
> Issue Type: Bug
>Reporter: Mustafa İman
>Assignee: Mustafa İman
>Priority: Major
> Labels: pull-request-available
> Attachments: castexception.txt, explainplan.txt
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I do insert overwrite select on a partitioned table. Partition column is
> specified dynamically from select query. "ceil" function is applied on a
> string column to specify partition for each row. Reduce sink gets confused
> about the type of partition column. It leads to following cast exception in
> runtime:
> {code:java}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSerializeRow.serializePrimitiveWrite(VectorSerializeRow.java:452)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSerializeRow.serializeWrite(VectorSerializeRow.java:279)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSerializeRow.serializeWrite(VectorSerializeRow.java:258)
> at
> org.apache.hadoop.hive.ql.exec.vector.reducesink.VectorReduceSinkObjectHashOperator.processKey(VectorReduceSinkObjectHashOperator.java:305)
> ... 28 more
> {code}
> The problem is reproducible by running mvn test
> -Dtest=TestMiniLlapLocalCliDriver -Dqfile=insert0.q with "set
> hive.stats.autogather=false". The additional config option causes insert
> statements to be vectorized so the vectorization bug appears.
> insert0.q:
> [https://github.com/apache/hive/blob/fb046c77257d648d0ee232356bdf665772b28bdd/ql/src/test/queries/clientpositive/insert0.q]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24903) Change String.getBytes() to DFSUtil.string2Bytes(String) to avoid Unsupported Encoding Exception
[ https://issues.apache.org/jira/browse/HIVE-24903?focusedWorklogId=628081=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628081 ] ASF GitHub Bot logged work on HIVE-24903: - Author: ASF GitHub Bot Created on: 27/Jul/21 00:09 Start Date: 27/Jul/21 00:09 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #2127: URL: https://github.com/apache/hive/pull/2127#issuecomment-887111678 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 628081) Time Spent: 40m (was: 0.5h) > Change String.getBytes() to DFSUtil.string2Bytes(String) to avoid Unsupported > Encoding Exception > > > Key: HIVE-24903 > URL: https://issues.apache.org/jira/browse/HIVE-24903 > Project: Hive > Issue Type: Bug >Reporter: dbgp2021 >Priority: Major > Labels: pull-request-available > Attachments: 1.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Hello, > I found that DFSUtil.string2Bytes(String) can be used here instead of > String.getBytes(). Otherwise, the API String.getBytes() may cause potential > risk of UnsupportedEncodingException since the behavior of this method when > the string cannot be encoded in the default charset is unspecified. One > recommended API is DFSUtil.string2Bytes(String) which provides more control > over the encoding process and can avoid this exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25384) Bump ORC to 1.6.9
[ https://issues.apache.org/jira/browse/HIVE-25384?focusedWorklogId=628052=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628052 ] ASF GitHub Bot logged work on HIVE-25384: - Author: ASF GitHub Bot Created on: 26/Jul/21 22:40 Start Date: 26/Jul/21 22:40 Worklog Time Spent: 10m Work Description: dongjoon-hyun commented on pull request #2530: URL: https://github.com/apache/hive/pull/2530#issuecomment-887074768 Thank you so much, @sunchao ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 628052) Time Spent: 1h 10m (was: 1h) > Bump ORC to 1.6.9 > - > > Key: HIVE-25384 > URL: https://issues.apache.org/jira/browse/HIVE-25384 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > ORC-804 affects ORC 1.6.0 ~ 1.6.8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25384) Bump ORC to 1.6.9
[ https://issues.apache.org/jira/browse/HIVE-25384?focusedWorklogId=628054=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628054 ] ASF GitHub Bot logged work on HIVE-25384: - Author: ASF GitHub Bot Created on: 26/Jul/21 22:40 Start Date: 26/Jul/21 22:40 Worklog Time Spent: 10m Work Description: dongjoon-hyun edited a comment on pull request #2530: URL: https://github.com/apache/hive/pull/2530#issuecomment-887074768 Thank you so much, @sunchao and @pgaref ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 628054) Time Spent: 1h 20m (was: 1h 10m) > Bump ORC to 1.6.9 > - > > Key: HIVE-25384 > URL: https://issues.apache.org/jira/browse/HIVE-25384 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > ORC-804 affects ORC 1.6.0 ~ 1.6.8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25190) BytesColumnVector fails when the aggregate size is > 1gb
[
https://issues.apache.org/jira/browse/HIVE-25190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen O'Malley updated HIVE-25190:
-
Fix Version/s: storage-2.9.0
storage-2.8.1
storage-2.7.3
> BytesColumnVector fails when the aggregate size is > 1gb
>
>
> Key: HIVE-25190
> URL: https://issues.apache.org/jira/browse/HIVE-25190
> Project: Hive
> Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
>Priority: Major
> Labels: pull-request-available
> Fix For: storage-2.7.3, storage-2.8.1, storage-2.9.0
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Currently, BytesColumnVector will allocate a buffer for small values (< 1mb),
> but fail with:
> {code:java}
> new RuntimeException("Overflow of newLength. smallBuffer.length="
> + smallBuffer.length + ", nextElemLength=" + nextElemLength);
> {code:java}
> if the aggregate size of the buffer crosses over 1gb.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-25386) hive-storage-api should not have guava compile dependency
[ https://issues.apache.org/jira/browse/HIVE-25386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley resolved HIVE-25386. -- Fix Version/s: storage-2.9.0 storage-2.8.1 Resolution: Fixed I committed this, thanks, Dongjoon! > hive-storage-api should not have guava compile dependency > - > > Key: HIVE-25386 > URL: https://issues.apache.org/jira/browse/HIVE-25386 > Project: Hive > Issue Type: Bug > Components: storage-api >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Blocker > Labels: pull-request-available > Fix For: storage-2.8.1, storage-2.9.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > https://mvnrepository.com/artifact/org.apache.hive/hive-storage-api/2.8.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25384) Bump ORC to 1.6.9
[ https://issues.apache.org/jira/browse/HIVE-25384?focusedWorklogId=627992=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627992 ] ASF GitHub Bot logged work on HIVE-25384: - Author: ASF GitHub Bot Created on: 26/Jul/21 20:31 Start Date: 26/Jul/21 20:31 Worklog Time Spent: 10m Work Description: sunchao merged pull request #2530: URL: https://github.com/apache/hive/pull/2530 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 627992) Time Spent: 50m (was: 40m) > Bump ORC to 1.6.9 > - > > Key: HIVE-25384 > URL: https://issues.apache.org/jira/browse/HIVE-25384 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > ORC-804 affects ORC 1.6.0 ~ 1.6.8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25384) Bump ORC to 1.6.9
[ https://issues.apache.org/jira/browse/HIVE-25384?focusedWorklogId=627993=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627993 ] ASF GitHub Bot logged work on HIVE-25384: - Author: ASF GitHub Bot Created on: 26/Jul/21 20:31 Start Date: 26/Jul/21 20:31 Worklog Time Spent: 10m Work Description: sunchao commented on pull request #2530: URL: https://github.com/apache/hive/pull/2530#issuecomment-887006232 Merged. Thanks @dongjoon-hyun ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 627993) Time Spent: 1h (was: 50m) > Bump ORC to 1.6.9 > - > > Key: HIVE-25384 > URL: https://issues.apache.org/jira/browse/HIVE-25384 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > ORC-804 affects ORC 1.6.0 ~ 1.6.8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25386) hive-storage-api should not have guava compile dependency
[ https://issues.apache.org/jira/browse/HIVE-25386?focusedWorklogId=627911=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627911 ] ASF GitHub Bot logged work on HIVE-25386: - Author: ASF GitHub Bot Created on: 26/Jul/21 18:02 Start Date: 26/Jul/21 18:02 Worklog Time Spent: 10m Work Description: dongjoon-hyun commented on pull request #2531: URL: https://github.com/apache/hive/pull/2531#issuecomment-886911624 Thank you , @pgaref and @omalley ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 627911) Time Spent: 1h 10m (was: 1h) > hive-storage-api should not have guava compile dependency > - > > Key: HIVE-25386 > URL: https://issues.apache.org/jira/browse/HIVE-25386 > Project: Hive > Issue Type: Bug > Components: storage-api >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Blocker > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > https://mvnrepository.com/artifact/org.apache.hive/hive-storage-api/2.8.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25137) getAllWriteEventInfo should go through the HMS client instead of using RawStore directly
[
https://issues.apache.org/jira/browse/HIVE-25137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pravin Sinha resolved HIVE-25137.
-
Resolution: Fixed
Committed to master.
Thanks for the patch [~hsnusonic] !!!
> getAllWriteEventInfo should go through the HMS client instead of using
> RawStore directly
>
>
> Key: HIVE-25137
> URL: https://issues.apache.org/jira/browse/HIVE-25137
> Project: Hive
> Issue Type: Improvement
>Reporter: Pratyush Madhukar
>Assignee: Yu-Wen Lai
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {code:java}
> private List getAllWriteEventInfo(Context withinContext)
> throws Exception {
> String contextDbName =
> StringUtils.normalizeIdentifier(withinContext.replScope.getDbName());
> RawStore rawStore =
> HiveMetaStore.HMSHandler.getMSForConf(withinContext.hiveConf);
> List writeEventInfoList
> = rawStore.getAllWriteEventInfo(eventMessage.getTxnId(),
> contextDbName, null);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25137) getAllWriteEventInfo should go through the HMS client instead of using RawStore directly
[
https://issues.apache.org/jira/browse/HIVE-25137?focusedWorklogId=627907=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627907
]
ASF GitHub Bot logged work on HIVE-25137:
-
Author: ASF GitHub Bot
Created on: 26/Jul/21 17:51
Start Date: 26/Jul/21 17:51
Worklog Time Spent: 10m
Work Description: pkumarsinha merged pull request #2457:
URL: https://github.com/apache/hive/pull/2457
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 627907)
Time Spent: 50m (was: 40m)
> getAllWriteEventInfo should go through the HMS client instead of using
> RawStore directly
>
>
> Key: HIVE-25137
> URL: https://issues.apache.org/jira/browse/HIVE-25137
> Project: Hive
> Issue Type: Improvement
>Reporter: Pratyush Madhukar
>Assignee: Yu-Wen Lai
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {code:java}
> private List getAllWriteEventInfo(Context withinContext)
> throws Exception {
> String contextDbName =
> StringUtils.normalizeIdentifier(withinContext.replScope.getDbName());
> RawStore rawStore =
> HiveMetaStore.HMSHandler.getMSForConf(withinContext.hiveConf);
> List writeEventInfoList
> = rawStore.getAllWriteEventInfo(eventMessage.getTxnId(),
> contextDbName, null);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24945) PTF: Support vectorization for lead/lag functions
[ https://issues.apache.org/jira/browse/HIVE-24945?focusedWorklogId=627890=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627890 ] ASF GitHub Bot logged work on HIVE-24945: - Author: ASF GitHub Bot Created on: 26/Jul/21 17:06 Start Date: 26/Jul/21 17:06 Worklog Time Spent: 10m Work Description: ramesh0201 commented on pull request #2278: URL: https://github.com/apache/hive/pull/2278#issuecomment-886874510 +1. LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 627890) Time Spent: 2h (was: 1h 50m) > PTF: Support vectorization for lead/lag functions > - > > Key: HIVE-24945 > URL: https://issues.apache.org/jira/browse/HIVE-24945 > Project: Hive > Issue Type: Sub-task >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25386) hive-storage-api should not have guava compile dependency
[ https://issues.apache.org/jira/browse/HIVE-25386?focusedWorklogId=627863=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627863 ] ASF GitHub Bot logged work on HIVE-25386: - Author: ASF GitHub Bot Created on: 26/Jul/21 16:29 Start Date: 26/Jul/21 16:29 Worklog Time Spent: 10m Work Description: omalley closed pull request #2531: URL: https://github.com/apache/hive/pull/2531 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 627863) Time Spent: 1h (was: 50m) > hive-storage-api should not have guava compile dependency > - > > Key: HIVE-25386 > URL: https://issues.apache.org/jira/browse/HIVE-25386 > Project: Hive > Issue Type: Bug > Components: storage-api >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Blocker > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > https://mvnrepository.com/artifact/org.apache.hive/hive-storage-api/2.8.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25356) JDBCSplitFilterAboveJoinRule's onMatch method throws exception
[
https://issues.apache.org/jira/browse/HIVE-25356?focusedWorklogId=627860=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627860
]
ASF GitHub Bot logged work on HIVE-25356:
-
Author: ASF GitHub Bot
Created on: 26/Jul/21 16:26
Start Date: 26/Jul/21 16:26
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #2504:
URL: https://github.com/apache/hive/pull/2504#discussion_r676760020
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCAbstractSplitFilterRule.java
##
@@ -127,7 +127,9 @@ public void onMatch(RelOptRuleCall call, SqlDialect
dialect) {
ArrayList validJdbcNode = visitor.getValidJdbcNode();
ArrayList invalidJdbcNode = visitor.getInvalidJdbcNode();
-assert validJdbcNode.size() != 0 && invalidJdbcNode.size() != 0;
+if( validJdbcNode.size() == 0 || invalidJdbcNode.size() == 0) {
+ return;
+}
Review comment:
If we cannot split the filter then we don't need to enter the `onMatch`
method. A few lines above there seems to be a method:
```
public boolean canBeSplit() {
return !validJdbcNode.isEmpty() && !invalidJdbcNode.isEmpty();
}
```
with the same condition.
I think the check should be part of `matches` method so we don't fire the
rule if there is nothing to do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 627860)
Time Spent: 40m (was: 0.5h)
> JDBCSplitFilterAboveJoinRule's onMatch method throws exception
> ---
>
> Key: HIVE-25356
> URL: https://issues.apache.org/jira/browse/HIVE-25356
> Project: Hive
> Issue Type: Bug
>Reporter: Soumyakanti Das
>Assignee: Soumyakanti Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
>
> The stack trace is produced by [JDBCAbstractSplitFilterRule.java#L181
> |https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCAbstractSplitFilterRule.java#L181].
> In the onMatch method, a HiveFilter is being cast to HiveJdbcConverter.
> {code:java}
> java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter cannot be
> cast to
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.jdbc.HiveJdbcConverter
> java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter cannot be
> cast to
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.jdbc.HiveJdbcConverter
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.jdbc.JDBCAbstractSplitFilterRule$JDBCSplitFilterAboveJoinRule.onMatch(JDBCAbstractSplitFilterRule.java:181)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542) at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407) at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:271)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:74)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202) at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189) at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2440)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2406)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPostJoinOrderingTransform(CalcitePlanner.java:2326)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1735)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1588)
> at
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180) at
> org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126) at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1340)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:559)
>
[jira] [Work logged] (HIVE-25356) JDBCSplitFilterAboveJoinRule's onMatch method throws exception
[
https://issues.apache.org/jira/browse/HIVE-25356?focusedWorklogId=627754=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627754
]
ASF GitHub Bot logged work on HIVE-25356:
-
Author: ASF GitHub Bot
Created on: 26/Jul/21 13:52
Start Date: 26/Jul/21 13:52
Worklog Time Spent: 10m
Work Description: soumyakanti3578 commented on a change in pull request
#2504:
URL: https://github.com/apache/hive/pull/2504#discussion_r676623393
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCAbstractSplitFilterRule.java
##
@@ -127,7 +127,9 @@ public void onMatch(RelOptRuleCall call, SqlDialect
dialect) {
ArrayList validJdbcNode = visitor.getValidJdbcNode();
ArrayList invalidJdbcNode = visitor.getInvalidJdbcNode();
-assert validJdbcNode.size() != 0 && invalidJdbcNode.size() != 0;
+if( validJdbcNode.size() == 0 || invalidJdbcNode.size() == 0) {
+ return;
+}
Review comment:
Yes, unfortunately, I was always hitting this assertion error after I
fixed the rel index in the onMatch method. I'll add a test case for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 627754)
Time Spent: 0.5h (was: 20m)
> JDBCSplitFilterAboveJoinRule's onMatch method throws exception
> ---
>
> Key: HIVE-25356
> URL: https://issues.apache.org/jira/browse/HIVE-25356
> Project: Hive
> Issue Type: Bug
>Reporter: Soumyakanti Das
>Assignee: Soumyakanti Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
>
> The stack trace is produced by [JDBCAbstractSplitFilterRule.java#L181
> |https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCAbstractSplitFilterRule.java#L181].
> In the onMatch method, a HiveFilter is being cast to HiveJdbcConverter.
> {code:java}
> java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter cannot be
> cast to
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.jdbc.HiveJdbcConverter
> java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter cannot be
> cast to
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.jdbc.HiveJdbcConverter
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.jdbc.JDBCAbstractSplitFilterRule$JDBCSplitFilterAboveJoinRule.onMatch(JDBCAbstractSplitFilterRule.java:181)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542) at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407) at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:271)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:74)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202) at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189) at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2440)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2406)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPostJoinOrderingTransform(CalcitePlanner.java:2326)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1735)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1588)
> at
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180) at
> org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126) at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1340)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:559)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12512)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:452)
> at
>
[jira] [Work logged] (HIVE-25067) Add more tests to Iceberg partition pruning
[ https://issues.apache.org/jira/browse/HIVE-25067?focusedWorklogId=627738=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627738 ] ASF GitHub Bot logged work on HIVE-25067: - Author: ASF GitHub Bot Created on: 26/Jul/21 13:21 Start Date: 26/Jul/21 13:21 Worklog Time Spent: 10m Work Description: marton-bod commented on a change in pull request #2521: URL: https://github.com/apache/hive/pull/2521#discussion_r676597621 ## File path: iceberg/iceberg-handler/src/test/queries/positive/dynamic_partition_pruning.q ## @@ -0,0 +1,160 @@ +--! qt:dataset:srcpart +set hive.compute.query.using.stats=false; +set hive.mapred.mode=nonstrict; +set hive.explain.user=false; +set hive.optimize.ppd=true; +set hive.ppd.remove.duplicatefilters=true; +set hive.tez.dynamic.partition.pruning=true; +set hive.optimize.metadataonly=false; +set hive.optimize.index.filter=true; +set hive.tez.min.bloom.filter.entries=1; +set hive.tez.bigtable.minsize.semijoin.reduction=1; + +select distinct ds from srcpart; +select distinct hr from srcpart; + +CREATE TABLE srcpart_iceberg (key STRING, value STRING) +PARTITIONED BY (ds STRING, hr STRING) STORED BY iceberg; +INSERT INTO srcpart_iceberg select * from srcpart; + +EXPLAIN create table srcpart_date_n2 as select ds as ds, ds as `date` from srcpart group by ds; +create table srcpart_date_n2 as select ds as ds, ds as `date` from srcpart group by ds; +create table srcpart_hour_n0 as select hr as hr, hr as hour from srcpart group by hr; +create table srcpart_date_hour_n0 as select ds as ds, ds as `date`, hr as hr, hr as hour from srcpart group by ds, hr; +create table srcpart_double_hour_n0 as select (hr*2) as hr, hr as hour from srcpart group by hr; + +-- single column, single key +EXPLAIN select count(*) from srcpart_iceberg join srcpart_date_n2 on (srcpart_iceberg.ds = srcpart_date_n2.ds) where srcpart_date_n2.`date` = '2008-04-08'; +select count(*) from srcpart_iceberg join srcpart_date_n2 on (srcpart_iceberg.ds = srcpart_date_n2.ds) where srcpart_date_n2.`date` = '2008-04-08'; +set hive.tez.dynamic.partition.pruning=false; +EXPLAIN select count(*) from srcpart_iceberg join srcpart_date_n2 on (srcpart_iceberg.ds = srcpart_date_n2.ds) where srcpart_date_n2.`date` = '2008-04-08'; +select count(*) from srcpart_iceberg join srcpart_date_n2 on (srcpart_iceberg.ds = srcpart_date_n2.ds) where srcpart_date_n2.`date` = '2008-04-08'; +set hive.tez.dynamic.partition.pruning=true; +select count(*) from srcpart where ds = '2008-04-08'; + +-- multiple sources, single key +EXPLAIN select count(*) from srcpart_iceberg join srcpart_date_n2 on (srcpart_iceberg.ds = srcpart_date_n2.ds) join srcpart_hour_n0 on (srcpart_iceberg.hr = srcpart_hour_n0.hr) +where srcpart_date_n2.`date` = '2008-04-08' and srcpart_hour_n0.hour = 11; +select count(*) from srcpart_iceberg join srcpart_date_n2 on (srcpart_iceberg.ds = srcpart_date_n2.ds) join srcpart_hour_n0 on (srcpart_iceberg.hr = srcpart_hour_n0.hr) +where srcpart_date_n2.`date` = '2008-04-08' and srcpart_hour_n0.hour = 11; +set hive.tez.dynamic.partition.pruning=false; +EXPLAIN select count(*) from srcpart_iceberg join srcpart_date_n2 on (srcpart_iceberg.ds = srcpart_date_n2.ds) join srcpart_hour_n0 on (srcpart_iceberg.hr = srcpart_hour_n0.hr) +where srcpart_date_n2.`date` = '2008-04-08' and srcpart_hour_n0.hour = 11; +select count(*) from srcpart_iceberg join srcpart_date_n2 on (srcpart_iceberg.ds = srcpart_date_n2.ds) join srcpart_hour_n0 on (srcpart_iceberg.hr = srcpart_hour_n0.hr) +where srcpart_date_n2.`date` = '2008-04-08' and srcpart_hour_n0.hour = 11; +set hive.tez.dynamic.partition.pruning=true; +select count(*) from srcpart where hr = 11 and ds = '2008-04-08'; + +-- multiple columns single source +EXPLAIN select count(*) from srcpart_iceberg join srcpart_date_hour_n0 on (srcpart_iceberg.ds = srcpart_date_hour_n0.ds and srcpart_iceberg.hr = srcpart_date_hour_n0.hr) where srcpart_date_hour_n0.`date` = '2008-04-08' and srcpart_date_hour_n0.hour = 11; +select count(*) from srcpart_iceberg join srcpart_date_hour_n0 on (srcpart_iceberg.ds = srcpart_date_hour_n0.ds and srcpart_iceberg.hr = srcpart_date_hour_n0.hr) where srcpart_date_hour_n0.`date` = '2008-04-08' and srcpart_date_hour_n0.hour = 11; +set hive.tez.dynamic.partition.pruning=false; +EXPLAIN select count(*) from srcpart_iceberg join srcpart_date_hour_n0 on (srcpart_iceberg.ds = srcpart_date_hour_n0.ds and srcpart_iceberg.hr = srcpart_date_hour_n0.hr) where srcpart_date_hour_n0.`date` = '2008-04-08' and srcpart_date_hour_n0.hour = 11; +select count(*) from srcpart_iceberg join srcpart_date_hour_n0 on (srcpart_iceberg.ds = srcpart_date_hour_n0.ds and srcpart_iceberg.hr = srcpart_date_hour_n0.hr) where srcpart_date_hour_n0.`date` = '2008-04-08' and srcpart_date_hour_n0.hour = 11; +set hive.tez.dynamic.partition.pruning=true;
[jira] [Assigned] (HIVE-25389) Fix TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin
[ https://issues.apache.org/jira/browse/HIVE-25389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan reassigned HIVE-25389: --- Assignee: Nikhil Gupta > Fix TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin > > > Key: HIVE-25389 > URL: https://issues.apache.org/jira/browse/HIVE-25389 > Project: Hive > Issue Type: Test >Reporter: Peter Vary >Assignee: Nikhil Gupta >Priority: Major > > Running only the single test method: > TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin fails > immediately, but even if we run the whole test class, it fails sometimes. > - Single test method run: http://ci.hive.apache.org/job/hive-flaky-check/348/ > - Whole class run with fail: > http://ci.hive.apache.org/job/hive-flaky-check/351/ > We need to fix this before reenabling the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25067) Add more tests to Iceberg partition pruning
[ https://issues.apache.org/jira/browse/HIVE-25067?focusedWorklogId=627715=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627715 ] ASF GitHub Bot logged work on HIVE-25067: - Author: ASF GitHub Bot Created on: 26/Jul/21 12:58 Start Date: 26/Jul/21 12:58 Worklog Time Spent: 10m Work Description: pvary commented on a change in pull request #2521: URL: https://github.com/apache/hive/pull/2521#discussion_r676579166 ## File path: iceberg/iceberg-handler/src/test/queries/positive/dynamic_partition_pruning.q ## @@ -0,0 +1,161 @@ +--! qt:dataset:srcpart +SET hive.vectorized.execution.enabled=false; Review comment: It is working - this part left here because the original patch was before vectorization was enabled. Removed the line. Thanks, Peter -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 627715) Time Spent: 0.5h (was: 20m) > Add more tests to Iceberg partition pruning > --- > > Key: HIVE-25067 > URL: https://issues.apache.org/jira/browse/HIVE-25067 > Project: Hive > Issue Type: Test >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > As we have qtest for Iceberg now, it would be good to add some partition > pruning qtest to have better coverage -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25389) Fix TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin
[ https://issues.apache.org/jira/browse/HIVE-25389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387312#comment-17387312 ] Peter Vary commented on HIVE-25389: --- I had to disable the whole class, because disabling only the flaky one made the other tests flaky > Fix TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin > > > Key: HIVE-25389 > URL: https://issues.apache.org/jira/browse/HIVE-25389 > Project: Hive > Issue Type: Test >Reporter: Peter Vary >Priority: Major > > Running only the single test method: > TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin fails > immediately, but even if we run the whole test class, it fails sometimes. > - Single test method run: http://ci.hive.apache.org/job/hive-flaky-check/348/ > - Whole class run with fail: > http://ci.hive.apache.org/job/hive-flaky-check/351/ > We need to fix this before reenabling the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24235) Drop and recreate table during MR compaction leaves behind base/delta directory
[
https://issues.apache.org/jira/browse/HIVE-24235?focusedWorklogId=627701=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627701
]
ASF GitHub Bot logged work on HIVE-24235:
-
Author: ASF GitHub Bot
Created on: 26/Jul/21 12:22
Start Date: 26/Jul/21 12:22
Worklog Time Spent: 10m
Work Description: klcopp commented on a change in pull request #2503:
URL: https://github.com/apache/hive/pull/2503#discussion_r676553716
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##
@@ -716,6 +757,13 @@ void open(CompactionInfo ci) throws TException {
}
this.txnId = msc.openTxn(ci.runAs, TxnType.COMPACTION);
status = TxnStatus.OPEN;
+
+ LockRequest lockRequest = createLockRequest(ci, txnId);
+ LockResponse res = msc.lock(lockRequest);
Review comment:
Or does this happen automatically on commit/abort?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 627701)
Time Spent: 1h (was: 50m)
> Drop and recreate table during MR compaction leaves behind base/delta
> directory
> ---
>
> Key: HIVE-24235
> URL: https://issues.apache.org/jira/browse/HIVE-24235
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> If a table is dropped and recreated during MR compaction, the table directory
> and a base (or delta, if minor compaction) directory could be created, with
> or without data, while the table "does not exist".
> E.g.
> {code:java}
> create table c (i int) stored as orc tblproperties
> ("NO_AUTO_COMPACTION"="true", "transactional"="true");
> insert into c values (9);
> insert into c values (9);
> alter table c compact 'major';
> While compaction job is running: {
> drop table c;
> create table c (i int) stored as orc tblproperties
> ("NO_AUTO_COMPACTION"="true", "transactional"="true");
> }
> {code}
> The table directory should be empty, but table directory could look like this
> after the job is finished:
> {code:java}
> Oct 6 14:23 c/base_002_v101/._orc_acid_version.crc
> Oct 6 14:23 c/base_002_v101/.bucket_0.crc
> Oct 6 14:23 c/base_002_v101/_orc_acid_version
> Oct 6 14:23 c/base_002_v101/bucket_0
> {code}
> or perhaps just:
> {code:java}
> Oct 6 14:23 c/base_002_v101/._orc_acid_version.crc
> Oct 6 14:23 c/base_002_v101/_orc_acid_version
> {code}
> Insert another row and you have:
> {code:java}
> Oct 6 14:33 base_002_v101/
> Oct 6 14:33 base_002_v101/._orc_acid_version.crc
> Oct 6 14:33 base_002_v101/.bucket_0.crc
> Oct 6 14:33 base_002_v101/_orc_acid_version
> Oct 6 14:33 base_002_v101/bucket_0
> Oct 6 14:35 delta_001_001_/._orc_acid_version.crc
> Oct 6 14:35 delta_001_001_/.bucket_0_0.crc
> Oct 6 14:35 delta_001_001_/_orc_acid_version
> Oct 6 14:35 delta_001_001_/bucket_0_0
> {code}
> Selecting from the table will result in this error because the highest valid
> writeId for this table is 1:
> {code:java}
> thrift.ThriftCLIService: Error fetching results:
> org.apache.hive.service.cli.HiveSQLException: Unable to get the next row set
> at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:482)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> ...
> Caused by: java.io.IOException: java.lang.RuntimeException: ORC split
> generation failed with exception: java.io.IOException: Not enough history
> available for (1,x). Oldest available base:
> .../warehouse/b/base_004_v092
> {code}
> Solution: Resolve the table again after compaction is finished; compare the
> id with the table id from when compaction began. If the ids do not match,
> abort the compaction's transaction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24235) Drop and recreate table during MR compaction leaves behind base/delta directory
[
https://issues.apache.org/jira/browse/HIVE-24235?focusedWorklogId=627700=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627700
]
ASF GitHub Bot logged work on HIVE-24235:
-
Author: ASF GitHub Bot
Created on: 26/Jul/21 12:21
Start Date: 26/Jul/21 12:21
Worklog Time Spent: 10m
Work Description: klcopp commented on a change in pull request #2503:
URL: https://github.com/apache/hive/pull/2503#discussion_r676552709
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##
@@ -716,6 +757,13 @@ void open(CompactionInfo ci) throws TException {
}
this.txnId = msc.openTxn(ci.runAs, TxnType.COMPACTION);
status = TxnStatus.OPEN;
+
+ LockRequest lockRequest = createLockRequest(ci, txnId);
+ LockResponse res = msc.lock(lockRequest);
Review comment:
I might have missed it but do we unlock this as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 627700)
Time Spent: 50m (was: 40m)
> Drop and recreate table during MR compaction leaves behind base/delta
> directory
> ---
>
> Key: HIVE-24235
> URL: https://issues.apache.org/jira/browse/HIVE-24235
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> If a table is dropped and recreated during MR compaction, the table directory
> and a base (or delta, if minor compaction) directory could be created, with
> or without data, while the table "does not exist".
> E.g.
> {code:java}
> create table c (i int) stored as orc tblproperties
> ("NO_AUTO_COMPACTION"="true", "transactional"="true");
> insert into c values (9);
> insert into c values (9);
> alter table c compact 'major';
> While compaction job is running: {
> drop table c;
> create table c (i int) stored as orc tblproperties
> ("NO_AUTO_COMPACTION"="true", "transactional"="true");
> }
> {code}
> The table directory should be empty, but table directory could look like this
> after the job is finished:
> {code:java}
> Oct 6 14:23 c/base_002_v101/._orc_acid_version.crc
> Oct 6 14:23 c/base_002_v101/.bucket_0.crc
> Oct 6 14:23 c/base_002_v101/_orc_acid_version
> Oct 6 14:23 c/base_002_v101/bucket_0
> {code}
> or perhaps just:
> {code:java}
> Oct 6 14:23 c/base_002_v101/._orc_acid_version.crc
> Oct 6 14:23 c/base_002_v101/_orc_acid_version
> {code}
> Insert another row and you have:
> {code:java}
> Oct 6 14:33 base_002_v101/
> Oct 6 14:33 base_002_v101/._orc_acid_version.crc
> Oct 6 14:33 base_002_v101/.bucket_0.crc
> Oct 6 14:33 base_002_v101/_orc_acid_version
> Oct 6 14:33 base_002_v101/bucket_0
> Oct 6 14:35 delta_001_001_/._orc_acid_version.crc
> Oct 6 14:35 delta_001_001_/.bucket_0_0.crc
> Oct 6 14:35 delta_001_001_/_orc_acid_version
> Oct 6 14:35 delta_001_001_/bucket_0_0
> {code}
> Selecting from the table will result in this error because the highest valid
> writeId for this table is 1:
> {code:java}
> thrift.ThriftCLIService: Error fetching results:
> org.apache.hive.service.cli.HiveSQLException: Unable to get the next row set
> at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:482)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> ...
> Caused by: java.io.IOException: java.lang.RuntimeException: ORC split
> generation failed with exception: java.io.IOException: Not enough history
> available for (1,x). Oldest available base:
> .../warehouse/b/base_004_v092
> {code}
> Solution: Resolve the table again after compaction is finished; compare the
> id with the table id from when compaction began. If the ids do not match,
> abort the compaction's transaction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-25190) BytesColumnVector fails when the aggregate size is > 1gb
[
https://issues.apache.org/jira/browse/HIVE-25190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Panagiotis Garefalakis resolved HIVE-25190.
---
Resolution: Fixed
> BytesColumnVector fails when the aggregate size is > 1gb
>
>
> Key: HIVE-25190
> URL: https://issues.apache.org/jira/browse/HIVE-25190
> Project: Hive
> Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Currently, BytesColumnVector will allocate a buffer for small values (< 1mb),
> but fail with:
> {code:java}
> new RuntimeException("Overflow of newLength. smallBuffer.length="
> + smallBuffer.length + ", nextElemLength=" + nextElemLength);
> {code:java}
> if the aggregate size of the buffer crosses over 1gb.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-25190) BytesColumnVector fails when the aggregate size is > 1gb
[
https://issues.apache.org/jira/browse/HIVE-25190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387281#comment-17387281
]
Panagiotis Garefalakis commented on HIVE-25190:
---
Thanks [~dongjoon] -- I was hesitating to close as we need a new storage-api
version (as the fix) -- should be 2.8.1
> BytesColumnVector fails when the aggregate size is > 1gb
>
>
> Key: HIVE-25190
> URL: https://issues.apache.org/jira/browse/HIVE-25190
> Project: Hive
> Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Currently, BytesColumnVector will allocate a buffer for small values (< 1mb),
> but fail with:
> {code:java}
> new RuntimeException("Overflow of newLength. smallBuffer.length="
> + smallBuffer.length + ", nextElemLength=" + nextElemLength);
> {code:java}
> if the aggregate size of the buffer crosses over 1gb.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25365) Insufficient privileges to show partitions when partition columns are authorized
[ https://issues.apache.org/jira/browse/HIVE-25365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-25365: --- Priority: Major (was: Minor) > Insufficient privileges to show partitions when partition columns are > authorized > > > Key: HIVE-25365 > URL: https://issues.apache.org/jira/browse/HIVE-25365 > Project: Hive > Issue Type: Bug > Components: Authorization >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When the privileges of partition columns have granted to user, showing > partitions still needs select privilege on the table, though they are able to > query from partition columns. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25389) Fix TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin
[ https://issues.apache.org/jira/browse/HIVE-25389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387278#comment-17387278 ] Peter Vary commented on HIVE-25389: --- CC: [~sankarh], [~gupta.nikhil0007], [~ashish-kumar-sharma] > Fix TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin > > > Key: HIVE-25389 > URL: https://issues.apache.org/jira/browse/HIVE-25389 > Project: Hive > Issue Type: Test >Reporter: Peter Vary >Priority: Major > > Running only the single test method: > TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin fails > immediately, but even if we run the whole test class, it fails. > - Single test method run: http://ci.hive.apache.org/job/hive-flaky-check/348/ > - Whole class run with fail: > http://ci.hive.apache.org/job/hive-flaky-check/351/ > We need to fix this before reenabling the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25389) Fix TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin
[ https://issues.apache.org/jira/browse/HIVE-25389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-25389: -- Description: Running only the single test method: TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin fails immediately, but even if we run the whole test class, it fails sometimes. - Single test method run: http://ci.hive.apache.org/job/hive-flaky-check/348/ - Whole class run with fail: http://ci.hive.apache.org/job/hive-flaky-check/351/ We need to fix this before reenabling the tests. was: Running only the single test method: TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin fails immediately, but even if we run the whole test class, it fails. - Single test method run: http://ci.hive.apache.org/job/hive-flaky-check/348/ - Whole class run with fail: http://ci.hive.apache.org/job/hive-flaky-check/351/ We need to fix this before reenabling the tests. > Fix TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin > > > Key: HIVE-25389 > URL: https://issues.apache.org/jira/browse/HIVE-25389 > Project: Hive > Issue Type: Test >Reporter: Peter Vary >Priority: Major > > Running only the single test method: > TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin fails > immediately, but even if we run the whole test class, it fails sometimes. > - Single test method run: http://ci.hive.apache.org/job/hive-flaky-check/348/ > - Whole class run with fail: > http://ci.hive.apache.org/job/hive-flaky-check/351/ > We need to fix this before reenabling the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25389) Fix TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin
[ https://issues.apache.org/jira/browse/HIVE-25389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-25389: -- Description: Running only the single test method: TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin fails immediately, but even if we run the whole test class, it fails. - Single test method run: http://ci.hive.apache.org/job/hive-flaky-check/348/ - Whole class run with fail: http://ci.hive.apache.org/job/hive-flaky-check/351/ We need to fix this before reenabling the tests. was: Running only the single test method: TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin fails immediately, but even if we run the whole test class, it fails. - Single test method run: http://ci.hive.apache.org/job/hive-flaky-check/348/ - Whole class run with fail: http://ci.hive.apache.org/job/hive-flaky-check/351/ We need to fix this before reenabling the tests. > Fix TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin > > > Key: HIVE-25389 > URL: https://issues.apache.org/jira/browse/HIVE-25389 > Project: Hive > Issue Type: Test >Reporter: Peter Vary >Priority: Major > > Running only the single test method: > TestKillQueryWithAuthorizationDisabled#testKillQueryByTagAdmin fails > immediately, but even if we run the whole test class, it fails. > - Single test method run: http://ci.hive.apache.org/job/hive-flaky-check/348/ > - Whole class run with fail: > http://ci.hive.apache.org/job/hive-flaky-check/351/ > We need to fix this before reenabling the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25386) hive-storage-api should not have guava compile dependency
[ https://issues.apache.org/jira/browse/HIVE-25386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis reassigned HIVE-25386: - Assignee: Dongjoon Hyun > hive-storage-api should not have guava compile dependency > - > > Key: HIVE-25386 > URL: https://issues.apache.org/jira/browse/HIVE-25386 > Project: Hive > Issue Type: Bug > Components: storage-api >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Blocker > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > https://mvnrepository.com/artifact/org.apache.hive/hive-storage-api/2.8.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25374) Fix TestReplicationOnHDFSEncryptedZones
[
https://issues.apache.org/jira/browse/HIVE-25374?focusedWorklogId=627660=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627660
]
ASF GitHub Bot logged work on HIVE-25374:
-
Author: ASF GitHub Bot
Created on: 26/Jul/21 11:15
Start Date: 26/Jul/21 11:15
Worklog Time Spent: 10m
Work Description: ArkoSharma commented on a change in pull request #2519:
URL: https://github.com/apache/hive/pull/2519#discussion_r676511419
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationOnHDFSEncryptedZones.java
##
@@ -113,46 +129,96 @@ public void
targetAndSourceHaveDifferentEncryptionZoneKeys() throws Throwable {
"'" + HiveConf.ConfVars.HIVE_DISTCP_DOAS_USER.varname + "'='"
+ UserGroupInformation.getCurrentUser().getUserName()
+"'");
WarehouseInstance.Tuple tuple =
-primary.run("use " + primaryDbName)
-.run("create table encrypted_table (id int, value string)")
-.run("insert into table encrypted_table values (1,'value1')")
-.run("insert into table encrypted_table values (2,'value2')")
+primary.run("use " + primaryDbName)
+.run("create table encrypted_table (id int, value string)")
+.run("insert into table encrypted_table values
(1,'value1')")
+.run("insert into table encrypted_table values
(2,'value2')")
+.dump(primaryDbName, dumpWithClause);
+
+replica
+.run("repl load " + primaryDbName + " into " + replicatedDbName
++ " with('hive.repl.add.raw.reserved.namespace'='true', "
++ "'hive.repl.replica.external.table.base.dir'='" +
replica.externalTableWarehouseRoot + "', "
++ "'hive.exec.copyfile.maxsize'='0',
'distcp.options.skipcrccheck'='')")
+.run("use " + replicatedDbName)
+.run("repl status " + replicatedDbName)
+.verifyResult(tuple.lastReplicationId);
+
+try {
+ replica
+ .run("select value from encrypted_table")
+ .verifyResults(new String[] { "value1", "value2" });
+ Assert.fail("Src EZKey shouldn't be present on target");
+} catch (IOException e) {
+ Assert.assertTrue(e.getCause().getMessage().contains("KeyVersion name
'test_key@0' does not exist"));
+}
+
+//read should pass without raw-byte distcp
+dumpWithClause = Arrays.asList( "'" +
HiveConf.ConfVars.REPL_EXTERNAL_TABLE_BASE_DIR.varname + "'='"
++ replica.externalTableWarehouseRoot + "'");
+tuple = primary.run("use " + primaryDbName)
+.run("create external table encrypted_table2 (id int, value
string)")
+.run("insert into table encrypted_table2 values (1,'value1')")
+.run("insert into table encrypted_table2 values (2,'value2')")
.dump(primaryDbName, dumpWithClause);
replica
-.run("repl load " + primaryDbName + " into " + replicatedDbName
-+ " with('hive.repl.add.raw.reserved.namespace'='true', "
-+ "'hive.repl.replica.external.table.base.dir'='" +
replica.externalTableWarehouseRoot + "', "
-+ "'distcp.options.pugpbx'='',
'distcp.options.skipcrccheck'='')")
-.run("use " + replicatedDbName)
-.run("repl status " + replicatedDbName)
-.verifyResult(tuple.lastReplicationId)
-.run("select value from encrypted_table")
-.verifyFailure(new String[] { "value1", "value2" });
+.run("repl load " + primaryDbName + " into " + replicatedDbName
++ " with('hive.repl.replica.external.table.base.dir'='" +
replica.externalTableWarehouseRoot + "', "
++ "'hive.exec.copyfile.maxsize'='0',
'distcp.options.skipcrccheck'='')")
+.run("use " + replicatedDbName)
+.run("repl status " + replicatedDbName)
+.verifyResult(tuple.lastReplicationId)
+.run("select value from encrypted_table2")
+.verifyResults(new String[] { "value1", "value2" });
}
@Ignore("this is ignored as minidfs cluster as of writing this test looked
like did not copy the "
+ "files correctly")
@Test
public void targetAndSourceHaveSameEncryptionZoneKeys() throws Throwable {
-WarehouseInstance replica = new WarehouseInstance(LOG, miniDFSCluster,
+String replicaBaseDir =
Files.createTempDirectory("replica2").toFile().getAbsolutePath();
+Configuration replicaConf = new Configuration();
+replicaConf.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR, replicaBaseDir);
+replicaConf.set("dfs.client.use.datanode.hostname", "true");
+replicaConf.set("hadoop.proxyuser." + Utils.getUGI().getShortUserName() +
".hosts", "*");
+replicaConf.set("hadoop.security.key.provider.path",
[jira] [Work logged] (HIVE-25067) Add more tests to Iceberg partition pruning
[ https://issues.apache.org/jira/browse/HIVE-25067?focusedWorklogId=627640=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627640 ] ASF GitHub Bot logged work on HIVE-25067: - Author: ASF GitHub Bot Created on: 26/Jul/21 10:44 Start Date: 26/Jul/21 10:44 Worklog Time Spent: 10m Work Description: marton-bod commented on a change in pull request #2521: URL: https://github.com/apache/hive/pull/2521#discussion_r676491652 ## File path: iceberg/iceberg-handler/src/test/queries/positive/dynamic_partition_pruning.q ## @@ -0,0 +1,161 @@ +--! qt:dataset:srcpart +SET hive.vectorized.execution.enabled=false; Review comment: Do we need to turn this off? Will DPP work with vectorization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 627640) Time Spent: 20m (was: 10m) > Add more tests to Iceberg partition pruning > --- > > Key: HIVE-25067 > URL: https://issues.apache.org/jira/browse/HIVE-25067 > Project: Hive > Issue Type: Test >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > As we have qtest for Iceberg now, it would be good to add some partition > pruning qtest to have better coverage -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25368) Code does not build in IDE and a small fix
[
https://issues.apache.org/jira/browse/HIVE-25368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary resolved HIVE-25368.
---
Fix Version/s: 4.0.0
Resolution: Fixed
Pushed to master.
Thanks [~kgyrtkirk] for the review!
> Code does not build in IDE and a small fix
> --
>
> Key: HIVE-25368
> URL: https://issues.apache.org/jira/browse/HIVE-25368
> Project: Hive
> Issue Type: Task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The code does not build in IntelliJ because of the generic usage.
> Also there is a small test case issue in {{WarehouseInstance.java}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-25377) Creating Iceberg table where some columns has comments but the last one does not fails
[
https://issues.apache.org/jira/browse/HIVE-25377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary resolved HIVE-25377.
---
Fix Version/s: 4.0.0
Resolution: Fixed
Pushed to master.
Thanks for the review [~lpinter]!
> Creating Iceberg table where some columns has comments but the last one does
> not fails
> --
>
> Key: HIVE-25377
> URL: https://issues.apache.org/jira/browse/HIVE-25377
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently this create command fails:
> {code}
> CREATE TABLE srcpart_iceberg (key STRING COMMENT 'This is the key', value
> STRING) STORED BY iceberg
> {code}
> The exception is this:
> {code}
> Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
> compiling statement: FAILED: Execution Error, return code 4 from
> org.apache.hadoop.hive.ql.ddl.DDLTask. java.lang.IndexOutOfBoundsException:
> Index: 2, Size: 2
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:376)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:246)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:270)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:286)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:544)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:512)
> at
> org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:270)
> at
> org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:142)
> ... 42 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1276)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1281)
> at
> org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.createTableNonReplaceMode(CreateTableOperation.java:139)
> at
> org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:97)
> at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:84)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:361)
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:334)
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:245)
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:108)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:348)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:204)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:153)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:148)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:164)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:233)
> ... 48 more
> Caused by: java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
> at java.util.ArrayList.rangeCheck(ArrayList.java:657)
> at java.util.ArrayList.get(ArrayList.java:433)
> at
> org.apache.iceberg.hive.HiveSchemaConverter.convertInternal(HiveSchemaConverter.java:67)
> at
> org.apache.iceberg.hive.HiveSchemaConverter.convert(HiveSchemaConverter.java:55)
> at
> org.apache.iceberg.hive.HiveSchemaUtil.convert(HiveSchemaUtil.java:121)
> at
> org.apache.iceberg.mr.hive.HiveIcebergSerDe.hiveSchemaOrThrow(HiveIcebergSerDe.java:270)
> at
> org.apache.iceberg.mr.hive.HiveIcebergSerDe.initialize(HiveIcebergSerDe.java:116)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:323)
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:305)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1234)
> ... 64 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25377) Creating Iceberg table where some columns has comments but the last one does not fails
[
https://issues.apache.org/jira/browse/HIVE-25377?focusedWorklogId=627616=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627616
]
ASF GitHub Bot logged work on HIVE-25377:
-
Author: ASF GitHub Bot
Created on: 26/Jul/21 09:45
Start Date: 26/Jul/21 09:45
Worklog Time Spent: 10m
Work Description: pvary merged pull request #2522:
URL: https://github.com/apache/hive/pull/2522
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 627616)
Time Spent: 20m (was: 10m)
> Creating Iceberg table where some columns has comments but the last one does
> not fails
> --
>
> Key: HIVE-25377
> URL: https://issues.apache.org/jira/browse/HIVE-25377
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently this create command fails:
> {code}
> CREATE TABLE srcpart_iceberg (key STRING COMMENT 'This is the key', value
> STRING) STORED BY iceberg
> {code}
> The exception is this:
> {code}
> Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
> compiling statement: FAILED: Execution Error, return code 4 from
> org.apache.hadoop.hive.ql.ddl.DDLTask. java.lang.IndexOutOfBoundsException:
> Index: 2, Size: 2
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:376)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:246)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:270)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:286)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:544)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:512)
> at
> org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:270)
> at
> org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:142)
> ... 42 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1276)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1281)
> at
> org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.createTableNonReplaceMode(CreateTableOperation.java:139)
> at
> org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:97)
> at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:84)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:361)
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:334)
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:245)
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:108)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:348)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:204)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:153)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:148)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:164)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:233)
> ... 48 more
> Caused by: java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
> at java.util.ArrayList.rangeCheck(ArrayList.java:657)
> at java.util.ArrayList.get(ArrayList.java:433)
> at
> org.apache.iceberg.hive.HiveSchemaConverter.convertInternal(HiveSchemaConverter.java:67)
> at
> org.apache.iceberg.hive.HiveSchemaConverter.convert(HiveSchemaConverter.java:55)
> at
> org.apache.iceberg.hive.HiveSchemaUtil.convert(HiveSchemaUtil.java:121)
> at
> org.apache.iceberg.mr.hive.HiveIcebergSerDe.hiveSchemaOrThrow(HiveIcebergSerDe.java:270)
> at
> org.apache.iceberg.mr.hive.HiveIcebergSerDe.initialize(HiveIcebergSerDe.java:116)
> at
>
[jira] [Work logged] (HIVE-25356) JDBCSplitFilterAboveJoinRule's onMatch method throws exception
[
https://issues.apache.org/jira/browse/HIVE-25356?focusedWorklogId=627615=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627615
]
ASF GitHub Bot logged work on HIVE-25356:
-
Author: ASF GitHub Bot
Created on: 26/Jul/21 09:42
Start Date: 26/Jul/21 09:42
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #2504:
URL: https://github.com/apache/hive/pull/2504#discussion_r676448741
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCAbstractSplitFilterRule.java
##
@@ -127,7 +127,9 @@ public void onMatch(RelOptRuleCall call, SqlDialect
dialect) {
ArrayList validJdbcNode = visitor.getValidJdbcNode();
ArrayList invalidJdbcNode = visitor.getInvalidJdbcNode();
-assert validJdbcNode.size() != 0 && invalidJdbcNode.size() != 0;
+if( validJdbcNode.size() == 0 || invalidJdbcNode.size() == 0) {
+ return;
+}
Review comment:
Did you hit this assertion error? My understanding is that the previous
code guarantees that this should never happen so the assertion seems valid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 627615)
Time Spent: 20m (was: 10m)
> JDBCSplitFilterAboveJoinRule's onMatch method throws exception
> ---
>
> Key: HIVE-25356
> URL: https://issues.apache.org/jira/browse/HIVE-25356
> Project: Hive
> Issue Type: Bug
>Reporter: Soumyakanti Das
>Assignee: Soumyakanti Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
>
> The stack trace is produced by [JDBCAbstractSplitFilterRule.java#L181
> |https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCAbstractSplitFilterRule.java#L181].
> In the onMatch method, a HiveFilter is being cast to HiveJdbcConverter.
> {code:java}
> java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter cannot be
> cast to
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.jdbc.HiveJdbcConverter
> java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter cannot be
> cast to
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.jdbc.HiveJdbcConverter
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.jdbc.JDBCAbstractSplitFilterRule$JDBCSplitFilterAboveJoinRule.onMatch(JDBCAbstractSplitFilterRule.java:181)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542) at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407) at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:271)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:74)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202) at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189) at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2440)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2406)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPostJoinOrderingTransform(CalcitePlanner.java:2326)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1735)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1588)
> at
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180) at
> org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126) at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1340)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:559)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12512)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:452)
> at
>
[jira] [Work logged] (HIVE-25115) Compaction queue entries may accumulate in "ready for cleaning" state
[
https://issues.apache.org/jira/browse/HIVE-25115?focusedWorklogId=627596=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627596
]
ASF GitHub Bot logged work on HIVE-25115:
-
Author: ASF GitHub Bot
Created on: 26/Jul/21 09:15
Start Date: 26/Jul/21 09:15
Worklog Time Spent: 10m
Work Description: klcopp commented on a change in pull request #2277:
URL: https://github.com/apache/hive/pull/2277#discussion_r676430685
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##
@@ -282,15 +282,12 @@ private ValidReaderWriteIdList
getValidCleanerWriteIdList(CompactionInfo ci, Tab
assert rsp != null && rsp.getTblValidWriteIdsSize() == 1;
ValidReaderWriteIdList validWriteIdList =
TxnCommonUtils.createValidReaderWriteIdList(rsp.getTblValidWriteIds().get(0));
-boolean delayedCleanupEnabled =
conf.getBoolVar(HIVE_COMPACTOR_DELAYED_CLEANUP_ENABLED);
-if (delayedCleanupEnabled) {
- /*
- * If delayed cleanup enabled, we need to filter the obsoletes dir list,
to only remove directories that were made obsolete by this compaction
- * If we have a higher retentionTime it is possible for a second
compaction to run on the same partition. Cleaning up the first compaction
- * should not touch the newer obsolete directories to not to violate the
retentionTime for those.
- */
- validWriteIdList =
validWriteIdList.updateHighWatermark(ci.highestWriteId);
-}
+/*
+ * We need to filter the obsoletes dir list, to only remove directories
that were made obsolete by this compaction
+ * If we have a higher retentionTime it is possible for a second
compaction to run on the same partition. Cleaning up the first compaction
+ * should not touch the newer obsolete directories to not to violate the
retentionTime for those.
+ */
+validWriteIdList = validWriteIdList.updateHighWatermark(ci.highestWriteId);
Review comment:
And we're 100% sure that we're lowering it and not raising it? Maybe we
could include some sort of assertion that ci.highestWriteId <= previous high
watermark?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 627596)
Time Spent: 3h (was: 2h 50m)
> Compaction queue entries may accumulate in "ready for cleaning" state
> -
>
> Key: HIVE-25115
> URL: https://issues.apache.org/jira/browse/HIVE-25115
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> If the Cleaner does not delete any files, the compaction queue entry is
> thrown back to the queue and remains in "ready for cleaning" state.
> Problem: If 2 compactions run on the same table and enter "ready for
> cleaning" state at the same time, only one "cleaning" will remove obsolete
> files, the other entry will remain in the queue in "ready for cleaning" state.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25249) Fix TestWorker
[ https://issues.apache.org/jira/browse/HIVE-25249?focusedWorklogId=627594=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627594 ] ASF GitHub Bot logged work on HIVE-25249: - Author: ASF GitHub Bot Created on: 26/Jul/21 09:13 Start Date: 26/Jul/21 09:13 Worklog Time Spent: 10m Work Description: klcopp commented on pull request #2474: URL: https://github.com/apache/hive/pull/2474#issuecomment-886526177 > reverted visibility ids check, there was another issue What is MetastoreConf.ConfVars.TXN_OPENTXN_TIMEOUT=2 doing now? Or what was the issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 627594) Time Spent: 1h 10m (was: 1h) > Fix TestWorker > -- > > Key: HIVE-25249 > URL: https://issues.apache.org/jira/browse/HIVE-25249 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > http://ci.hive.apache.org/job/hive-precommit/job/PR-2381/1/ > http://ci.hive.apache.org/job/hive-flaky-check/236/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25374) Fix TestReplicationOnHDFSEncryptedZones
[
https://issues.apache.org/jira/browse/HIVE-25374?focusedWorklogId=627590=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627590
]
ASF GitHub Bot logged work on HIVE-25374:
-
Author: ASF GitHub Bot
Created on: 26/Jul/21 08:53
Start Date: 26/Jul/21 08:53
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #2519:
URL: https://github.com/apache/hive/pull/2519#discussion_r676273599
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationOnHDFSEncryptedZones.java
##
@@ -113,46 +129,96 @@ public void
targetAndSourceHaveDifferentEncryptionZoneKeys() throws Throwable {
"'" + HiveConf.ConfVars.HIVE_DISTCP_DOAS_USER.varname + "'='"
+ UserGroupInformation.getCurrentUser().getUserName()
+"'");
WarehouseInstance.Tuple tuple =
-primary.run("use " + primaryDbName)
-.run("create table encrypted_table (id int, value string)")
-.run("insert into table encrypted_table values (1,'value1')")
-.run("insert into table encrypted_table values (2,'value2')")
+primary.run("use " + primaryDbName)
+.run("create table encrypted_table (id int, value string)")
+.run("insert into table encrypted_table values
(1,'value1')")
+.run("insert into table encrypted_table values
(2,'value2')")
+.dump(primaryDbName, dumpWithClause);
+
+replica
+.run("repl load " + primaryDbName + " into " + replicatedDbName
++ " with('hive.repl.add.raw.reserved.namespace'='true', "
++ "'hive.repl.replica.external.table.base.dir'='" +
replica.externalTableWarehouseRoot + "', "
++ "'hive.exec.copyfile.maxsize'='0',
'distcp.options.skipcrccheck'='')")
+.run("use " + replicatedDbName)
+.run("repl status " + replicatedDbName)
+.verifyResult(tuple.lastReplicationId);
+
+try {
+ replica
+ .run("select value from encrypted_table")
+ .verifyResults(new String[] { "value1", "value2" });
+ Assert.fail("Src EZKey shouldn't be present on target");
+} catch (IOException e) {
+ Assert.assertTrue(e.getCause().getMessage().contains("KeyVersion name
'test_key@0' does not exist"));
+}
+
+//read should pass without raw-byte distcp
+dumpWithClause = Arrays.asList( "'" +
HiveConf.ConfVars.REPL_EXTERNAL_TABLE_BASE_DIR.varname + "'='"
++ replica.externalTableWarehouseRoot + "'");
+tuple = primary.run("use " + primaryDbName)
+.run("create external table encrypted_table2 (id int, value
string)")
+.run("insert into table encrypted_table2 values (1,'value1')")
+.run("insert into table encrypted_table2 values (2,'value2')")
.dump(primaryDbName, dumpWithClause);
replica
-.run("repl load " + primaryDbName + " into " + replicatedDbName
-+ " with('hive.repl.add.raw.reserved.namespace'='true', "
-+ "'hive.repl.replica.external.table.base.dir'='" +
replica.externalTableWarehouseRoot + "', "
-+ "'distcp.options.pugpbx'='',
'distcp.options.skipcrccheck'='')")
-.run("use " + replicatedDbName)
-.run("repl status " + replicatedDbName)
-.verifyResult(tuple.lastReplicationId)
-.run("select value from encrypted_table")
-.verifyFailure(new String[] { "value1", "value2" });
+.run("repl load " + primaryDbName + " into " + replicatedDbName
++ " with('hive.repl.replica.external.table.base.dir'='" +
replica.externalTableWarehouseRoot + "', "
++ "'hive.exec.copyfile.maxsize'='0',
'distcp.options.skipcrccheck'='')")
+.run("use " + replicatedDbName)
+.run("repl status " + replicatedDbName)
+.verifyResult(tuple.lastReplicationId)
+.run("select value from encrypted_table2")
+.verifyResults(new String[] { "value1", "value2" });
}
@Ignore("this is ignored as minidfs cluster as of writing this test looked
like did not copy the "
+ "files correctly")
@Test
public void targetAndSourceHaveSameEncryptionZoneKeys() throws Throwable {
-WarehouseInstance replica = new WarehouseInstance(LOG, miniDFSCluster,
+String replicaBaseDir =
Files.createTempDirectory("replica2").toFile().getAbsolutePath();
Review comment:
The test is still disabled. Need to remove Ignore
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationOnHDFSEncryptedZones.java
##
@@ -113,46 +129,96 @@ public void
targetAndSourceHaveDifferentEncryptionZoneKeys() throws Throwable {
"'" +
[jira] [Updated] (HIVE-25377) Creating Iceberg table where some columns has comments but the last one does not fails
[
https://issues.apache.org/jira/browse/HIVE-25377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-25377:
--
Description:
Currently this create command fails:
{code}
CREATE TABLE srcpart_iceberg (key STRING STRING COMMENT 'This is the key',
value STRING) STORED BY iceberg
{code}
The exception is this:
{code}
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling
statement: FAILED: Execution Error, return code 4 from
org.apache.hadoop.hive.ql.ddl.DDLTask. java.lang.IndexOutOfBoundsException:
Index: 2, Size: 2
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:376)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:246)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:270)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:286)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:544)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:512)
at
org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:270)
at
org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:142)
... 42 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1276)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1281)
at
org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.createTableNonReplaceMode(CreateTableOperation.java:139)
at
org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:97)
at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:84)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:361)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:334)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:245)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:108)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:348)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:204)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:153)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:148)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:164)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:233)
... 48 more
Caused by: java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
at java.util.ArrayList.rangeCheck(ArrayList.java:657)
at java.util.ArrayList.get(ArrayList.java:433)
at
org.apache.iceberg.hive.HiveSchemaConverter.convertInternal(HiveSchemaConverter.java:67)
at
org.apache.iceberg.hive.HiveSchemaConverter.convert(HiveSchemaConverter.java:55)
at
org.apache.iceberg.hive.HiveSchemaUtil.convert(HiveSchemaUtil.java:121)
at
org.apache.iceberg.mr.hive.HiveIcebergSerDe.hiveSchemaOrThrow(HiveIcebergSerDe.java:270)
at
org.apache.iceberg.mr.hive.HiveIcebergSerDe.initialize(HiveIcebergSerDe.java:116)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:323)
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:305)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1234)
... 64 more
{code}
was:
Currently this create command fails:
{code}
CREATE TABLE srcpart_iceberg (key STRING, value STRING) STORED BY iceberg
{code}
The exception is this:
{code}
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling
statement: FAILED: Execution Error, return code 4 from
org.apache.hadoop.hive.ql.ddl.DDLTask. java.lang.IndexOutOfBoundsException:
Index: 2, Size: 2
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:376)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:246)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:270)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:286)
at
[jira] [Updated] (HIVE-25377) Creating Iceberg table where some columns has comments but the last one does not fails
[
https://issues.apache.org/jira/browse/HIVE-25377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-25377:
--
Description:
Currently this create command fails:
{code}
CREATE TABLE srcpart_iceberg (key STRING COMMENT 'This is the key', value
STRING) STORED BY iceberg
{code}
The exception is this:
{code}
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling
statement: FAILED: Execution Error, return code 4 from
org.apache.hadoop.hive.ql.ddl.DDLTask. java.lang.IndexOutOfBoundsException:
Index: 2, Size: 2
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:376)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:246)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:270)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:286)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:544)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:512)
at
org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:270)
at
org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:142)
... 42 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1276)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1281)
at
org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.createTableNonReplaceMode(CreateTableOperation.java:139)
at
org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:97)
at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:84)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:361)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:334)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:245)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:108)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:348)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:204)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:153)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:148)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:164)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:233)
... 48 more
Caused by: java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
at java.util.ArrayList.rangeCheck(ArrayList.java:657)
at java.util.ArrayList.get(ArrayList.java:433)
at
org.apache.iceberg.hive.HiveSchemaConverter.convertInternal(HiveSchemaConverter.java:67)
at
org.apache.iceberg.hive.HiveSchemaConverter.convert(HiveSchemaConverter.java:55)
at
org.apache.iceberg.hive.HiveSchemaUtil.convert(HiveSchemaUtil.java:121)
at
org.apache.iceberg.mr.hive.HiveIcebergSerDe.hiveSchemaOrThrow(HiveIcebergSerDe.java:270)
at
org.apache.iceberg.mr.hive.HiveIcebergSerDe.initialize(HiveIcebergSerDe.java:116)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:323)
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:305)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1234)
... 64 more
{code}
was:
Currently this create command fails:
{code}
CREATE TABLE srcpart_iceberg (key STRING STRING COMMENT 'This is the key',
value STRING) STORED BY iceberg
{code}
The exception is this:
{code}
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling
statement: FAILED: Execution Error, return code 4 from
org.apache.hadoop.hive.ql.ddl.DDLTask. java.lang.IndexOutOfBoundsException:
Index: 2, Size: 2
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:376)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:246)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:270)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:286)
at
[jira] [Updated] (HIVE-25385) Prevent Hive Server 2 process failures when InterruptedException encountered (Part 1)
[ https://issues.apache.org/jira/browse/HIVE-25385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-25385: Summary: Prevent Hive Server 2 process failures when InterruptedException encountered (Part 1) (was: Prevent Hive Server 2 process failures when InterruptedException encountered) > Prevent Hive Server 2 process failures when InterruptedException encountered > (Part 1) > - > > Key: HIVE-25385 > URL: https://issues.apache.org/jira/browse/HIVE-25385 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Major > Labels: pull-request-available > Attachments: hive-interrupted-part1-01-3_1.patch, > hive-interrupted-part1-01-master.patch > > Time Spent: 10m > Remaining Estimate: 0h > > To prevent Hive Server 2 process failure, wrap InterruptedException with > another Exception like MetaException, HiveSQLException, etc. Otherwise, > InterruptedException rises to Thread.run and kills the process. > Example of problem stack trace: > java.lang.reflect.UndeclaredThrowableExceptionjava.lang.reflect.UndeclaredThrowableException > at com.sun.proxy.$Proxy44.heartbeat(Unknown Source) at > sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2990) > at com.sun.proxy.$Proxy44.heartbeat(Unknown Source) at > org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.heartbeat(DbTxnManager.java:622) > at > org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.lambda$run$0(DbTxnManager.java:999) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > at > org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.run(DbTxnManager.java:998) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748)Caused by: > java.lang.InterruptedException: sleep interrupted at > java.lang.Thread.sleep(Native Method) at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:259) > ... 19 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25385) Prevent Hive Server 2 process failures when InterruptedException encountered
[ https://issues.apache.org/jira/browse/HIVE-25385?focusedWorklogId=627565=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627565 ] ASF GitHub Bot logged work on HIVE-25385: - Author: ASF GitHub Bot Created on: 26/Jul/21 07:29 Start Date: 26/Jul/21 07:29 Worklog Time Spent: 10m Work Description: mattmccline-microsoft opened a new pull request #2532: URL: https://github.com/apache/hive/pull/2532 …ception encountered ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 627565) Remaining Estimate: 0h Time Spent: 10m > Prevent Hive Server 2 process failures when InterruptedException encountered > > > Key: HIVE-25385 > URL: https://issues.apache.org/jira/browse/HIVE-25385 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Major > Attachments: hive-interrupted-part1-01-3_1.patch, > hive-interrupted-part1-01-master.patch > > Time Spent: 10m > Remaining Estimate: 0h > > To prevent Hive Server 2 process failure, wrap InterruptedException with > another Exception like MetaException, HiveSQLException, etc. Otherwise, > InterruptedException rises to Thread.run and kills the process. > Example of problem stack trace: > java.lang.reflect.UndeclaredThrowableExceptionjava.lang.reflect.UndeclaredThrowableException > at com.sun.proxy.$Proxy44.heartbeat(Unknown Source) at > sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2990) > at com.sun.proxy.$Proxy44.heartbeat(Unknown Source) at > org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.heartbeat(DbTxnManager.java:622) > at > org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.lambda$run$0(DbTxnManager.java:999) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > at > org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.run(DbTxnManager.java:998) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748)Caused by: > java.lang.InterruptedException: sleep interrupted at > java.lang.Thread.sleep(Native Method) at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:259) > ... 19 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25385) Prevent Hive Server 2 process failures when InterruptedException encountered
[ https://issues.apache.org/jira/browse/HIVE-25385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25385: -- Labels: pull-request-available (was: ) > Prevent Hive Server 2 process failures when InterruptedException encountered > > > Key: HIVE-25385 > URL: https://issues.apache.org/jira/browse/HIVE-25385 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Major > Labels: pull-request-available > Attachments: hive-interrupted-part1-01-3_1.patch, > hive-interrupted-part1-01-master.patch > > Time Spent: 10m > Remaining Estimate: 0h > > To prevent Hive Server 2 process failure, wrap InterruptedException with > another Exception like MetaException, HiveSQLException, etc. Otherwise, > InterruptedException rises to Thread.run and kills the process. > Example of problem stack trace: > java.lang.reflect.UndeclaredThrowableExceptionjava.lang.reflect.UndeclaredThrowableException > at com.sun.proxy.$Proxy44.heartbeat(Unknown Source) at > sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2990) > at com.sun.proxy.$Proxy44.heartbeat(Unknown Source) at > org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.heartbeat(DbTxnManager.java:622) > at > org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.lambda$run$0(DbTxnManager.java:999) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > at > org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.run(DbTxnManager.java:998) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748)Caused by: > java.lang.InterruptedException: sleep interrupted at > java.lang.Thread.sleep(Native Method) at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:259) > ... 19 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25385) Prevent Hive Server 2 process failures when InterruptedException encountered
[ https://issues.apache.org/jira/browse/HIVE-25385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-25385: Attachment: hive-interrupted-part1-01-master.patch > Prevent Hive Server 2 process failures when InterruptedException encountered > > > Key: HIVE-25385 > URL: https://issues.apache.org/jira/browse/HIVE-25385 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Major > Attachments: hive-interrupted-part1-01-3_1.patch, > hive-interrupted-part1-01-master.patch > > > To prevent Hive Server 2 process failure, wrap InterruptedException with > another Exception like MetaException, HiveSQLException, etc. Otherwise, > InterruptedException rises to Thread.run and kills the process. > Example of problem stack trace: > java.lang.reflect.UndeclaredThrowableExceptionjava.lang.reflect.UndeclaredThrowableException > at com.sun.proxy.$Proxy44.heartbeat(Unknown Source) at > sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2990) > at com.sun.proxy.$Proxy44.heartbeat(Unknown Source) at > org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.heartbeat(DbTxnManager.java:622) > at > org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.lambda$run$0(DbTxnManager.java:999) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > at > org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.run(DbTxnManager.java:998) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748)Caused by: > java.lang.InterruptedException: sleep interrupted at > java.lang.Thread.sleep(Native Method) at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:259) > ... 19 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25386) hive-storage-api should not have guava compile dependency
[
https://issues.apache.org/jira/browse/HIVE-25386?focusedWorklogId=627559=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627559
]
ASF GitHub Bot logged work on HIVE-25386:
-
Author: ASF GitHub Bot
Created on: 26/Jul/21 07:05
Start Date: 26/Jul/21 07:05
Worklog Time Spent: 10m
Work Description: dongjoon-hyun commented on a change in pull request
#2531:
URL: https://github.com/apache/hive/pull/2531#discussion_r676341646
##
File path: storage-api/pom.xml
##
@@ -118,6 +118,7 @@
com.google.guava
guava
${guava.version}
+ test
Review comment:
This was removed in the following commit without the reason of removal.
I hope this recovers the dependency without breaking CIs.
-
https://github.com/apache/hive/pull/1784/files#diff-e5e45b8481f3d87f4fa561d45b22fa20bb5084e5240d825717e6bde511b733baL122
If some CI fails, it means the regression already spread to the other
modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 627559)
Time Spent: 50m (was: 40m)
> hive-storage-api should not have guava compile dependency
> -
>
> Key: HIVE-25386
> URL: https://issues.apache.org/jira/browse/HIVE-25386
> Project: Hive
> Issue Type: Bug
> Components: storage-api
>Affects Versions: 4.0.0
>Reporter: Dongjoon Hyun
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> https://mvnrepository.com/artifact/org.apache.hive/hive-storage-api/2.8.0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25386) hive-storage-api should not have guava compile dependency
[
https://issues.apache.org/jira/browse/HIVE-25386?focusedWorklogId=627557=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627557
]
ASF GitHub Bot logged work on HIVE-25386:
-
Author: ASF GitHub Bot
Created on: 26/Jul/21 07:04
Start Date: 26/Jul/21 07:04
Worklog Time Spent: 10m
Work Description: dongjoon-hyun commented on a change in pull request
#2531:
URL: https://github.com/apache/hive/pull/2531#discussion_r676341646
##
File path: storage-api/pom.xml
##
@@ -118,6 +118,7 @@
com.google.guava
guava
${guava.version}
+ test
Review comment:
This was removed here without the reason of removal. I hope this
recovers the dependency without breaking CIs.
-
https://github.com/apache/hive/pull/1784/files#diff-e5e45b8481f3d87f4fa561d45b22fa20bb5084e5240d825717e6bde511b733baL122
If some CI fails, it means the regression already spread to the other
modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 627557)
Time Spent: 40m (was: 0.5h)
> hive-storage-api should not have guava compile dependency
> -
>
> Key: HIVE-25386
> URL: https://issues.apache.org/jira/browse/HIVE-25386
> Project: Hive
> Issue Type: Bug
> Components: storage-api
>Affects Versions: 4.0.0
>Reporter: Dongjoon Hyun
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> https://mvnrepository.com/artifact/org.apache.hive/hive-storage-api/2.8.0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25386) hive-storage-api should not have guava compile dependency
[ https://issues.apache.org/jira/browse/HIVE-25386?focusedWorklogId=627556=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627556 ] ASF GitHub Bot logged work on HIVE-25386: - Author: ASF GitHub Bot Created on: 26/Jul/21 07:02 Start Date: 26/Jul/21 07:02 Worklog Time Spent: 10m Work Description: dongjoon-hyun edited a comment on pull request #2531: URL: https://github.com/apache/hive/pull/2531#issuecomment-886431327 cc @belugabehr and @miklosgergely from https://github.com/apache/hive/pull/1784 which introduced this regression. cc @pgaref , @omalley , @sunchao , @pavibhai from Apache Hive `hive-storage-api` 2.8.0 release dev mailing thread. cc @kbendick from Apache ORC PR, https://github.com/apache/orc/pull/773 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 627556) Time Spent: 0.5h (was: 20m) > hive-storage-api should not have guava compile dependency > - > > Key: HIVE-25386 > URL: https://issues.apache.org/jira/browse/HIVE-25386 > Project: Hive > Issue Type: Bug > Components: storage-api >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Blocker > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > https://mvnrepository.com/artifact/org.apache.hive/hive-storage-api/2.8.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25386) hive-storage-api should not have guava compile dependency
[ https://issues.apache.org/jira/browse/HIVE-25386?focusedWorklogId=627555=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627555 ] ASF GitHub Bot logged work on HIVE-25386: - Author: ASF GitHub Bot Created on: 26/Jul/21 07:00 Start Date: 26/Jul/21 07:00 Worklog Time Spent: 10m Work Description: dongjoon-hyun commented on pull request #2531: URL: https://github.com/apache/hive/pull/2531#issuecomment-886431327 cc @belugabehr and @miklosgergely from https://github.com/apache/hive/pull/1784 which introduced this regression. cc @pgaref , @omalley , @sunchao , @pavibhai from Apache Hive `hive-storage-api` 2.8.0 release thread. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 627555) Time Spent: 20m (was: 10m) > hive-storage-api should not have guava compile dependency > - > > Key: HIVE-25386 > URL: https://issues.apache.org/jira/browse/HIVE-25386 > Project: Hive > Issue Type: Bug > Components: storage-api >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Blocker > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > https://mvnrepository.com/artifact/org.apache.hive/hive-storage-api/2.8.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24542) Prepare Guava for Upgrades
[ https://issues.apache.org/jira/browse/HIVE-24542?focusedWorklogId=627554=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627554 ] ASF GitHub Bot logged work on HIVE-24542: - Author: ASF GitHub Bot Created on: 26/Jul/21 06:57 Start Date: 26/Jul/21 06:57 Worklog Time Spent: 10m Work Description: dongjoon-hyun commented on pull request #1784: URL: https://github.com/apache/hive/pull/1784#issuecomment-886429839 I made a PR to fix this. - https://github.com/apache/hive/pull/2531 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 627554) Time Spent: 40m (was: 0.5h) > Prepare Guava for Upgrades > -- > > Key: HIVE-24542 > URL: https://issues.apache.org/jira/browse/HIVE-24542 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Hive is currently using some Guava methods that are removed in future > versions, also, in some projects, the version of Guava being used is being > implicitly inherited from other projects even though Hive has a defined > version. Be explicit about it. > These actions will make upgrading Guava versions easier in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25385) Prevent Hive Server 2 process failures when InterruptedException encountered
[ https://issues.apache.org/jira/browse/HIVE-25385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-25385: Attachment: hive-interrupted-part1-01-3_1.patch > Prevent Hive Server 2 process failures when InterruptedException encountered > > > Key: HIVE-25385 > URL: https://issues.apache.org/jira/browse/HIVE-25385 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Major > Attachments: hive-interrupted-part1-01-3_1.patch > > > To prevent Hive Server 2 process failure, wrap InterruptedException with > another Exception like MetaException, HiveSQLException, etc. Otherwise, > InterruptedException rises to Thread.run and kills the process. > Example of problem stack trace: > java.lang.reflect.UndeclaredThrowableExceptionjava.lang.reflect.UndeclaredThrowableException > at com.sun.proxy.$Proxy44.heartbeat(Unknown Source) at > sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2990) > at com.sun.proxy.$Proxy44.heartbeat(Unknown Source) at > org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.heartbeat(DbTxnManager.java:622) > at > org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.lambda$run$0(DbTxnManager.java:999) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > at > org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.run(DbTxnManager.java:998) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748)Caused by: > java.lang.InterruptedException: sleep interrupted at > java.lang.Thread.sleep(Native Method) at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:259) > ... 19 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25386) hive-storage-api should not have guava compile dependency
[ https://issues.apache.org/jira/browse/HIVE-25386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25386: -- Labels: pull-request-available (was: ) > hive-storage-api should not have guava compile dependency > - > > Key: HIVE-25386 > URL: https://issues.apache.org/jira/browse/HIVE-25386 > Project: Hive > Issue Type: Bug > Components: storage-api >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Blocker > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > https://mvnrepository.com/artifact/org.apache.hive/hive-storage-api/2.8.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25386) hive-storage-api should not have guava compile dependency
[ https://issues.apache.org/jira/browse/HIVE-25386?focusedWorklogId=627553=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627553 ] ASF GitHub Bot logged work on HIVE-25386: - Author: ASF GitHub Bot Created on: 26/Jul/21 06:46 Start Date: 26/Jul/21 06:46 Worklog Time Spent: 10m Work Description: dongjoon-hyun opened a new pull request #2531: URL: https://github.com/apache/hive/pull/2531 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 627553) Remaining Estimate: 0h Time Spent: 10m > hive-storage-api should not have guava compile dependency > - > > Key: HIVE-25386 > URL: https://issues.apache.org/jira/browse/HIVE-25386 > Project: Hive > Issue Type: Bug > Components: storage-api >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Blocker > Time Spent: 10m > Remaining Estimate: 0h > > https://mvnrepository.com/artifact/org.apache.hive/hive-storage-api/2.8.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25386) hive-storage-api should not have guava compile dependency
[ https://issues.apache.org/jira/browse/HIVE-25386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated HIVE-25386: - Priority: Blocker (was: Major) > hive-storage-api should not have guava compile dependency > - > > Key: HIVE-25386 > URL: https://issues.apache.org/jira/browse/HIVE-25386 > Project: Hive > Issue Type: Bug > Components: storage-api >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Blocker > > https://mvnrepository.com/artifact/org.apache.hive/hive-storage-api/2.8.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24542) Prepare Guava for Upgrades
[
https://issues.apache.org/jira/browse/HIVE-24542?focusedWorklogId=627552=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627552
]
ASF GitHub Bot logged work on HIVE-24542:
-
Author: ASF GitHub Bot
Created on: 26/Jul/21 06:43
Start Date: 26/Jul/21 06:43
Worklog Time Spent: 10m
Work Description: dongjoon-hyun commented on a change in pull request
#1784:
URL: https://github.com/apache/hive/pull/1784#discussion_r676330914
##
File path: storage-api/pom.xml
##
@@ -113,14 +113,13 @@
slf4j-api
${slf4j.version}
-
-
com.google.guava
guava
${guava.version}
- test
Review comment:
Hi, All. This seems to cause a regression at `hive-storage-api` 2.8.0.
- https://mvnrepository.com/artifact/org.apache.hive/hive-storage-api/2.8.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 627552)
Time Spent: 0.5h (was: 20m)
> Prepare Guava for Upgrades
> --
>
> Key: HIVE-24542
> URL: https://issues.apache.org/jira/browse/HIVE-24542
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Hive is currently using some Guava methods that are removed in future
> versions, also, in some projects, the version of Guava being used is being
> implicitly inherited from other projects even though Hive has a defined
> version. Be explicit about it.
> These actions will make upgrading Guava versions easier in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-25381) Hive impersonation Failed when load data of managed tables set as hive
[
https://issues.apache.org/jira/browse/HIVE-25381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387091#comment-17387091
]
Brahma Reddy Battula edited comment on HIVE-25381 at 7/26/21, 6:40 AM:
---
I guess, you need to configure the owner(hive.load.data.owner) of the loaded
files to a user who create the table to mitigate this.
was (Author: brahmareddy):
I guess, you need to configure the owner of the loaded files to a user who
create the table to mitigate this.
> Hive impersonation Failed when load data of managed tables set as hive
> --
>
> Key: HIVE-25381
> URL: https://issues.apache.org/jira/browse/HIVE-25381
> Project: Hive
> Issue Type: Bug
>Reporter: Ranith Sardar
>Assignee: Ranith Sardar
>Priority: Minor
> Fix For: 3.1.0, 4.0.0
>
>
> When hive.server2.enable.doAs = True and setting hive as the default value
> for "hive.load.data.owner" property, this will cause below logic(in
> Hive.java-needToCopy{color:#24292e}({color})) to fail always as the
> framework is validating the owner of the file against the value which we set
> in the property hive.load.data.owner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-25381) Hive impersonation Failed when load data of managed tables set as hive
[
https://issues.apache.org/jira/browse/HIVE-25381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387091#comment-17387091
]
Brahma Reddy Battula commented on HIVE-25381:
-
I guess, you need to configure the owner of the loaded files to a user who
create the table to mitigate this.
> Hive impersonation Failed when load data of managed tables set as hive
> --
>
> Key: HIVE-25381
> URL: https://issues.apache.org/jira/browse/HIVE-25381
> Project: Hive
> Issue Type: Bug
>Reporter: Ranith Sardar
>Assignee: Ranith Sardar
>Priority: Minor
> Fix For: 3.1.0, 4.0.0
>
>
> When hive.server2.enable.doAs = True and setting hive as the default value
> for "hive.load.data.owner" property, this will cause below logic(in
> Hive.java-needToCopy{color:#24292e}({color})) to fail always as the
> framework is validating the owner of the file against the value which we set
> in the property hive.load.data.owner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25384) Bump ORC to 1.6.9
[ https://issues.apache.org/jira/browse/HIVE-25384?focusedWorklogId=627545=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627545 ] ASF GitHub Bot logged work on HIVE-25384: - Author: ASF GitHub Bot Created on: 26/Jul/21 06:27 Start Date: 26/Jul/21 06:27 Worklog Time Spent: 10m Work Description: dongjoon-hyun commented on pull request #2530: URL: https://github.com/apache/hive/pull/2530#issuecomment-886414909 Thank you so much, @sunchao ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 627545) Time Spent: 40m (was: 0.5h) > Bump ORC to 1.6.9 > - > > Key: HIVE-25384 > URL: https://issues.apache.org/jira/browse/HIVE-25384 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > ORC-804 affects ORC 1.6.0 ~ 1.6.8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24918) Handle failover case during Repl Dump
[ https://issues.apache.org/jira/browse/HIVE-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla updated HIVE-24918: -- Description: To handle: a) If failover start config is enabled, dump operation will perform following tasks: 1. Enable repl.failover.enabled prop for srcDb. 2. Abort all the open transactions related to db under replication. 3. Capture current Notification Id as cursor point. 4. Fetch list of all open transactions. If still there are any txns for srcDb, abort them. Keep track of open txns, aborted txns and txnsWithoutLocks. 5. Capture current notification ID as failover eventId and dump all the events till this eventId. 6. Store all the info captured in Step 2, 3, 4, 5 in __failover_metadata_ in dump dir. 7. Dump all the events till failover eventId. 8. Pass location of __failover_metadata, failover eventId_ and failover ready status in the replication_metrics. 9. Just before creation of DUMP ACK file, place failover_ready marker file in dump dir. b) Skip subsequent repl dump instances once we have the failover_ready marker file placed. c) If failover start config is disabled + dump dir contains _failover__ready marker file + repl.failover.enabled is unset for srcDb, then it should be treated as a bootstrap dump as this is the first dump operation in reverse direction. e) If failover start config is disabled + dump dir contains _failover__ready marker file + repl.failover.enabled is set for srcDb, then dump has to rollback the failover process. was: To handle: a) Whenever user wants to go ahead with failover, during the next or subsequent repl dump operation upon confirming that there are no pending open transaction events, It should create a _failover_ready marker file in the dump dir. This marker file would contain scheduled query name that has generated this dump. b) Skip next repl dump instances once we have the marker file placed. > Handle failover case during Repl Dump > - > > Key: HIVE-24918 > URL: https://issues.apache.org/jira/browse/HIVE-24918 > Project: Hive > Issue Type: New Feature >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Time Spent: 7h 50m > Remaining Estimate: 0h > > To handle: > a) If failover start config is enabled, dump operation will perform following > tasks: > 1. Enable repl.failover.enabled prop for srcDb. > 2. Abort all the open transactions related to db under replication. > 3. Capture current Notification Id as cursor point. > 4. Fetch list of all open transactions. If still there are any txns > for srcDb, abort them. Keep track of open txns, aborted txns and > txnsWithoutLocks. > 5. Capture current notification ID as failover eventId and dump all > the events till this eventId. > 6. Store all the info captured in Step 2, 3, 4, 5 in > __failover_metadata_ in dump dir. > 7. Dump all the events till failover eventId. > 8. Pass location of __failover_metadata, failover eventId_ and > failover ready status in the replication_metrics. > 9. Just before creation of DUMP ACK file, place failover_ready marker > file in dump dir. > b) Skip subsequent repl dump instances once we have the failover_ready marker > file placed. > c) If failover start config is disabled + dump dir contains _failover__ready > marker file + repl.failover.enabled is unset for srcDb, then it should be > treated as a bootstrap dump as this is the first dump operation in reverse > direction. > e) If failover start config is disabled + dump dir contains _failover__ready > marker file + repl.failover.enabled is set for srcDb, then dump has to > rollback the failover process. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25384) Bump ORC to 1.6.9
[ https://issues.apache.org/jira/browse/HIVE-25384?focusedWorklogId=627537=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-627537 ] ASF GitHub Bot logged work on HIVE-25384: - Author: ASF GitHub Bot Created on: 26/Jul/21 05:59 Start Date: 26/Jul/21 05:59 Worklog Time Spent: 10m Work Description: dongjoon-hyun commented on pull request #2530: URL: https://github.com/apache/hive/pull/2530#issuecomment-886399730 Also, cc @sunchao -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 627537) Time Spent: 0.5h (was: 20m) > Bump ORC to 1.6.9 > - > > Key: HIVE-25384 > URL: https://issues.apache.org/jira/browse/HIVE-25384 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > ORC-804 affects ORC 1.6.0 ~ 1.6.8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
