[jira] [Commented] (HIVE-25446) Wrong execption thrown if capacity<=0
[ https://issues.apache.org/jira/browse/HIVE-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504745#comment-17504745 ] Ashish Sharma commented on HIVE-25446: -- Execption is generate when code try to find nextPowerofTwo() for values greater then 1073741824 which result in -ve number. Above problem is solved as part of https://issues.apache.org/jira/browse/HIVE-25583 . I am correcting the exception to first check for <=0 then check for power of 2. > Wrong execption thrown if capacity<=0 > - > > Key: HIVE-25446 > URL: https://issues.apache.org/jira/browse/HIVE-25446 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Matt McCline >Assignee: Ashish Sharma >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Encountered this in a very large query: > Caused by: java.lang.AssertionError: Capacity must be a power of two > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.validateCapacity(VectorMapJoinFastHashTable.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.(VectorMapJoinFastHashTable.java:77) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.(VectorMapJoinFastBytesHashTable.java:132) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashMap.(VectorMapJoinFastBytesHashMap.java:166) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastStringHashMap.(VectorMapJoinFastStringHashMap.java:43) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.createHashTable(VectorMapJoinFastTableContainer.java:137) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.(VectorMapJoinFastTableContainer.java:86) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:122) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25990) Optimise multiple copies in case of CTAS in external tables for Object stores
[
https://issues.apache.org/jira/browse/HIVE-25990?focusedWorklogId=739882=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739882
]
ASF GitHub Bot logged work on HIVE-25990:
-
Author: ASF GitHub Bot
Created on: 11/Mar/22 06:06
Start Date: 11/Mar/22 06:06
Worklog Time Spent: 10m
Work Description: rbalamohan commented on a change in pull request #3058:
URL: https://github.com/apache/hive/pull/3058#discussion_r824412526
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
##
@@ -1496,7 +1499,13 @@ public static void mvFileToFinalPath(Path specPath,
Configuration hconf,
// for CTAS or Create MV statements
perfLogger.perfLogBegin("FileSinkOperator",
"moveSpecifiedFileStatus");
LOG.debug("CTAS/Create MV: Files being renamed: " +
filesKept.toString());
- moveSpecifiedFilesInParallel(hconf, fs, tmpPath, specPath,
filesKept);
+ if (conf.getTable() != null &&
conf.getTable().getTableType().equals(TableType.EXTERNAL_TABLE)) {
+// Do this optimisation only for External tables.
+createFileList(filesKept, tmpPath, specPath, fs);
+ } else {
+List filesKeptPaths = filesKept.stream().map(x ->
x.getPath().toString()).collect(Collectors.toList());
Review comment:
filesKept is a set. Is it possible to retain as "set" after the mapping?
(Something like Collectors.toSet()). if so, no need to change the signature of
moveSpecifiedFilesInParallel?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 739882)
Time Spent: 50m (was: 40m)
> Optimise multiple copies in case of CTAS in external tables for Object stores
> -
>
> Key: HIVE-25990
> URL: https://issues.apache.org/jira/browse/HIVE-25990
> Project: Hive
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Presently for CTAS with external tables, there are two renames, operations,
> one from tmp to _ext and then from _ext to actual target.
> In case of object stores, the renames lead to actual copy. Avoid renaming by
> avoiding rename from tmp to _ext, but by creating a list of files to be
> copied in that directly, which can be consumed in the move task, to copy
> directly from tmp to actual target.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25963) Temporary table creation with not null constraint gets converted to external table
[
https://issues.apache.org/jira/browse/HIVE-25963?focusedWorklogId=739736=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739736
]
ASF GitHub Bot logged work on HIVE-25963:
-
Author: ASF GitHub Bot
Created on: 10/Mar/22 21:45
Start Date: 10/Mar/22 21:45
Worklog Time Spent: 10m
Work Description: nrg4878 commented on pull request #3040:
URL: https://github.com/apache/hive/pull/3040#issuecomment-1064532858
Fix has been merged. Please close the PR and the jira as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 739736)
Time Spent: 1h 40m (was: 1.5h)
> Temporary table creation with not null constraint gets converted to external
> table
> ---
>
> Key: HIVE-25963
> URL: https://issues.apache.org/jira/browse/HIVE-25963
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Standalone Metastore
>Reporter: Sourabh Goyal
>Assignee: Sourabh Goyal
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> When creating a temporary table with not null, constraint it gets covered to
> external table. For example:
> create temporary table t2 (a int not null);
> table t2' metadata looks like:
> {code:java}
> +---+++
> | col_name| data_type
>| comment |
> +---+++
> | a | int
>||
> | | NULL
>| NULL |
> | # Detailed Table Information | NULL
>| NULL |
> | Database: | default
>| NULL |
> | OwnerType:| USER
>| NULL |
> | Owner:| sourabh
>| NULL |
> | CreateTime: | Tue Feb 15 15:20:13 PST 2022
>| NULL |
> | LastAccessTime: | UNKNOWN
>| NULL |
> | Retention:| 0
>| NULL |
> | Location: |
> hdfs://localhost:9000/tmp/hive/sourabh/80d374a8-cd7a-4fcf-ae72-51b04ff9c3d8/_tmp_space.db/4574446d-c144-48f9-b4b6-2e9ee0ce5be4
> | NULL |
> | Table Type: | EXTERNAL_TABLE
>| NULL |
> | Table Parameters: | NULL
>| NULL |
> | | COLUMN_STATS_ACCURATE
>| {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\"}} |
> | | EXTERNAL
>| TRUE |
> | | TRANSLATED_TO_EXTERNAL
>| TRUE |
> | | bucketing_version
>| 2 |
> | | external.table.purge
>| TRUE |
> | | numFiles
>| 0 |
> |
[jira] [Work logged] (HIVE-25575) Add support for JWT authentication in HTTP mode
[
https://issues.apache.org/jira/browse/HIVE-25575?focusedWorklogId=739709=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739709
]
ASF GitHub Bot logged work on HIVE-25575:
-
Author: ASF GitHub Bot
Created on: 10/Mar/22 20:38
Start Date: 10/Mar/22 20:38
Worklog Time Spent: 10m
Work Description: sourabh912 commented on a change in pull request #3006:
URL: https://github.com/apache/hive/pull/3006#discussion_r824135827
##
File path: service/src/java/org/apache/hive/service/auth/jwt/JWTValidator.java
##
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hive.service.auth.jwt;
+
+import com.nimbusds.jose.JOSEException;
+import com.nimbusds.jose.JWSHeader;
+import com.nimbusds.jose.JWSObject;
+import com.nimbusds.jose.JWSVerifier;
+import com.nimbusds.jose.crypto.factories.DefaultJWSVerifierFactory;
+import com.nimbusds.jose.jwk.AsymmetricJWK;
+import com.nimbusds.jose.jwk.JWK;
+import com.nimbusds.jwt.JWTClaimsSet;
+import com.nimbusds.jwt.SignedJWT;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.sasl.AuthenticationException;
+import java.io.IOException;
+import java.security.Key;
+import java.text.ParseException;
+import java.util.Date;
+import java.util.List;
+
+public class JWTValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(JWTValidator.class.getName());
+ private final URLBasedJWKSProvider jwksProvider;
+ private static final DefaultJWSVerifierFactory verifierFactory = new
DefaultJWSVerifierFactory();
+
+ public JWTValidator(HiveConf conf) throws IOException, ParseException {
+this.jwksProvider = new URLBasedJWKSProvider(conf);
+ }
+
+ public String validateJWTAndExtractUser(String signedJwt) throws
ParseException, AuthenticationException {
+final SignedJWT parsedJwt = SignedJWT.parse(signedJwt);
+List matchedJWKS = jwksProvider.getJWKs(parsedJwt.getHeader());
+
+// verify signature
+for (JWK matchedJWK : matchedJWKS) {
+ try {
+JWSVerifier verifier = getVerifier(parsedJwt.getHeader(), matchedJWK);
+if (parsedJwt.verify(verifier)) {
+ break;
+}
+ } catch (JOSEException e) {
+LOG.info("Failed to verify JWT {} by JWK {} because {}",
parsedJwt.getHeader(), matchedJWK.getKeyID(),
+e.getMessage());
+ }
+}
+if (parsedJwt.getState() != JWSObject.State.VERIFIED) {
+ throw new AuthenticationException("Failed to verify JWT signature");
+}
+
+// verify claims
+JWTClaimsSet claimsSet = parsedJwt.getJWTClaimsSet();
+Date expirationTime = claimsSet.getExpirationTime();
+if (expirationTime != null) {
+ Date now = new Date();
+ if (now.after(expirationTime)) {
+throw new AuthenticationException("JWT has been expired");
+ }
+}
+
+// We assume the subject of claims is the query user
+return claimsSet.getSubject();
+ }
+
+ private static JWSVerifier getVerifier(JWSHeader header, JWK jwk) throws
JOSEException {
+Key key = null;
+if (jwk instanceof AsymmetricJWK) {
Review comment:
I think the application (in this case HS2) should not enforce that
**only** asymmetric algorithm is allowed though we can say that asymmetric is
preferable. I don't see a very valid reason of supporting only asymmetric algo
for token verification unless I am missing something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 739709)
Time Spent: 3h 20m (was: 3h 10m)
> Add support for JWT authentication in HTTP mode
> ---
>
> Key: HIVE-25575
> URL: https://issues.apache.org/jira/browse/HIVE-25575
>
[jira] [Work logged] (HIVE-26015) HBase table with Ranger authentication fails; needs URLEncoding
[
https://issues.apache.org/jira/browse/HIVE-26015?focusedWorklogId=739692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739692
]
ASF GitHub Bot logged work on HIVE-26015:
-
Author: ASF GitHub Bot
Created on: 10/Mar/22 19:44
Start Date: 10/Mar/22 19:44
Worklog Time Spent: 10m
Work Description: scarlin-cloudera commented on a change in pull request
#3084:
URL: https://github.com/apache/hive/pull/3084#discussion_r824095006
##
File path:
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
##
@@ -296,7 +297,8 @@ public URI getURIForAuth(Table table) throws
URISyntaxException {
String hbase_host = tableProperties.containsKey(HBASE_HOST_NAME)?
tableProperties.get(HBASE_HOST_NAME) : hbaseConf.get(HBASE_HOST_NAME);
String hbase_port = tableProperties.containsKey(HBASE_CLIENT_PORT)?
tableProperties.get(HBASE_CLIENT_PORT) : hbaseConf.get(HBASE_CLIENT_PORT);
String table_name =
tableProperties.getOrDefault(HBaseSerDe.HBASE_TABLE_NAME, null);
-String column_family =
tableProperties.getOrDefault(HBaseSerDe.HBASE_COLUMNS_MAPPING, null);
+String column_family =
+
URLEncoder.encode(tableProperties.getOrDefault(HBaseSerDe.HBASE_COLUMNS_MAPPING,
null));
if (column_family != null)
return new
URI(HBASE_PREFIX+"//"+hbase_host+":"+hbase_port+"/"+table_name+"/"+column_family);
Review comment:
Changed the code so we re resolving the whole URI now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 739692)
Time Spent: 1h 10m (was: 1h)
> HBase table with Ranger authentication fails; needs URLEncoding
> ---
>
> Key: HIVE-26015
> URL: https://issues.apache.org/jira/browse/HIVE-26015
> Project: Hive
> Issue Type: New Feature
>Reporter: Steve Carlin
>Assignee: Steve Carlin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> A Create table statement is failing for HBase going through Ranger.
> The stack trace shows a problem with the getURIForAuth method.
> The table is creating someting like this:
> CREATE EXTERNAL TABLE `mytesttbl`( `field1` string COMMENT 'from
> deserializer',`field2` string COMMENT 'from deserializer',`field3` string
> COMMENT 'from deserializer',`field4` string COMMENT 'from
> deserializer',`field5` string COMMENT 'from deserializer',`field6` int
> COMMENT 'from deserializer', `field7` string COMMENT 'from deserializer',
> `field8` int COMMENT 'from deserializer') ROW FORMAT SERDE
> 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'=':key,field1,field2,field3,field4,field5#b,field6,cf:field7#b','serialization.format'='1')
> TBLPROPERTIES ( 'hbase.table.name'='mytesttbl');
> Essentially, the SERDEPROPERTIES contain hash tabs which is causing a problem
> when creating a URI
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26015) HBase table with Ranger authentication fails; needs URLEncoding
[
https://issues.apache.org/jira/browse/HIVE-26015?focusedWorklogId=739691=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739691
]
ASF GitHub Bot logged work on HIVE-26015:
-
Author: ASF GitHub Bot
Created on: 10/Mar/22 19:43
Start Date: 10/Mar/22 19:43
Worklog Time Spent: 10m
Work Description: scarlin-cloudera commented on a change in pull request
#3084:
URL: https://github.com/apache/hive/pull/3084#discussion_r824094639
##
File path:
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
##
@@ -296,7 +297,8 @@ public URI getURIForAuth(Table table) throws
URISyntaxException {
String hbase_host = tableProperties.containsKey(HBASE_HOST_NAME)?
tableProperties.get(HBASE_HOST_NAME) : hbaseConf.get(HBASE_HOST_NAME);
String hbase_port = tableProperties.containsKey(HBASE_CLIENT_PORT)?
tableProperties.get(HBASE_CLIENT_PORT) : hbaseConf.get(HBASE_CLIENT_PORT);
String table_name =
tableProperties.getOrDefault(HBaseSerDe.HBASE_TABLE_NAME, null);
Review comment:
Changed it so we're URLEncoding the whole URI now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 739691)
Time Spent: 1h (was: 50m)
> HBase table with Ranger authentication fails; needs URLEncoding
> ---
>
> Key: HIVE-26015
> URL: https://issues.apache.org/jira/browse/HIVE-26015
> Project: Hive
> Issue Type: New Feature
>Reporter: Steve Carlin
>Assignee: Steve Carlin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> A Create table statement is failing for HBase going through Ranger.
> The stack trace shows a problem with the getURIForAuth method.
> The table is creating someting like this:
> CREATE EXTERNAL TABLE `mytesttbl`( `field1` string COMMENT 'from
> deserializer',`field2` string COMMENT 'from deserializer',`field3` string
> COMMENT 'from deserializer',`field4` string COMMENT 'from
> deserializer',`field5` string COMMENT 'from deserializer',`field6` int
> COMMENT 'from deserializer', `field7` string COMMENT 'from deserializer',
> `field8` int COMMENT 'from deserializer') ROW FORMAT SERDE
> 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'=':key,field1,field2,field3,field4,field5#b,field6,cf:field7#b','serialization.format'='1')
> TBLPROPERTIES ( 'hbase.table.name'='mytesttbl');
> Essentially, the SERDEPROPERTIES contain hash tabs which is causing a problem
> when creating a URI
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-25994) Analyze table runs into ClassNotFoundException-s in case binary distribution is used
[
https://issues.apache.org/jira/browse/HIVE-25994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504525#comment-17504525
]
László Bodor commented on HIVE-25994:
-
TezTask runs in HS2 not in Tez, it's the HS2 abstraction of a tez job
RootInputInitializerManager runs usually in TezAM, as it's part of the split
generation
however, in case of a test, all of these can run in the same JVM, so in order
to find the root cause it's crucial to know the exact setup
> Analyze table runs into ClassNotFoundException-s in case binary distribution
> is used
>
>
> Key: HIVE-25994
> URL: https://issues.apache.org/jira/browse/HIVE-25994
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 4.0.0-alpha-1
>
>
> any nightly release can be used to reproduce this:
> {code}
> create table t (a integer); insert into t values (1) ; analyze table t
> compute statistics for columns;
> {code}
> results in
> {code}
> DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1,
> vertexId=vertex_164683
> 1571866_0006_2_00, diagnostics=[Vertex vertex_1646831571866_0006_2_00 [Map 1]
> killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: test
> initializer failed, vertex=vertex_1646831571866_0006_2_00 [Map 1],
> java.lang.RuntimeException: Failed to load plan: file:/tmp/dev/eebb53b4-db79
> -48b9-b78e-cd71fbe1b9d3/hive_2022-03-09_19-00-08_579_8816359375110151189-14/dev/_tez_scratch_dir/55415d69-07cf-45c3-8c57-fa607633a580/map.xml
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:535)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:366)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.prepare(HiveSplitGenerator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:164)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager.lambda$runInitializer$3(RootInputInitializerManager.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager.runInitializer(RootInputInitializerManager.java:193)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager.runInitializerAndProcessResult(RootInputInitializerManager.java:174)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager.lambda$createAndStartInitializing$2(RootInputInitializerManager.java:168)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.NoClassDefFoundError: org/antlr/runtime/tree/CommonTree
> Serialization trace:
> tableSpec (org.apache.hadoop.hive.ql.metadata.Table)
> tableMetadata (org.apache.hadoop.hive.ql.plan.TableScanDesc)
> conf (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:147)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:124)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:729)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:216)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:124)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:729)
> at
>
[jira] [Updated] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-25522:
-
Fix Version/s: 3.1.3
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.3, 4.0.0
>
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
>
[jira] [Commented] (HIVE-25522) NullPointerException in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504517#comment-17504517
]
Naveen Gangam commented on HIVE-25522:
--
[~szehon] Thanks for the patch. I have included this into the release branch.
> NullPointerException in TxnHandler
> --
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.2
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
>
[jira] [Commented] (HIVE-26025) Remove IMetaStoreClient#listPartitionNames which is not used
[
https://issues.apache.org/jira/browse/HIVE-26025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504398#comment-17504398
]
Zoltán Borók-Nagy commented on HIVE-26025:
--
Hi [~pvary],
Impala only uses
{noformat}
List listPartitionNames(String db_name, String tbl_name,
short max_parts) throws NoSuchObjectException, MetaException, TException;
{noformat}
> Remove IMetaStoreClient#listPartitionNames which is not used
>
>
> Key: HIVE-26025
> URL: https://issues.apache.org/jira/browse/HIVE-26025
> Project: Hive
> Issue Type: Task
>Reporter: Peter Vary
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the following method is not used and not yet released:
> {code:java}
> List listPartitionNames(String catName, String dbName, String tblName,
> String defaultPartName, byte[] exprBytes, String order, short maxParts)
> throws MetaException, TException, NoSuchObjectException; {code}
> We should not release unused methods
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26017) Insert with partition value containing colon and space is creating partition having wrong value
[ https://issues.apache.org/jira/browse/HIVE-26017?focusedWorklogId=739575=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739575 ] ASF GitHub Bot logged work on HIVE-26017: - Author: ASF GitHub Bot Created on: 10/Mar/22 16:18 Start Date: 10/Mar/22 16:18 Worklog Time Spent: 10m Work Description: maheshk114 merged pull request #3087: URL: https://github.com/apache/hive/pull/3087 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 739575) Time Spent: 0.5h (was: 20m) > Insert with partition value containing colon and space is creating partition > having wrong value > --- > > Key: HIVE-26017 > URL: https://issues.apache.org/jira/browse/HIVE-26017 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The path used for generating the dynamic partition value is obtained from > uri. This is causing the serialised value to be used for partition name > generation and wrong names are generated. The path value should be used, not > the URI. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25845) Support ColumnIndexes for Parq files
[ https://issues.apache.org/jira/browse/HIVE-25845?focusedWorklogId=739573=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739573 ] ASF GitHub Bot logged work on HIVE-25845: - Author: ASF GitHub Bot Created on: 10/Mar/22 16:16 Start Date: 10/Mar/22 16:16 Worklog Time Spent: 10m Work Description: rbalamohan commented on pull request #3091: URL: https://github.com/apache/hive/pull/3091#issuecomment-1064235522 All test failures are related to LLAP cache. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 739573) Time Spent: 20m (was: 10m) > Support ColumnIndexes for Parq files > > > Key: HIVE-25845 > URL: https://issues.apache.org/jira/browse/HIVE-25845 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > https://issues.apache.org/jira/browse/PARQUET-1201 > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java#L271-L273] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25838) Hive SQL using TEZ as execution engine not giving result on empty partition
[ https://issues.apache.org/jira/browse/HIVE-25838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504390#comment-17504390 ] Jose Martinez Poblete commented on HIVE-25838: -- Hive is working as expected. See HIVE-15397 > Hive SQL using TEZ as execution engine not giving result on empty partition > --- > > Key: HIVE-25838 > URL: https://issues.apache.org/jira/browse/HIVE-25838 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: dinesh >Priority: Major > > Hive SQL's on empty partitions giving no result instead of 0 rows or actual > value. For example - > --Create external Table > 1) Create external table test_tbl ( name string) partitioned by ( company > string, processdate string) stored as orc location '/my/some/random/location'; > – Add partion > 2) Alter table test_tbl add partition ( company='aquaifer', > processdate='20220101'); > > – Execute following SQL's which returns no records. > 3) select max( company ) , processdate from test_tbl group by processdate ; > 4) select max(processdate ) from test_tbl ; > > Same SQL (#3 & #4 above) , when execute with SPARK, returns '0' count and > '20220101' respectively. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26017) Insert with partition value containing colon and space is creating partition having wrong value
[ https://issues.apache.org/jira/browse/HIVE-26017?focusedWorklogId=739555=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739555 ] ASF GitHub Bot logged work on HIVE-26017: - Author: ASF GitHub Bot Created on: 10/Mar/22 15:44 Start Date: 10/Mar/22 15:44 Worklog Time Spent: 10m Work Description: ramesh0201 commented on pull request #3087: URL: https://github.com/apache/hive/pull/3087#issuecomment-1064201280 +1, Looks good to me -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 739555) Time Spent: 20m (was: 10m) > Insert with partition value containing colon and space is creating partition > having wrong value > --- > > Key: HIVE-26017 > URL: https://issues.apache.org/jira/browse/HIVE-26017 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The path used for generating the dynamic partition value is obtained from > uri. This is causing the serialised value to be used for partition name > generation and wrong names are generated. The path value should be used, not > the URI. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-26003) DROP FUNCTION silently passes when function doesn't exist
[
https://issues.apache.org/jira/browse/HIVE-26003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor reassigned HIVE-26003:
---
Assignee: (was: László Bodor)
> DROP FUNCTION silently passes when function doesn't exist
> -
>
> Key: HIVE-26003
> URL: https://issues.apache.org/jira/browse/HIVE-26003
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Priority: Major
>
> DROP FUNCTION silently passes when a function doesn't exist, which is bad,
> especially because hive has "DROP FUNCTION IF EXISTS".
> I was working with functions when I found that "DROP FUNCTION myfunc" passed,
> and I thought it simply dropped the function, but then it kept working. I
> realized I was supposed to call "DROP FUNCTION default.myfunc" because it's
> registered as "default.myfunc". This "default" usecase is just one example
> where DROP FUNCTION seems to work expected but silently causes confusion.
> {code}
> CREATE FUNCTION qtest_get_java_boolean AS
> 'org.apache.hadoop.hive.ql.udf.generic.GenericUDFTestGetJavaBoolean';
> describe function extended qtest_get_java_boolean;
> drop function if exists qtest_get_java_boolean_typo; #PASS, find
> drop function qtest_get_java_boolean_typo; #PASS, should fail I believe
> {code}
> UPDATE: okay, I've just realized there is
> hive.exec.drop.ignorenonexistent=true which causes this
> I still don't like this, why do we ignore non-existent functions if we have a
> separate "if exist" clause? at least a message should appear that the
> function is invalid but we don't throw SemanticException
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-25989) CTLT HBaseStorageHandler is dropping underlying HBase table when failed
[
https://issues.apache.org/jira/browse/HIVE-25989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504190#comment-17504190
]
Marton Bod commented on HIVE-25989:
---
Pushed to master. Thanks [~pvary] for the review and [~nareshpr] for reporting
the issue!
> CTLT HBaseStorageHandler is dropping underlying HBase table when failed
> ---
>
> Key: HIVE-25989
> URL: https://issues.apache.org/jira/browse/HIVE-25989
> Project: Hive
> Issue Type: Bug
>Reporter: Naresh P R
>Assignee: Marton Bod
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> With hive.strict.managed.tables & hive.create.as.acid,
> Hive-Hbase rollback code is assuming it is a createTable failure instead of
> CTLT & removing underlying hbase table while rolling back at here.
> [https://github.com/apache/hive/blob/master/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java#L187-L195]
>
> Repro
>
> {code:java}
> hbase
> =
> hbase shell
> create 'hbase_hive_table', 'cf'
> beeline
> ===
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> set hive.strict.managed.tables=true;
> set hive.create.as.acid=true;
> set hive.create.as.insert.only=true;
> set hive.default.fileformat.managed=ORC;
> > CREATE EXTERNAL TABLE `hbase_hive_table`(
> `key` int COMMENT '',
> `value` string COMMENT '')
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'=':key,cf:cf')
> TBLPROPERTIES ('hbase.table.name'='hbase_hive_table');
> > select * from hbase_hive_table;
> +---+-+
> | hbase_hive_table.key | hbase_hive_table.value |
> +---+-+
> +---+-+
> > create table new_hbase_hive_table like hbase_hive_table;
> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: The table must
> be stored using an ACID compliant format (such as ORC):
> default.new_hbase_hive_table
> > select * from hbase_hive_table;
> Error: java.io.IOException: org.apache.hadoop.hbase.TableNotFoundException:
> hbase_hive_table
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25989) CTLT HBaseStorageHandler is dropping underlying HBase table when failed
[
https://issues.apache.org/jira/browse/HIVE-25989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Bod resolved HIVE-25989.
---
Resolution: Fixed
> CTLT HBaseStorageHandler is dropping underlying HBase table when failed
> ---
>
> Key: HIVE-25989
> URL: https://issues.apache.org/jira/browse/HIVE-25989
> Project: Hive
> Issue Type: Bug
>Reporter: Naresh P R
>Assignee: Marton Bod
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> With hive.strict.managed.tables & hive.create.as.acid,
> Hive-Hbase rollback code is assuming it is a createTable failure instead of
> CTLT & removing underlying hbase table while rolling back at here.
> [https://github.com/apache/hive/blob/master/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java#L187-L195]
>
> Repro
>
> {code:java}
> hbase
> =
> hbase shell
> create 'hbase_hive_table', 'cf'
> beeline
> ===
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> set hive.strict.managed.tables=true;
> set hive.create.as.acid=true;
> set hive.create.as.insert.only=true;
> set hive.default.fileformat.managed=ORC;
> > CREATE EXTERNAL TABLE `hbase_hive_table`(
> `key` int COMMENT '',
> `value` string COMMENT '')
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'=':key,cf:cf')
> TBLPROPERTIES ('hbase.table.name'='hbase_hive_table');
> > select * from hbase_hive_table;
> +---+-+
> | hbase_hive_table.key | hbase_hive_table.value |
> +---+-+
> +---+-+
> > create table new_hbase_hive_table like hbase_hive_table;
> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: The table must
> be stored using an ACID compliant format (such as ORC):
> default.new_hbase_hive_table
> > select * from hbase_hive_table;
> Error: java.io.IOException: org.apache.hadoop.hbase.TableNotFoundException:
> hbase_hive_table
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25989) CTLT HBaseStorageHandler is dropping underlying HBase table when failed
[
https://issues.apache.org/jira/browse/HIVE-25989?focusedWorklogId=739384=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739384
]
ASF GitHub Bot logged work on HIVE-25989:
-
Author: ASF GitHub Bot
Created on: 10/Mar/22 11:10
Start Date: 10/Mar/22 11:10
Worklog Time Spent: 10m
Work Description: marton-bod merged pull request #3076:
URL: https://github.com/apache/hive/pull/3076
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 739384)
Time Spent: 2h 40m (was: 2.5h)
> CTLT HBaseStorageHandler is dropping underlying HBase table when failed
> ---
>
> Key: HIVE-25989
> URL: https://issues.apache.org/jira/browse/HIVE-25989
> Project: Hive
> Issue Type: Bug
>Reporter: Naresh P R
>Assignee: Marton Bod
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> With hive.strict.managed.tables & hive.create.as.acid,
> Hive-Hbase rollback code is assuming it is a createTable failure instead of
> CTLT & removing underlying hbase table while rolling back at here.
> [https://github.com/apache/hive/blob/master/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java#L187-L195]
>
> Repro
>
> {code:java}
> hbase
> =
> hbase shell
> create 'hbase_hive_table', 'cf'
> beeline
> ===
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> set hive.strict.managed.tables=true;
> set hive.create.as.acid=true;
> set hive.create.as.insert.only=true;
> set hive.default.fileformat.managed=ORC;
> > CREATE EXTERNAL TABLE `hbase_hive_table`(
> `key` int COMMENT '',
> `value` string COMMENT '')
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'=':key,cf:cf')
> TBLPROPERTIES ('hbase.table.name'='hbase_hive_table');
> > select * from hbase_hive_table;
> +---+-+
> | hbase_hive_table.key | hbase_hive_table.value |
> +---+-+
> +---+-+
> > create table new_hbase_hive_table like hbase_hive_table;
> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: The table must
> be stored using an ACID compliant format (such as ORC):
> default.new_hbase_hive_table
> > select * from hbase_hive_table;
> Error: java.io.IOException: org.apache.hadoop.hbase.TableNotFoundException:
> hbase_hive_table
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25943) Introduce compaction cleaner failed attempts threshold
[ https://issues.apache.org/jira/browse/HIVE-25943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25943. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thanks for your contribution [~veghlaci05]! > Introduce compaction cleaner failed attempts threshold > -- > > Key: HIVE-25943 > URL: https://issues.apache.org/jira/browse/HIVE-25943 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 6h > Remaining Estimate: 0h > > If the cleaner fails for some reason, the compaction entity status remains in > "ready for cleaning", therefore the cleaner will pick up this entity > resulting in an endless try. The number of failed cleaning attempts should be > counted and if they reach a certain threshold the cleaner must skip all the > cleaning attempts on that compaction entity. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25943) Introduce compaction cleaner failed attempts threshold
[ https://issues.apache.org/jira/browse/HIVE-25943?focusedWorklogId=739381=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739381 ] ASF GitHub Bot logged work on HIVE-25943: - Author: ASF GitHub Bot Created on: 10/Mar/22 11:06 Start Date: 10/Mar/22 11:06 Worklog Time Spent: 10m Work Description: klcopp merged pull request #3034: URL: https://github.com/apache/hive/pull/3034 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 739381) Time Spent: 6h (was: 5h 50m) > Introduce compaction cleaner failed attempts threshold > -- > > Key: HIVE-25943 > URL: https://issues.apache.org/jira/browse/HIVE-25943 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 6h > Remaining Estimate: 0h > > If the cleaner fails for some reason, the compaction entity status remains in > "ready for cleaning", therefore the cleaner will pick up this entity > resulting in an endless try. The number of failed cleaning attempts should be > counted and if they reach a certain threshold the cleaner must skip all the > cleaning attempts on that compaction entity. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25645) Query-based compaction doesn't work when partition column type is boolean
[ https://issues.apache.org/jira/browse/HIVE-25645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25645. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thanks for your contribution [~veghlaci05]! > Query-based compaction doesn't work when partition column type is boolean > - > > Key: HIVE-25645 > URL: https://issues.apache.org/jira/browse/HIVE-25645 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25645) Query-based compaction doesn't work when partition column type is boolean
[ https://issues.apache.org/jira/browse/HIVE-25645?focusedWorklogId=739377=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739377 ] ASF GitHub Bot logged work on HIVE-25645: - Author: ASF GitHub Bot Created on: 10/Mar/22 11:03 Start Date: 10/Mar/22 11:03 Worklog Time Spent: 10m Work Description: klcopp merged pull request #3079: URL: https://github.com/apache/hive/pull/3079 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 739377) Time Spent: 1h 20m (was: 1h 10m) > Query-based compaction doesn't work when partition column type is boolean > - > > Key: HIVE-25645 > URL: https://issues.apache.org/jira/browse/HIVE-25645 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-26021) Change integration tests under DBInstallBase to regular unit tests
[
https://issues.apache.org/jira/browse/HIVE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504186#comment-17504186
]
Peter Vary commented on HIVE-26021:
---
Good to know!
Well done and thanks [~zabetak]!
> Change integration tests under DBInstallBase to regular unit tests
> --
>
> Key: HIVE-26021
> URL: https://issues.apache.org/jira/browse/HIVE-26021
> Project: Hive
> Issue Type: Improvement
> Components: Tests
>Reporter: Stamatis Zampetakis
>Priority: Major
>
> After HIVE-18588, some tests including those under
> [DBInstallBase|https://github.com/apache/hive/blob/1139c4b14db82a9e2316196819b35cfb713f34b5/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/DbInstallBase.java]
> class have been marked as integration tests mainly to keep the test duration
> low.
> Nowadays, Hive developers rarely run all tests locally so separating between
> integration tests and unit tests does not provide a clear benefit. The
> separation adds maintenance cost and makes their execution more difficult
> scaring people away.
> The goal of this issue is to change the tests under {{DBInstallBase}} from
> "integration" tests back to regular unit tests and run them as part of the
> standard maven test phase without any fancy arguments.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-26025) Remove IMetaStoreClient#listPartitionNames which is not used
[
https://issues.apache.org/jira/browse/HIVE-26025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary reassigned HIVE-26025:
-
Assignee: Zhihua Deng
> Remove IMetaStoreClient#listPartitionNames which is not used
>
>
> Key: HIVE-26025
> URL: https://issues.apache.org/jira/browse/HIVE-26025
> Project: Hive
> Issue Type: Task
>Reporter: Peter Vary
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the following method is not used and not yet released:
> {code:java}
> List listPartitionNames(String catName, String dbName, String tblName,
> String defaultPartName, byte[] exprBytes, String order, short maxParts)
> throws MetaException, TException, NoSuchObjectException; {code}
> We should not release unused methods
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-26025) Remove IMetaStoreClient#listPartitionNames which is not used
[
https://issues.apache.org/jira/browse/HIVE-26025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504185#comment-17504185
]
Peter Vary commented on HIVE-26025:
---
[~boroknagyz]: Is this method used by Impala?
> Remove IMetaStoreClient#listPartitionNames which is not used
>
>
> Key: HIVE-26025
> URL: https://issues.apache.org/jira/browse/HIVE-26025
> Project: Hive
> Issue Type: Task
>Reporter: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the following method is not used and not yet released:
> {code:java}
> List listPartitionNames(String catName, String dbName, String tblName,
> String defaultPartName, byte[] exprBytes, String order, short maxParts)
> throws MetaException, TException, NoSuchObjectException; {code}
> We should not release unused methods
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-26025) Remove IMetaStoreClient#listPartitionNames which is not used
[
https://issues.apache.org/jira/browse/HIVE-26025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504183#comment-17504183
]
Zhihua Deng commented on HIVE-26025:
I opened a pr for this: https://github.com/apache/hive/pull/3093
> Remove IMetaStoreClient#listPartitionNames which is not used
>
>
> Key: HIVE-26025
> URL: https://issues.apache.org/jira/browse/HIVE-26025
> Project: Hive
> Issue Type: Task
>Reporter: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the following method is not used and not yet released:
> {code:java}
> List listPartitionNames(String catName, String dbName, String tblName,
> String defaultPartName, byte[] exprBytes, String order, short maxParts)
> throws MetaException, TException, NoSuchObjectException; {code}
> We should not release unused methods
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-26025) Remove IMetaStoreClient#listPartitionNames which is not used
[
https://issues.apache.org/jira/browse/HIVE-26025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26025:
--
Labels: pull-request-available (was: )
> Remove IMetaStoreClient#listPartitionNames which is not used
>
>
> Key: HIVE-26025
> URL: https://issues.apache.org/jira/browse/HIVE-26025
> Project: Hive
> Issue Type: Task
>Reporter: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the following method is not used and not yet released:
> {code:java}
> List listPartitionNames(String catName, String dbName, String tblName,
> String defaultPartName, byte[] exprBytes, String order, short maxParts)
> throws MetaException, TException, NoSuchObjectException; {code}
> We should not release unused methods
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26025) Remove IMetaStoreClient#listPartitionNames which is not used
[
https://issues.apache.org/jira/browse/HIVE-26025?focusedWorklogId=739368=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739368
]
ASF GitHub Bot logged work on HIVE-26025:
-
Author: ASF GitHub Bot
Created on: 10/Mar/22 10:56
Start Date: 10/Mar/22 10:56
Worklog Time Spent: 10m
Work Description: dengzhhu653 opened a new pull request #3093:
URL: https://github.com/apache/hive/pull/3093
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 739368)
Remaining Estimate: 0h
Time Spent: 10m
> Remove IMetaStoreClient#listPartitionNames which is not used
>
>
> Key: HIVE-26025
> URL: https://issues.apache.org/jira/browse/HIVE-26025
> Project: Hive
> Issue Type: Task
>Reporter: Peter Vary
>Priority: Major
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the following method is not used and not yet released:
> {code:java}
> List listPartitionNames(String catName, String dbName, String tblName,
> String defaultPartName, byte[] exprBytes, String order, short maxParts)
> throws MetaException, TException, NoSuchObjectException; {code}
> We should not release unused methods
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-26021) Change integration tests under DBInstallBase to regular unit tests
[
https://issues.apache.org/jira/browse/HIVE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504179#comment-17504179
]
Stamatis Zampetakis commented on HIVE-26021:
[~pvary] actually we were already running the following in CI:
* ITestMysql
* ITestPostgres
* ITestDerby
and after merging HIVE-26022 we are also running ITestOracle.
The main inconvenience is that we need special arguments in order to run them
locally and special instructions in the Jenkinsfile.
> Change integration tests under DBInstallBase to regular unit tests
> --
>
> Key: HIVE-26021
> URL: https://issues.apache.org/jira/browse/HIVE-26021
> Project: Hive
> Issue Type: Improvement
> Components: Tests
>Reporter: Stamatis Zampetakis
>Priority: Major
>
> After HIVE-18588, some tests including those under
> [DBInstallBase|https://github.com/apache/hive/blob/1139c4b14db82a9e2316196819b35cfb713f34b5/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/DbInstallBase.java]
> class have been marked as integration tests mainly to keep the test duration
> low.
> Nowadays, Hive developers rarely run all tests locally so separating between
> integration tests and unit tests does not provide a clear benefit. The
> separation adds maintenance cost and makes their execution more difficult
> scaring people away.
> The goal of this issue is to change the tests under {{DBInstallBase}} from
> "integration" tests back to regular unit tests and run them as part of the
> standard maven test phase without any fancy arguments.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (HIVE-26021) Change integration tests under DBInstallBase to regular unit tests
[
https://issues.apache.org/jira/browse/HIVE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504179#comment-17504179
]
Stamatis Zampetakis edited comment on HIVE-26021 at 3/10/22, 10:48 AM:
---
[~pvary] actually we were already running the following in CI:
* ITestMysql
* ITestPostgres
* ITestDerby
and after merging HIVE-26022 we are also running ITestOracle.
The main inconvenience is that we need special arguments in order to run them
locally and special instructions in the Jenkinsfile.
was (Author: zabetak):
[~pvary] actually we were already running the following in CI:
* ITestMysql
* ITestPostgres
* ITestDerby
and after merging HIVE-26022 we are also running ITestOracle.
The main inconvenience is that we need special arguments in order to run them
locally and special instructions in the Jenkinsfile.
> Change integration tests under DBInstallBase to regular unit tests
> --
>
> Key: HIVE-26021
> URL: https://issues.apache.org/jira/browse/HIVE-26021
> Project: Hive
> Issue Type: Improvement
> Components: Tests
>Reporter: Stamatis Zampetakis
>Priority: Major
>
> After HIVE-18588, some tests including those under
> [DBInstallBase|https://github.com/apache/hive/blob/1139c4b14db82a9e2316196819b35cfb713f34b5/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/DbInstallBase.java]
> class have been marked as integration tests mainly to keep the test duration
> low.
> Nowadays, Hive developers rarely run all tests locally so separating between
> integration tests and unit tests does not provide a clear benefit. The
> separation adds maintenance cost and makes their execution more difficult
> scaring people away.
> The goal of this issue is to change the tests under {{DBInstallBase}} from
> "integration" tests back to regular unit tests and run them as part of the
> standard maven test phase without any fancy arguments.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25446) Wrong execption thrown if capacity<=0
[ https://issues.apache.org/jira/browse/HIVE-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25446: -- Labels: pull-request-available (was: ) > Wrong execption thrown if capacity<=0 > - > > Key: HIVE-25446 > URL: https://issues.apache.org/jira/browse/HIVE-25446 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Matt McCline >Assignee: Ashish Sharma >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Encountered this in a very large query: > Caused by: java.lang.AssertionError: Capacity must be a power of two > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.validateCapacity(VectorMapJoinFastHashTable.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.(VectorMapJoinFastHashTable.java:77) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.(VectorMapJoinFastBytesHashTable.java:132) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashMap.(VectorMapJoinFastBytesHashMap.java:166) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastStringHashMap.(VectorMapJoinFastStringHashMap.java:43) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.createHashTable(VectorMapJoinFastTableContainer.java:137) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.(VectorMapJoinFastTableContainer.java:86) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:122) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25446) Wrong execption thrown if capacity<=0
[ https://issues.apache.org/jira/browse/HIVE-25446?focusedWorklogId=739359=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739359 ] ASF GitHub Bot logged work on HIVE-25446: - Author: ASF GitHub Bot Created on: 10/Mar/22 10:43 Start Date: 10/Mar/22 10:43 Worklog Time Spent: 10m Work Description: ashish-kumar-sharma opened a new pull request #3092: URL: https://github.com/apache/hive/pull/3092 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 739359) Remaining Estimate: 0h Time Spent: 10m > Wrong execption thrown if capacity<=0 > - > > Key: HIVE-25446 > URL: https://issues.apache.org/jira/browse/HIVE-25446 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Matt McCline >Assignee: Ashish Sharma >Priority: Trivial > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Encountered this in a very large query: > Caused by: java.lang.AssertionError: Capacity must be a power of two > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.validateCapacity(VectorMapJoinFastHashTable.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.(VectorMapJoinFastHashTable.java:77) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.(VectorMapJoinFastBytesHashTable.java:132) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashMap.(VectorMapJoinFastBytesHashMap.java:166) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastStringHashMap.(VectorMapJoinFastStringHashMap.java:43) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.createHashTable(VectorMapJoinFastTableContainer.java:137) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.(VectorMapJoinFastTableContainer.java:86) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:122) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-26021) Change integration tests under DBInstallBase to regular unit tests
[
https://issues.apache.org/jira/browse/HIVE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504161#comment-17504161
]
Peter Vary commented on HIVE-26021:
---
Fixed the Oracle error:
[https://github.com/apache/hive/pull/3081/commits/ae33179be2b2d3f70f716dd2bc5254f9963cfb42#diff-7e59262e7e9224f7dcb232a8b8d29f795b9ddd36339350396c17ba4d0eb871a6R1284]
I am not sure, but I guess that the tests took 2 (3) mins each after the docker
image has been downloaded. This might incur extra costs/time for the CI, if we
do it for every run. [~kgyrtkirk] might know more about them.
If we do not want to enable them on the PreCommit then one possible solution
could be to run them on the nightly build - so we can at least be sure that the
tests are running and the scripts are ok.
Your thoughts?
> Change integration tests under DBInstallBase to regular unit tests
> --
>
> Key: HIVE-26021
> URL: https://issues.apache.org/jira/browse/HIVE-26021
> Project: Hive
> Issue Type: Improvement
> Components: Tests
>Reporter: Stamatis Zampetakis
>Priority: Major
>
> After HIVE-18588, some tests including those under
> [DBInstallBase|https://github.com/apache/hive/blob/1139c4b14db82a9e2316196819b35cfb713f34b5/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/DbInstallBase.java]
> class have been marked as integration tests mainly to keep the test duration
> low.
> Nowadays, Hive developers rarely run all tests locally so separating between
> integration tests and unit tests does not provide a clear benefit. The
> separation adds maintenance cost and makes their execution more difficult
> scaring people away.
> The goal of this issue is to change the tests under {{DBInstallBase}} from
> "integration" tests back to regular unit tests and run them as part of the
> standard maven test phase without any fancy arguments.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26019) Upgrade com.jayway.jsonpath from 2.4.0 to 2.7.0
[ https://issues.apache.org/jira/browse/HIVE-26019?focusedWorklogId=739347=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739347 ] ASF GitHub Bot logged work on HIVE-26019: - Author: ASF GitHub Bot Created on: 10/Mar/22 10:20 Start Date: 10/Mar/22 10:20 Worklog Time Spent: 10m Work Description: zabetak commented on pull request #3075: URL: https://github.com/apache/hive/pull/3075#issuecomment-1063893680 > is there a specific reason to push these changes in 1 PR? @abstractdog I will push the commits separately. I created a single PR to save some testing resources. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 739347) Time Spent: 20m (was: 10m) > Upgrade com.jayway.jsonpath from 2.4.0 to 2.7.0 > --- > > Key: HIVE-26019 > URL: https://issues.apache.org/jira/browse/HIVE-26019 > Project: Hive > Issue Type: Task > Components: CBO >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-26022) Error: ORA-00904 when initializing metastore schema in Oracle
[
https://issues.apache.org/jira/browse/HIVE-26022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis resolved HIVE-26022.
Assignee: Stamatis Zampetakis
Resolution: Fixed
Fixed in
https://github.com/apache/hive/commit/d696b34a5765fe950ebe4bfffd36b9ea914dfaab.
Thanks for the review [~abstractdog]!
> Error: ORA-00904 when initializing metastore schema in Oracle
> -
>
> Key: HIVE-26022
> URL: https://issues.apache.org/jira/browse/HIVE-26022
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The Metastore schema tool fails to create the database schema when the
> underlying backend is Oracle.
> The initialization scripts fails while creating the "REPLICATION_METRICS"
> table:
> {noformat}
> 338/362 --Create table replication metrics
> 339/362 CREATE TABLE "REPLICATION_METRICS" (
> "RM_SCHEDULED_EXECUTION_ID" number PRIMARY KEY,
> "RM_POLICY" varchar2(256) NOT NULL,
> "RM_DUMP_EXECUTION_ID" number NOT NULL,
> "RM_METADATA" varchar2(4000),
> "RM_PROGRESS" varchar2(4000),
> "RM_START_TIME" integer NOT NULL,
> "MESSAGE_FORMAT" VARCHAR(16) DEFAULT 'json-0.2',
> );
> Error: ORA-00904: : invalid identifier (state=42000,code=904)
> {noformat}
> The problem can be reproduced by running the {{ITestOracle}}.
> {noformat}
> mvn -pl standalone-metastore/metastore-server verify -DskipITests=false
> -Dit.test=ITestOracle -Dtest=nosuch
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26022) Error: ORA-00904 when initializing metastore schema in Oracle
[
https://issues.apache.org/jira/browse/HIVE-26022?focusedWorklogId=739346=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739346
]
ASF GitHub Bot logged work on HIVE-26022:
-
Author: ASF GitHub Bot
Created on: 10/Mar/22 10:16
Start Date: 10/Mar/22 10:16
Worklog Time Spent: 10m
Work Description: zabetak closed pull request #3088:
URL: https://github.com/apache/hive/pull/3088
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 739346)
Time Spent: 20m (was: 10m)
> Error: ORA-00904 when initializing metastore schema in Oracle
> -
>
> Key: HIVE-26022
> URL: https://issues.apache.org/jira/browse/HIVE-26022
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Stamatis Zampetakis
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The Metastore schema tool fails to create the database schema when the
> underlying backend is Oracle.
> The initialization scripts fails while creating the "REPLICATION_METRICS"
> table:
> {noformat}
> 338/362 --Create table replication metrics
> 339/362 CREATE TABLE "REPLICATION_METRICS" (
> "RM_SCHEDULED_EXECUTION_ID" number PRIMARY KEY,
> "RM_POLICY" varchar2(256) NOT NULL,
> "RM_DUMP_EXECUTION_ID" number NOT NULL,
> "RM_METADATA" varchar2(4000),
> "RM_PROGRESS" varchar2(4000),
> "RM_START_TIME" integer NOT NULL,
> "MESSAGE_FORMAT" VARCHAR(16) DEFAULT 'json-0.2',
> );
> Error: ORA-00904: : invalid identifier (state=42000,code=904)
> {noformat}
> The problem can be reproduced by running the {{ITestOracle}}.
> {noformat}
> mvn -pl standalone-metastore/metastore-server verify -DskipITests=false
> -Dit.test=ITestOracle -Dtest=nosuch
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-26019) Upgrade com.jayway.jsonpath from 2.4.0 to 2.7.0
[ https://issues.apache.org/jira/browse/HIVE-26019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26019: -- Labels: pull-request-available (was: ) > Upgrade com.jayway.jsonpath from 2.4.0 to 2.7.0 > --- > > Key: HIVE-26019 > URL: https://issues.apache.org/jira/browse/HIVE-26019 > Project: Hive > Issue Type: Task > Components: CBO >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25935) Cleanup IMetaStoreClient#getPartitionsByNames APIs
[
https://issues.apache.org/jira/browse/HIVE-25935?focusedWorklogId=739340=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739340
]
ASF GitHub Bot logged work on HIVE-25935:
-
Author: ASF GitHub Bot
Created on: 10/Mar/22 09:51
Start Date: 10/Mar/22 09:51
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #3072:
URL: https://github.com/apache/hive/pull/3072#issuecomment-1063865317
> The `List listPartitionNames(String catName, String dbName, String
tblName, String defaultPartName, byte[] exprBytes, String order, short
maxParts)` method was added by me as part of
[HIVE-22458](https://issues.apache.org/jira/browse/HIVE-22458). Currently in
master we use `List listPartitionNames(PartitionsByExprRequest
request)` to replace others with the same method name, so I think the method
can also be cleaned...
Created https://issues.apache.org/jira/browse/HIVE-26025
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 739340)
Time Spent: 3h 20m (was: 3h 10m)
> Cleanup IMetaStoreClient#getPartitionsByNames APIs
> --
>
> Key: HIVE-25935
> URL: https://issues.apache.org/jira/browse/HIVE-25935
> Project: Hive
> Issue Type: Task
> Components: Metastore
>Reporter: Stamatis Zampetakis
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Currently the
> [IMetastoreClient|https://github.com/apache/hive/blob/4b7a948e45fd88372fef573be321cda40d189cc7/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java]
> interface has 8 variants of the {{getPartitionsByNames}} method. Going
> quickly over the concrete implementation it appears that not all of them are
> useful/necessary so a bit of cleanup is needed.
> Below a few potential problems I observed:
> * Some of the APIs are not used anywhere in the project (neither by
> production nor by test code).
> * Some of the APIs are deprecated in some concrete implementations but not
> globally at the interface level without an explanation why.
> * Some of the implementations simply throw without doing anything.
> * Many of the APIs are partially tested or not tested at all.
> HIVE-24743, HIVE-25281 are related since they introduce/deprecate some of the
> aforementioned APIs.
> It would be good to review the aforementioned APIs and decide what needs to
> stay and what needs to go as well as complete necessary when relevant.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26019) Upgrade com.jayway.jsonpath from 2.4.0 to 2.7.0
[ https://issues.apache.org/jira/browse/HIVE-26019?focusedWorklogId=739339=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739339 ] ASF GitHub Bot logged work on HIVE-26019: - Author: ASF GitHub Bot Created on: 10/Mar/22 09:51 Start Date: 10/Mar/22 09:51 Worklog Time Spent: 10m Work Description: abstractdog commented on pull request #3075: URL: https://github.com/apache/hive/pull/3075#issuecomment-1063865001 is there a specific reason to push these changes in 1 PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 739339) Remaining Estimate: 0h Time Spent: 10m > Upgrade com.jayway.jsonpath from 2.4.0 to 2.7.0 > --- > > Key: HIVE-26019 > URL: https://issues.apache.org/jira/browse/HIVE-26019 > Project: Hive > Issue Type: Task > Components: CBO >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25994) Analyze table runs into ClassNotFoundException-s in case binary distribution is used
[
https://issues.apache.org/jira/browse/HIVE-25994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504144#comment-17504144
]
Alessandro Solimando commented on HIVE-25994:
-
I have updated the stack trace with more info, it's clear now that the issue
happens in _org.apache.hadoop.hive.ql.exec.tez.TezTask_, so the issue is most
probably happening in the tez worker.
> Analyze table runs into ClassNotFoundException-s in case binary distribution
> is used
>
>
> Key: HIVE-25994
> URL: https://issues.apache.org/jira/browse/HIVE-25994
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 4.0.0-alpha-1
>
>
> any nightly release can be used to reproduce this:
> {code}
> create table t (a integer); insert into t values (1) ; analyze table t
> compute statistics for columns;
> {code}
> results in
> {code}
> DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1,
> vertexId=vertex_164683
> 1571866_0006_2_00, diagnostics=[Vertex vertex_1646831571866_0006_2_00 [Map 1]
> killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: test
> initializer failed, vertex=vertex_1646831571866_0006_2_00 [Map 1],
> java.lang.RuntimeException: Failed to load plan: file:/tmp/dev/eebb53b4-db79
> -48b9-b78e-cd71fbe1b9d3/hive_2022-03-09_19-00-08_579_8816359375110151189-14/dev/_tez_scratch_dir/55415d69-07cf-45c3-8c57-fa607633a580/map.xml
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:535)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:366)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.prepare(HiveSplitGenerator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:164)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager.lambda$runInitializer$3(RootInputInitializerManager.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager.runInitializer(RootInputInitializerManager.java:193)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager.runInitializerAndProcessResult(RootInputInitializerManager.java:174)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager.lambda$createAndStartInitializing$2(RootInputInitializerManager.java:168)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.NoClassDefFoundError: org/antlr/runtime/tree/CommonTree
> Serialization trace:
> tableSpec (org.apache.hadoop.hive.ql.metadata.Table)
> tableMetadata (org.apache.hadoop.hive.ql.plan.TableScanDesc)
> conf (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:147)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:124)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:729)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:216)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:124)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:729)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:216)
> at
>
[jira] [Work logged] (HIVE-25935) Cleanup IMetaStoreClient#getPartitionsByNames APIs
[
https://issues.apache.org/jira/browse/HIVE-25935?focusedWorklogId=739335=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739335
]
ASF GitHub Bot logged work on HIVE-25935:
-
Author: ASF GitHub Bot
Created on: 10/Mar/22 09:32
Start Date: 10/Mar/22 09:32
Worklog Time Spent: 10m
Work Description: pvary edited a comment on pull request #3072:
URL: https://github.com/apache/hive/pull/3072#issuecomment-1063822321
> IMetaStoreClient `listPartitionNames` at line 1309 can be cleaned, it
would be great if you can do me a favor for that.
> Thanks,
> Zhihua Deng
Could you please ellaborate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 739335)
Time Spent: 3h (was: 2h 50m)
> Cleanup IMetaStoreClient#getPartitionsByNames APIs
> --
>
> Key: HIVE-25935
> URL: https://issues.apache.org/jira/browse/HIVE-25935
> Project: Hive
> Issue Type: Task
> Components: Metastore
>Reporter: Stamatis Zampetakis
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Currently the
> [IMetastoreClient|https://github.com/apache/hive/blob/4b7a948e45fd88372fef573be321cda40d189cc7/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java]
> interface has 8 variants of the {{getPartitionsByNames}} method. Going
> quickly over the concrete implementation it appears that not all of them are
> useful/necessary so a bit of cleanup is needed.
> Below a few potential problems I observed:
> * Some of the APIs are not used anywhere in the project (neither by
> production nor by test code).
> * Some of the APIs are deprecated in some concrete implementations but not
> globally at the interface level without an explanation why.
> * Some of the implementations simply throw without doing anything.
> * Many of the APIs are partially tested or not tested at all.
> HIVE-24743, HIVE-25281 are related since they introduce/deprecate some of the
> aforementioned APIs.
> It would be good to review the aforementioned APIs and decide what needs to
> stay and what needs to go as well as complete necessary when relevant.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25935) Cleanup IMetaStoreClient#getPartitionsByNames APIs
[
https://issues.apache.org/jira/browse/HIVE-25935?focusedWorklogId=739336=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739336
]
ASF GitHub Bot logged work on HIVE-25935:
-
Author: ASF GitHub Bot
Created on: 10/Mar/22 09:32
Start Date: 10/Mar/22 09:32
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on pull request #3072:
URL: https://github.com/apache/hive/pull/3072#issuecomment-1063849177
The ` List listPartitionNames(String catName, String dbName, String
tblName, String defaultPartName, byte[] exprBytes, String order, short
maxParts) ` method was added by me as part of HIVE-22458.
Currently in master we use `List
listPartitionNames(PartitionsByExprRequest request)` to replace others with the
same method name, so I think the method can also be cleaned...
Sorry about that...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 739336)
Time Spent: 3h 10m (was: 3h)
> Cleanup IMetaStoreClient#getPartitionsByNames APIs
> --
>
> Key: HIVE-25935
> URL: https://issues.apache.org/jira/browse/HIVE-25935
> Project: Hive
> Issue Type: Task
> Components: Metastore
>Reporter: Stamatis Zampetakis
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Currently the
> [IMetastoreClient|https://github.com/apache/hive/blob/4b7a948e45fd88372fef573be321cda40d189cc7/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java]
> interface has 8 variants of the {{getPartitionsByNames}} method. Going
> quickly over the concrete implementation it appears that not all of them are
> useful/necessary so a bit of cleanup is needed.
> Below a few potential problems I observed:
> * Some of the APIs are not used anywhere in the project (neither by
> production nor by test code).
> * Some of the APIs are deprecated in some concrete implementations but not
> globally at the interface level without an explanation why.
> * Some of the implementations simply throw without doing anything.
> * Many of the APIs are partially tested or not tested at all.
> HIVE-24743, HIVE-25281 are related since they introduce/deprecate some of the
> aforementioned APIs.
> It would be good to review the aforementioned APIs and decide what needs to
> stay and what needs to go as well as complete necessary when relevant.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25994) Analyze table runs into ClassNotFoundException-s in case binary distribution is used
[
https://issues.apache.org/jira/browse/HIVE-25994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated HIVE-25994:
Description:
any nightly release can be used to reproduce this:
{code}
create table t (a integer); insert into t values (1) ; analyze table t compute
statistics for columns;
{code}
results in
{code}
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1,
vertexId=vertex_164683
1571866_0006_2_00, diagnostics=[Vertex vertex_1646831571866_0006_2_00 [Map 1]
killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: test
initializer failed, vertex=vertex_1646831571866_0006_2_00 [Map 1],
java.lang.RuntimeException: Failed to load plan: file:/tmp/dev/eebb53b4-db79
-48b9-b78e-cd71fbe1b9d3/hive_2022-03-09_19-00-08_579_8816359375110151189-14/dev/_tez_scratch_dir/55415d69-07cf-45c3-8c57-fa607633a580/map.xml
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:535)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:366)
at
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.prepare(HiveSplitGenerator.java:152)
at
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:164)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager.lambda$runInitializer$3(RootInputInitializerManager.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager.runInitializer(RootInputInitializerManager.java:193)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager.runInitializerAndProcessResult(RootInputInitializerManager.java:174)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager.lambda$createAndStartInitializing$2(RootInputInitializerManager.java:168)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
java.lang.NoClassDefFoundError: org/antlr/runtime/tree/CommonTree
Serialization trace:
tableSpec (org.apache.hadoop.hive.ql.metadata.Table)
tableMetadata (org.apache.hadoop.hive.ql.plan.TableScanDesc)
conf (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:147)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:124)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:729)
at
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:216)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:125)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:124)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:729)
at
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:216)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:125)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:124)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:810)
at
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:178)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:237)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:42)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:729)
at
[jira] [Work logged] (HIVE-25935) Cleanup IMetaStoreClient#getPartitionsByNames APIs
[
https://issues.apache.org/jira/browse/HIVE-25935?focusedWorklogId=739332=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739332
]
ASF GitHub Bot logged work on HIVE-25935:
-
Author: ASF GitHub Bot
Created on: 10/Mar/22 09:26
Start Date: 10/Mar/22 09:26
Worklog Time Spent: 10m
Work Description: pvary edited a comment on pull request #3072:
URL: https://github.com/apache/hive/pull/3072#issuecomment-1063822321
The ` List listPartitionNames(String catName, String dbName, String
tblName, String defaultPartName, byte[] exprBytes, String order, short
maxParts) ` method was added by me as part of HIVE-22458.
Currently in master we use `List
listPartitionNames(PartitionsByExprRequest request)` to replace others with the
same method name, so I think the method can also be cleaned...
Sorry for that...
> IMetaStoreClient `listPartitionNames` at line 1309 can be cleaned, it
would be great if you can do me a favor for that.
> Thanks,
> Zhihua Deng
Could you please ellaborate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 739332)
Time Spent: 2h 50m (was: 2h 40m)
> Cleanup IMetaStoreClient#getPartitionsByNames APIs
> --
>
> Key: HIVE-25935
> URL: https://issues.apache.org/jira/browse/HIVE-25935
> Project: Hive
> Issue Type: Task
> Components: Metastore
>Reporter: Stamatis Zampetakis
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Currently the
> [IMetastoreClient|https://github.com/apache/hive/blob/4b7a948e45fd88372fef573be321cda40d189cc7/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java]
> interface has 8 variants of the {{getPartitionsByNames}} method. Going
> quickly over the concrete implementation it appears that not all of them are
> useful/necessary so a bit of cleanup is needed.
> Below a few potential problems I observed:
> * Some of the APIs are not used anywhere in the project (neither by
> production nor by test code).
> * Some of the APIs are deprecated in some concrete implementations but not
> globally at the interface level without an explanation why.
> * Some of the implementations simply throw without doing anything.
> * Many of the APIs are partially tested or not tested at all.
> HIVE-24743, HIVE-25281 are related since they introduce/deprecate some of the
> aforementioned APIs.
> It would be good to review the aforementioned APIs and decide what needs to
> stay and what needs to go as well as complete necessary when relevant.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25446) Wrong execption thrown if capacity<=0
[ https://issues.apache.org/jira/browse/HIVE-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Sharma updated HIVE-25446: - Summary: Wrong execption thrown if capacity<=0 (was: Wrong Execption thrown if capacity<=0) > Wrong execption thrown if capacity<=0 > - > > Key: HIVE-25446 > URL: https://issues.apache.org/jira/browse/HIVE-25446 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Matt McCline >Assignee: Ashish Sharma >Priority: Trivial > Fix For: 4.0.0 > > > Encountered this in a very large query: > Caused by: java.lang.AssertionError: Capacity must be a power of two > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.validateCapacity(VectorMapJoinFastHashTable.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.(VectorMapJoinFastHashTable.java:77) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.(VectorMapJoinFastBytesHashTable.java:132) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashMap.(VectorMapJoinFastBytesHashMap.java:166) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastStringHashMap.(VectorMapJoinFastStringHashMap.java:43) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.createHashTable(VectorMapJoinFastTableContainer.java:137) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.(VectorMapJoinFastTableContainer.java:86) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:122) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25446) Wrong Execption thrown if capacity<=0
[ https://issues.apache.org/jira/browse/HIVE-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Sharma updated HIVE-25446: - Summary: Wrong Execption thrown if capacity<=0 (was: Wrong Execption thrown if capacity <= 0) > Wrong Execption thrown if capacity<=0 > - > > Key: HIVE-25446 > URL: https://issues.apache.org/jira/browse/HIVE-25446 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Matt McCline >Assignee: Ashish Sharma >Priority: Trivial > Fix For: 4.0.0 > > > Encountered this in a very large query: > Caused by: java.lang.AssertionError: Capacity must be a power of two > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.validateCapacity(VectorMapJoinFastHashTable.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.(VectorMapJoinFastHashTable.java:77) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.(VectorMapJoinFastBytesHashTable.java:132) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashMap.(VectorMapJoinFastBytesHashMap.java:166) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastStringHashMap.(VectorMapJoinFastStringHashMap.java:43) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.createHashTable(VectorMapJoinFastTableContainer.java:137) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.(VectorMapJoinFastTableContainer.java:86) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:122) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25446) Wrong Execption thrown if capacity <= 0
[ https://issues.apache.org/jira/browse/HIVE-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Sharma updated HIVE-25446: - Summary: Wrong Execption thrown if capacity <= 0 (was: VectorMapJoinFastHashTable.validateCapacity AssertionError: Capacity must be a power of two) > Wrong Execption thrown if capacity <= 0 > --- > > Key: HIVE-25446 > URL: https://issues.apache.org/jira/browse/HIVE-25446 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Matt McCline >Assignee: Ashish Sharma >Priority: Trivial > Fix For: 4.0.0 > > > Encountered this in a very large query: > Caused by: java.lang.AssertionError: Capacity must be a power of two > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.validateCapacity(VectorMapJoinFastHashTable.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.(VectorMapJoinFastHashTable.java:77) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.(VectorMapJoinFastBytesHashTable.java:132) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashMap.(VectorMapJoinFastBytesHashMap.java:166) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastStringHashMap.(VectorMapJoinFastStringHashMap.java:43) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.createHashTable(VectorMapJoinFastTableContainer.java:137) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.(VectorMapJoinFastTableContainer.java:86) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:122) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25446) VectorMapJoinFastHashTable.validateCapacity AssertionError: Capacity must be a power of two
[ https://issues.apache.org/jira/browse/HIVE-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Sharma updated HIVE-25446: - Parent: HIVE-24037 Issue Type: Sub-task (was: Task) > VectorMapJoinFastHashTable.validateCapacity AssertionError: Capacity must be > a power of two > --- > > Key: HIVE-25446 > URL: https://issues.apache.org/jira/browse/HIVE-25446 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Matt McCline >Assignee: Ashish Sharma >Priority: Trivial > Fix For: 4.0.0 > > > Encountered this in a very large query: > Caused by: java.lang.AssertionError: Capacity must be a power of two > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.validateCapacity(VectorMapJoinFastHashTable.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.(VectorMapJoinFastHashTable.java:77) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.(VectorMapJoinFastBytesHashTable.java:132) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashMap.(VectorMapJoinFastBytesHashMap.java:166) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastStringHashMap.(VectorMapJoinFastStringHashMap.java:43) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.createHashTable(VectorMapJoinFastTableContainer.java:137) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.(VectorMapJoinFastTableContainer.java:86) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:122) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work started] (HIVE-25446) VectorMapJoinFastHashTable.validateCapacity AssertionError: Capacity must be a power of two
[ https://issues.apache.org/jira/browse/HIVE-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25446 started by Ashish Sharma. > VectorMapJoinFastHashTable.validateCapacity AssertionError: Capacity must be > a power of two > --- > > Key: HIVE-25446 > URL: https://issues.apache.org/jira/browse/HIVE-25446 > Project: Hive > Issue Type: Task >Affects Versions: 3.1.1 >Reporter: Matt McCline >Assignee: Ashish Sharma >Priority: Trivial > Fix For: 4.0.0 > > > Encountered this in a very large query: > Caused by: java.lang.AssertionError: Capacity must be a power of two > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.validateCapacity(VectorMapJoinFastHashTable.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.(VectorMapJoinFastHashTable.java:77) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.(VectorMapJoinFastBytesHashTable.java:132) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashMap.(VectorMapJoinFastBytesHashMap.java:166) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastStringHashMap.(VectorMapJoinFastStringHashMap.java:43) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.createHashTable(VectorMapJoinFastTableContainer.java:137) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.(VectorMapJoinFastTableContainer.java:86) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:122) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25935) Cleanup IMetaStoreClient#getPartitionsByNames APIs
[
https://issues.apache.org/jira/browse/HIVE-25935?focusedWorklogId=739323=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739323
]
ASF GitHub Bot logged work on HIVE-25935:
-
Author: ASF GitHub Bot
Created on: 10/Mar/22 09:04
Start Date: 10/Mar/22 09:04
Worklog Time Spent: 10m
Work Description: pvary merged pull request #3072:
URL: https://github.com/apache/hive/pull/3072
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 739323)
Time Spent: 2h 40m (was: 2.5h)
> Cleanup IMetaStoreClient#getPartitionsByNames APIs
> --
>
> Key: HIVE-25935
> URL: https://issues.apache.org/jira/browse/HIVE-25935
> Project: Hive
> Issue Type: Task
> Components: Metastore
>Reporter: Stamatis Zampetakis
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Currently the
> [IMetastoreClient|https://github.com/apache/hive/blob/4b7a948e45fd88372fef573be321cda40d189cc7/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java]
> interface has 8 variants of the {{getPartitionsByNames}} method. Going
> quickly over the concrete implementation it appears that not all of them are
> useful/necessary so a bit of cleanup is needed.
> Below a few potential problems I observed:
> * Some of the APIs are not used anywhere in the project (neither by
> production nor by test code).
> * Some of the APIs are deprecated in some concrete implementations but not
> globally at the interface level without an explanation why.
> * Some of the implementations simply throw without doing anything.
> * Many of the APIs are partially tested or not tested at all.
> HIVE-24743, HIVE-25281 are related since they introduce/deprecate some of the
> aforementioned APIs.
> It would be good to review the aforementioned APIs and decide what needs to
> stay and what needs to go as well as complete necessary when relevant.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25935) Cleanup IMetaStoreClient#getPartitionsByNames APIs
[
https://issues.apache.org/jira/browse/HIVE-25935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary resolved HIVE-25935.
---
Resolution: Fixed
Pushed to master.
Thanks for raising the issue [~zabetak] and [~dengzh] for the review!
> Cleanup IMetaStoreClient#getPartitionsByNames APIs
> --
>
> Key: HIVE-25935
> URL: https://issues.apache.org/jira/browse/HIVE-25935
> Project: Hive
> Issue Type: Task
> Components: Metastore
>Reporter: Stamatis Zampetakis
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Currently the
> [IMetastoreClient|https://github.com/apache/hive/blob/4b7a948e45fd88372fef573be321cda40d189cc7/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java]
> interface has 8 variants of the {{getPartitionsByNames}} method. Going
> quickly over the concrete implementation it appears that not all of them are
> useful/necessary so a bit of cleanup is needed.
> Below a few potential problems I observed:
> * Some of the APIs are not used anywhere in the project (neither by
> production nor by test code).
> * Some of the APIs are deprecated in some concrete implementations but not
> globally at the interface level without an explanation why.
> * Some of the implementations simply throw without doing anything.
> * Many of the APIs are partially tested or not tested at all.
> HIVE-24743, HIVE-25281 are related since they introduce/deprecate some of the
> aforementioned APIs.
> It would be good to review the aforementioned APIs and decide what needs to
> stay and what needs to go as well as complete necessary when relevant.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25935) Cleanup IMetaStoreClient#getPartitionsByNames APIs
[
https://issues.apache.org/jira/browse/HIVE-25935?focusedWorklogId=739322=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739322
]
ASF GitHub Bot logged work on HIVE-25935:
-
Author: ASF GitHub Bot
Created on: 10/Mar/22 09:03
Start Date: 10/Mar/22 09:03
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #3072:
URL: https://github.com/apache/hive/pull/3072#issuecomment-1063822321
> IMetaStoreClient `listPartitionNames` at line 1309 can be cleaned, it
would be great if you can do me a favor for that.
> Thanks,
> Zhihua Deng
Could you please ellaborate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 739322)
Time Spent: 2.5h (was: 2h 20m)
> Cleanup IMetaStoreClient#getPartitionsByNames APIs
> --
>
> Key: HIVE-25935
> URL: https://issues.apache.org/jira/browse/HIVE-25935
> Project: Hive
> Issue Type: Task
> Components: Metastore
>Reporter: Stamatis Zampetakis
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Currently the
> [IMetastoreClient|https://github.com/apache/hive/blob/4b7a948e45fd88372fef573be321cda40d189cc7/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java]
> interface has 8 variants of the {{getPartitionsByNames}} method. Going
> quickly over the concrete implementation it appears that not all of them are
> useful/necessary so a bit of cleanup is needed.
> Below a few potential problems I observed:
> * Some of the APIs are not used anywhere in the project (neither by
> production nor by test code).
> * Some of the APIs are deprecated in some concrete implementations but not
> globally at the interface level without an explanation why.
> * Some of the implementations simply throw without doing anything.
> * Many of the APIs are partially tested or not tested at all.
> HIVE-24743, HIVE-25281 are related since they introduce/deprecate some of the
> aforementioned APIs.
> It would be good to review the aforementioned APIs and decide what needs to
> stay and what needs to go as well as complete necessary when relevant.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26016) Remove duplicate table exists check in create_table_core api of HMSHandler
[
https://issues.apache.org/jira/browse/HIVE-26016?focusedWorklogId=739318=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-739318
]
ASF GitHub Bot logged work on HIVE-26016:
-
Author: ASF GitHub Bot
Created on: 10/Mar/22 08:46
Start Date: 10/Mar/22 08:46
Worklog Time Spent: 10m
Work Description: wecharyu commented on a change in pull request #3085:
URL: https://github.com/apache/hive/pull/3085#discussion_r823475572
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
##
@@ -2333,11 +2333,6 @@ private void create_table_core(final RawStore ms, final
CreateTableRequest req)
isReplicated = isDbReplicationTarget(db);
firePreEvent(new PreCreateTableEvent(tbl, db, this));
- // get_table checks whether database exists, it should be moved here
- if (is_table_exists(ms, tbl.getCatName(), tbl.getDbName(),
tbl.getTableName())) {
-throw new AlreadyExistsException("Table " +
getCatalogQualifiedTableName(tbl)
-+ " already exists");
- }
Review comment:
Double check is not necessary, transactions of different `pm`s do not
affect each other, and the eventual consistency is guaranteed by the backend
RDBMS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 739318)
Time Spent: 0.5h (was: 20m)
> Remove duplicate table exists check in create_table_core api of HMSHandler
> --
>
> Key: HIVE-26016
> URL: https://issues.apache.org/jira/browse/HIVE-26016
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Yu Weiqiang
>Assignee: Yu Weiqiang
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently create_table_core() will check if table exists before opening
> transaction, it does not need to check again after opening transaction, can
> remove the unnecessary check code.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25446) VectorMapJoinFastHashTable.validateCapacity AssertionError: Capacity must be a power of two
[ https://issues.apache.org/jira/browse/HIVE-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Sharma updated HIVE-25446: - Issue Type: Task (was: Bug) > VectorMapJoinFastHashTable.validateCapacity AssertionError: Capacity must be > a power of two > --- > > Key: HIVE-25446 > URL: https://issues.apache.org/jira/browse/HIVE-25446 > Project: Hive > Issue Type: Task >Affects Versions: 3.1.1 >Reporter: Matt McCline >Assignee: Ashish Sharma >Priority: Major > Fix For: 4.0.0 > > > Encountered this in a very large query: > Caused by: java.lang.AssertionError: Capacity must be a power of two > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.validateCapacity(VectorMapJoinFastHashTable.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.(VectorMapJoinFastHashTable.java:77) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.(VectorMapJoinFastBytesHashTable.java:132) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashMap.(VectorMapJoinFastBytesHashMap.java:166) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastStringHashMap.(VectorMapJoinFastStringHashMap.java:43) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.createHashTable(VectorMapJoinFastTableContainer.java:137) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.(VectorMapJoinFastTableContainer.java:86) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:122) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25446) VectorMapJoinFastHashTable.validateCapacity AssertionError: Capacity must be a power of two
[ https://issues.apache.org/jira/browse/HIVE-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Sharma updated HIVE-25446: - Priority: Trivial (was: Major) > VectorMapJoinFastHashTable.validateCapacity AssertionError: Capacity must be > a power of two > --- > > Key: HIVE-25446 > URL: https://issues.apache.org/jira/browse/HIVE-25446 > Project: Hive > Issue Type: Task >Affects Versions: 3.1.1 >Reporter: Matt McCline >Assignee: Ashish Sharma >Priority: Trivial > Fix For: 4.0.0 > > > Encountered this in a very large query: > Caused by: java.lang.AssertionError: Capacity must be a power of two > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.validateCapacity(VectorMapJoinFastHashTable.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.(VectorMapJoinFastHashTable.java:77) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.(VectorMapJoinFastBytesHashTable.java:132) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashMap.(VectorMapJoinFastBytesHashMap.java:166) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastStringHashMap.(VectorMapJoinFastStringHashMap.java:43) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.createHashTable(VectorMapJoinFastTableContainer.java:137) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.(VectorMapJoinFastTableContainer.java:86) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:122) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) -- This message was sent by Atlassian Jira (v8.20.1#820001)
