[jira] [Commented] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034680#comment-15034680 ] Prasanth Jayachandran commented on HIVE-12558: -- I will take a look. Either to log file or metrics file sink. > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table names so that things like > cache hit ratio, etc., could be calculated for every table. > We need to output them to some metrics system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12551) Fix several kryo exceptions in branch-1
[
https://issues.apache.org/jira/browse/HIVE-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034543#comment-15034543
]
Sergey Shelukhin commented on HIVE-12551:
-
Hmm.. wouldn't it be simpler to get rid of the bogus ArrayList? Other than that
+1
> Fix several kryo exceptions in branch-1
> ---
>
> Key: HIVE-12551
> URL: https://issues.apache.org/jira/browse/HIVE-12551
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Labels: serialization
> Attachments: HIVE-12551.1.patch
>
>
> HIVE-11519, HIVE-12174 and the following exception are all caused by
> unregistered classes or serializers. HIVE-12175 should have fixed these
> issues for master branch.
> {code}
> Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.NullPointerException
> Serialization trace:
> chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> expr (org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor)
> childExpressions
> (org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterStringColumnBetween)
> conditionEvaluator
> (org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:367)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:276)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:1087)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:976)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:990)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:426)
> ... 27 more
> Caused by: java.lang.NullPointerException
> at java.util.Arrays$ArrayList.size(Arrays.java:3818)
> at java.util.AbstractList.add(AbstractList.java:108)
>
[jira] [Commented] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034702#comment-15034702 ] Prasanth Jayachandran commented on HIVE-12558: -- IMHO, long running services metrics should go to metrics system. Per query information should go to web view/console. May be we can add to the new HS2 webui. > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table names so that things like > cache hit ratio, etc., could be calculated for every table. > We need to output them to some metrics system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12500) JDBC driver not overlaying params supplied via properties object when reading params from ZK
[ https://issues.apache.org/jira/browse/HIVE-12500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-12500: - Attachment: (was: HIVE-12500.2.patch) > JDBC driver not overlaying params supplied via properties object when reading > params from ZK > > > Key: HIVE-12500 > URL: https://issues.apache.org/jira/browse/HIVE-12500 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.3.0, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-12500.1.patch, HIVE-12500.2.patch > > > It makes sense to setup the connection info in one place. Right now part of > connection configuration happens in Utils#parseURL and part in the > HiveConnection constructor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12556) Ctrl-C in beeline doesn't kill Tez query on HS2
[
https://issues.apache.org/jira/browse/HIVE-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034755#comment-15034755
]
Thejas M Nair commented on HIVE-12556:
--
To make it easier to create SyncDagClient, you can use the pattern followed for
synchronized metastoreclient -
{code:java}
/**
* Creates a synchronized wrapper for any {@link IMetaStoreClient}.
* This may be used by multi-threaded applications until we have
* fixed all reentrancy bugs.
*
* @param client unsynchronized client
*
* @return synchronized client
*/
public static IMetaStoreClient newSynchronizedClient(
IMetaStoreClient client) {
return (IMetaStoreClient) Proxy.newProxyInstance(
HiveMetaStoreClient.class.getClassLoader(),
new Class [] { IMetaStoreClient.class },
new SynchronizedHandler(client));
}
private static class SynchronizedHandler implements InvocationHandler {
private final IMetaStoreClient client;
SynchronizedHandler(IMetaStoreClient client) {
this.client = client;
}
@Override
public synchronized Object invoke(Object proxy, Method method, Object []
args)
throws Throwable {
try {
return method.invoke(client, args);
} catch (InvocationTargetException e) {
throw e.getTargetException();
}
}
}
{code}
> Ctrl-C in beeline doesn't kill Tez query on HS2
> ---
>
> Key: HIVE-12556
> URL: https://issues.apache.org/jira/browse/HIVE-12556
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12556.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034756#comment-15034756 ] Sergey Shelukhin commented on HIVE-12558: - Summary in Tez counters would be nice, perhaps it should be grouped by tables. However, it should also be easy to view metrics for past queries. AFAIK (judging by some JIRA I saw) Web UI only shows very recent queries. > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table names so that things like > cache hit ratio, etc., could be calculated for every table. > We need to output them to some metrics system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11531) Add mysql-style LIMIT support to Hive, or improve ROW_NUMBER performance-wise
[ https://issues.apache.org/jira/browse/HIVE-11531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034623#comment-15034623 ] Sergey Shelukhin commented on HIVE-11531: - There are result changes in some vectorization tests. I don't think the vectorized operator changes are correct - it will skip the rows from every batch as far as I see. Also what if selected is already in use? > Add mysql-style LIMIT support to Hive, or improve ROW_NUMBER performance-wise > - > > Key: HIVE-11531 > URL: https://issues.apache.org/jira/browse/HIVE-11531 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Sergey Shelukhin >Assignee: Hui Zheng > Attachments: HIVE-11531.02.patch, HIVE-11531.03.patch, > HIVE-11531.WIP.1.patch, HIVE-11531.WIP.2.patch, HIVE-11531.patch > > > For any UIs that involve pagination, it is useful to issue queries in the > form SELECT ... LIMIT X,Y where X,Y are coordinates inside the result to be > paginated (which can be extremely large by itself). At present, ROW_NUMBER > can be used to achieve this effect, but optimizations for LIMIT such as TopN > in ReduceSink do not apply to ROW_NUMBER. We can add first class support for > "skip" to existing limit, or improve ROW_NUMBER for better performance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034687#comment-15034687 ] Sergey Shelukhin commented on HIVE-12558: - We already output them to log file... that's not a good way to collect them :P > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table names so that things like > cache hit ratio, etc., could be calculated for every table. > We need to output them to some metrics system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12020) Revert log4j2 xml configuration to properties based configuration
[ https://issues.apache.org/jira/browse/HIVE-12020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12020: - Attachment: HIVE-12020.4.patch Previous run got stuck for long hours and was manually killed. Just wanted to make sure this patch does not cause the hang. So reuploading again. > Revert log4j2 xml configuration to properties based configuration > - > > Key: HIVE-12020 > URL: https://issues.apache.org/jira/browse/HIVE-12020 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-12020.1.patch, HIVE-12020.2.patch, > HIVE-12020.3.patch, HIVE-12020.4.patch, HIVE-12020.4.patch > > > Log4j 2.4 release brought back properties based configuration. We should > revert XML based configuration and use properties based configuration instead > (less verbose and will be similar to old log4j properties). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11878) ClassNotFoundException can possibly occur if multiple jars are registered one at a time in Hive
[
https://issues.apache.org/jira/browse/HIVE-11878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034590#comment-15034590
]
Jason Dere commented on HIVE-11878:
---
I think this is happening because the changes in conf/ivysettings.xml assume
the local maven repository is in file:${user.home}/.m2/repository. I set my
MAVEN_OPTS to specify a non-default directory using -Dmaven.repo.local and also
hit this error.
In the pom file, we use the maven.repo.local setting, would there be a way to
specify this in the ivy settings?
{noformat}
${maven.repo.local}
{noformat}
> ClassNotFoundException can possibly occur if multiple jars are registered
> one at a time in Hive
>
>
> Key: HIVE-11878
> URL: https://issues.apache.org/jira/browse/HIVE-11878
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1
>Reporter: Ratandeep Ratti
>Assignee: Ratandeep Ratti
> Labels: URLClassLoader
> Attachments: HIVE-11878 ClassLoader Issues when Registering
> Jars.pptx, HIVE-11878.2.patch, HIVE-11878.3.patch, HIVE-11878.patch,
> HIVE-11878_approach3.patch,
> HIVE-11878_approach3_per_session_clasloader.patch,
> HIVE-11878_approach3_with_review_comments.patch,
> HIVE-11878_approach3_with_review_comments1.patch, HIVE-11878_qtest.patch
>
>
> When we register a jar on the Hive console. Hive creates a fresh URL
> classloader which includes the path of the current jar to be registered and
> all the jar paths of the parent classloader. The parent classlaoder is the
> current ThreadContextClassLoader. Once the URLClassloader is created Hive
> sets that as the current ThreadContextClassloader.
> So if we register multiple jars in Hive, there will be multiple
> URLClassLoaders created, each classloader including the jars from its parent
> and the one extra jar to be registered. The last URLClassLoader created will
> end up as the current ThreadContextClassLoader. (See details:
> org.apache.hadoop.hive.ql.exec.Utilities#addToClassPath)
> Now here's an example in which the above strategy can lead to a CNF exception.
> We register 2 jars *j1* and *j2* in Hive console. *j1* contains the UDF class
> *c1* and internally relies on class *c2* in jar *j2*. We register *j1* first,
> the URLClassLoader *u1* is created and also set as the
> ThreadContextClassLoader. We register *j2* next, the new URLClassLoader
> created will be *u2* with *u1* as parent and *u2* becomes the new
> ThreadContextClassLoader. Note *u2* includes paths to both jars *j1* and *j2*
> whereas *u1* only has paths to *j1* (For details see:
> org.apache.hadoop.hive.ql.exec.Utilities#addToClassPath).
> Now when we register class *c1* under a temporary function in Hive, we load
> the class using {code} class.forName("c1", true,
> Thread.currentThread().getContextClassLoader()) {code} . The
> currentThreadContext class-loader is *u2*, and it has the path to the class
> *c1*, but note that Class-loaders work by delegating to parent class-loader
> first. In this case class *c1* will be found and *defined* by class-loader
> *u1*.
> Now *c1* from jar *j1* has *u1* as its class-loader. If a method (say
> initialize) is called in *c1*, which references the class *c2*, *c2* will not
> be found since the class-loader used to search for *c2* will be *u1* (Since
> the caller's class-loader is used to load a class)
> I've added a qtest to explain the problem. Please see the attached patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034638#comment-15034638 ] Sergey Shelukhin commented on HIVE-12558: - [~sseth] [~prasanth_j] fyi. I know next to nothing about how Hadoop metrics work, so I'd be interested to look when I have time, but it won't be fast so if you have a quick solution... ;) > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table name so that things like > cache hit ratio, etc., could be partitioned by table. > We need to output them to some metric system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12559) Vectorization on MR produces different results
[ https://issues.apache.org/jira/browse/HIVE-12559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034690#comment-15034690 ] Sergey Shelukhin commented on HIVE-12559: - IIRC we consider such small loss of precision on double to be by design. Also, MR is deprecated. > Vectorization on MR produces different results > -- > > Key: HIVE-12559 > URL: https://issues.apache.org/jira/browse/HIVE-12559 > Project: Hive > Issue Type: Bug >Reporter: Laljo John Pullokkaran >Assignee: Matt McCline > > Vectorization on MR produces different results for semantically equivalent > queries. > SET hive.vectorized.execution.enabled=true; > SET hive.auto.convert.join=true; > SET hive.auto.convert.join.noconditionaltask=true; > SET hive.auto.convert.join.noconditionaltask.size=10; > SET hive.cbo.enable=false; > select sum(v1.cdouble) from alltypesorc v3 join alltypesorc v1 on > v1.csmallint=v3.csmallint join alltypesorc v2 on v1.ctinyint=v2.ctinyint; > -- Produces 6.065190932488167E11 > select sum(v1.cdouble) from alltypesorc v1 join alltypesorc v2 on > v1.ctinyint=v2.ctinyint join alltypesorc v3 on v1.csmallint=v3.csmallint; > -- Produces 6.065190932486892E11 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12559) Vectorization on MR produces different results
[ https://issues.apache.org/jira/browse/HIVE-12559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034743#comment-15034743 ] Laljo John Pullokkaran commented on HIVE-12559: --- Yes that true, i think the join order change results in different summation sequence. Which might be causing this. > Vectorization on MR produces different results > -- > > Key: HIVE-12559 > URL: https://issues.apache.org/jira/browse/HIVE-12559 > Project: Hive > Issue Type: Bug >Reporter: Laljo John Pullokkaran >Assignee: Matt McCline > > Vectorization on MR produces different results for semantically equivalent > queries. > SET hive.vectorized.execution.enabled=true; > SET hive.auto.convert.join=true; > SET hive.auto.convert.join.noconditionaltask=true; > SET hive.auto.convert.join.noconditionaltask.size=10; > SET hive.cbo.enable=false; > select sum(v1.cdouble) from alltypesorc v3 join alltypesorc v1 on > v1.csmallint=v3.csmallint join alltypesorc v2 on v1.ctinyint=v2.ctinyint; > -- Produces 6.065190932488167E11 > select sum(v1.cdouble) from alltypesorc v1 join alltypesorc v2 on > v1.ctinyint=v2.ctinyint join alltypesorc v3 on v1.csmallint=v3.csmallint; > -- Produces 6.065190932486892E11 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11312) ORC format: where clause with CHAR data type not returning any rows
[
https://issues.apache.org/jira/browse/HIVE-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034742#comment-15034742
]
Prasanth Jayachandran commented on HIVE-11312:
--
[~sershe] Could you take another look?
> ORC format: where clause with CHAR data type not returning any rows
> ---
>
> Key: HIVE-11312
> URL: https://issues.apache.org/jira/browse/HIVE-11312
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.0, 1.3.0, 1.2.1, 2.0.0
>Reporter: Thomas Friedrich
>Assignee: Prasanth Jayachandran
> Labels: orc
> Attachments: HIVE-11312.1.patch, HIVE-11312.2.patch,
> HIVE-11312.3.patch, HIVE-11312.4.patch
>
>
> Test case:
> Setup:
> create table orc_test( col1 string, col2 char(10)) stored as orc
> tblproperties ("orc.compress"="NONE");
> insert into orc_test values ('val1', '1');
> Query:
> select * from orc_test where col2='1';
> Query returns no row.
> Problem is introduced with HIVE-10286, class RecordReaderImpl.java, method
> evaluatePredicateRange.
> Old code:
> - Object baseObj = predicate.getLiteral(PredicateLeaf.FileFormat.ORC);
> - Object minValue = getConvertedStatsObj(min, baseObj);
> - Object maxValue = getConvertedStatsObj(max, baseObj);
> - Object predObj = getBaseObjectForComparison(baseObj, minValue);
> New code:
> + Object baseObj = predicate.getLiteral();
> + Object minValue = getBaseObjectForComparison(predicate.getType(), min);
> + Object maxValue = getBaseObjectForComparison(predicate.getType(), max);
> + Object predObj = getBaseObjectForComparison(predicate.getType(), baseObj);
> The values for min and max are of type String which contain as many
> characters as the CHAR column indicated. For example if the type is CHAR(10),
> and the row has value 1, the value of String min is "1 ";
> Before Hive 1.2, the method getConvertedStatsObj would call
> StringUtils.stripEnd(statsObj.toString(), null); which would remove the
> trailing spaces from min and max. Later in the compareToRange method, it was
> able to compare "1" with "1".
> In Hive 1.2 with the use getBaseObjectForComparison method, it simply returns
> obj.String if the data type is String, which means minValue and maxValue are
> still "1 ".
> As a result, the compareToRange method will return a wrong value
> ("1".compareTo("1 ") -9 instead of 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12526) PerfLogger for hive compiler and optimizer
[ https://issues.apache.org/jira/browse/HIVE-12526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034600#comment-15034600 ] Ashutosh Chauhan commented on HIVE-12526: - Patch LGTM +1 For performance, its not inside any loop and is called for each optimization phase, so should not be a drain. [~hsubramaniyan] Can you post output of new log lines generated on a non-trivial query so that we have an idea of how it will look like. > PerfLogger for hive compiler and optimizer > -- > > Key: HIVE-12526 > URL: https://issues.apache.org/jira/browse/HIVE-12526 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-12526.1.patch, HIVE-12526.2.patch > > > This jira is intended to use the perflogger to track compilation times and > optimization times (calcite, tez compiler, physical compiler) etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12558: Description: Right now, LLAP logs counters for every fragment; most of them are IO related and could be very useful, they also include table names so that things like cache hit ratio, etc., could be calculated for every table. We need to output them to some metrics system (preserving the breakdown by table, possibly also adding query ID or even stage) so that they'd be usable without grep/sed/awk. was: Right now, LLAP logs counters for every fragment; most of them are IO related and could be very useful, they also include table name so that things like cache hit ratio, etc., could be partitioned by table. We need to output them to some metric system (preserving the breakdown by table, possibly also adding query ID or even stage) so that they'd be usable without grep/sed/awk. > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table names so that things like > cache hit ratio, etc., could be calculated for every table. > We need to output them to some metrics system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12558: Component/s: llap > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table name so that things like > cache hit ratio, etc., could be partitioned by table. > We need to output them to some metric system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-12558: Assignee: Prasanth Jayachandran > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table names so that things like > cache hit ratio, etc., could be calculated for every table. > We need to output them to some metrics system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12500) JDBC driver not overlaying params supplied via properties object when reading params from ZK
[ https://issues.apache.org/jira/browse/HIVE-12500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-12500: - Attachment: HIVE-12500.2.patch > JDBC driver not overlaying params supplied via properties object when reading > params from ZK > > > Key: HIVE-12500 > URL: https://issues.apache.org/jira/browse/HIVE-12500 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.3.0, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-12500.1.patch, HIVE-12500.2.patch, > HIVE-12500.2.patch > > > It makes sense to setup the connection info in one place. Right now part of > connection configuration happens in Utils#parseURL and part in the > HiveConnection constructor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11312) ORC format: where clause with CHAR data type not returning any rows
[
https://issues.apache.org/jira/browse/HIVE-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034747#comment-15034747
]
Sergey Shelukhin commented on HIVE-11312:
-
+1
> ORC format: where clause with CHAR data type not returning any rows
> ---
>
> Key: HIVE-11312
> URL: https://issues.apache.org/jira/browse/HIVE-11312
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.0, 1.3.0, 1.2.1, 2.0.0
>Reporter: Thomas Friedrich
>Assignee: Prasanth Jayachandran
> Labels: orc
> Attachments: HIVE-11312.1.patch, HIVE-11312.2.patch,
> HIVE-11312.3.patch, HIVE-11312.4.patch
>
>
> Test case:
> Setup:
> create table orc_test( col1 string, col2 char(10)) stored as orc
> tblproperties ("orc.compress"="NONE");
> insert into orc_test values ('val1', '1');
> Query:
> select * from orc_test where col2='1';
> Query returns no row.
> Problem is introduced with HIVE-10286, class RecordReaderImpl.java, method
> evaluatePredicateRange.
> Old code:
> - Object baseObj = predicate.getLiteral(PredicateLeaf.FileFormat.ORC);
> - Object minValue = getConvertedStatsObj(min, baseObj);
> - Object maxValue = getConvertedStatsObj(max, baseObj);
> - Object predObj = getBaseObjectForComparison(baseObj, minValue);
> New code:
> + Object baseObj = predicate.getLiteral();
> + Object minValue = getBaseObjectForComparison(predicate.getType(), min);
> + Object maxValue = getBaseObjectForComparison(predicate.getType(), max);
> + Object predObj = getBaseObjectForComparison(predicate.getType(), baseObj);
> The values for min and max are of type String which contain as many
> characters as the CHAR column indicated. For example if the type is CHAR(10),
> and the row has value 1, the value of String min is "1 ";
> Before Hive 1.2, the method getConvertedStatsObj would call
> StringUtils.stripEnd(statsObj.toString(), null); which would remove the
> trailing spaces from min and max. Later in the compareToRange method, it was
> able to compare "1" with "1".
> In Hive 1.2 with the use getBaseObjectForComparison method, it simply returns
> obj.String if the data type is String, which means minValue and maxValue are
> still "1 ".
> As a result, the compareToRange method will return a wrong value
> ("1".compareTo("1 ") -9 instead of 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11312) ORC format: where clause with CHAR data type not returning any rows
[
https://issues.apache.org/jira/browse/HIVE-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-11312:
-
Attachment: HIVE-11312.4.patch
Addressed review comments.
> ORC format: where clause with CHAR data type not returning any rows
> ---
>
> Key: HIVE-11312
> URL: https://issues.apache.org/jira/browse/HIVE-11312
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.0, 1.3.0, 1.2.1, 2.0.0
>Reporter: Thomas Friedrich
>Assignee: Prasanth Jayachandran
> Labels: orc
> Attachments: HIVE-11312.1.patch, HIVE-11312.2.patch,
> HIVE-11312.3.patch, HIVE-11312.4.patch
>
>
> Test case:
> Setup:
> create table orc_test( col1 string, col2 char(10)) stored as orc
> tblproperties ("orc.compress"="NONE");
> insert into orc_test values ('val1', '1');
> Query:
> select * from orc_test where col2='1';
> Query returns no row.
> Problem is introduced with HIVE-10286, class RecordReaderImpl.java, method
> evaluatePredicateRange.
> Old code:
> - Object baseObj = predicate.getLiteral(PredicateLeaf.FileFormat.ORC);
> - Object minValue = getConvertedStatsObj(min, baseObj);
> - Object maxValue = getConvertedStatsObj(max, baseObj);
> - Object predObj = getBaseObjectForComparison(baseObj, minValue);
> New code:
> + Object baseObj = predicate.getLiteral();
> + Object minValue = getBaseObjectForComparison(predicate.getType(), min);
> + Object maxValue = getBaseObjectForComparison(predicate.getType(), max);
> + Object predObj = getBaseObjectForComparison(predicate.getType(), baseObj);
> The values for min and max are of type String which contain as many
> characters as the CHAR column indicated. For example if the type is CHAR(10),
> and the row has value 1, the value of String min is "1 ";
> Before Hive 1.2, the method getConvertedStatsObj would call
> StringUtils.stripEnd(statsObj.toString(), null); which would remove the
> trailing spaces from min and max. Later in the compareToRange method, it was
> able to compare "1" with "1".
> In Hive 1.2 with the use getBaseObjectForComparison method, it simply returns
> obj.String if the data type is String, which means minValue and maxValue are
> still "1 ".
> As a result, the compareToRange method will return a wrong value
> ("1".compareTo("1 ") -9 instead of 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11312) ORC format: where clause with CHAR data type not returning any rows
[
https://issues.apache.org/jira/browse/HIVE-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034510#comment-15034510
]
Prasanth Jayachandran commented on HIVE-11312:
--
Make sense. Fixed in latest patch.
> ORC format: where clause with CHAR data type not returning any rows
> ---
>
> Key: HIVE-11312
> URL: https://issues.apache.org/jira/browse/HIVE-11312
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.0, 1.3.0, 1.2.1, 2.0.0
>Reporter: Thomas Friedrich
>Assignee: Prasanth Jayachandran
> Labels: orc
> Attachments: HIVE-11312.1.patch, HIVE-11312.2.patch,
> HIVE-11312.3.patch, HIVE-11312.4.patch
>
>
> Test case:
> Setup:
> create table orc_test( col1 string, col2 char(10)) stored as orc
> tblproperties ("orc.compress"="NONE");
> insert into orc_test values ('val1', '1');
> Query:
> select * from orc_test where col2='1';
> Query returns no row.
> Problem is introduced with HIVE-10286, class RecordReaderImpl.java, method
> evaluatePredicateRange.
> Old code:
> - Object baseObj = predicate.getLiteral(PredicateLeaf.FileFormat.ORC);
> - Object minValue = getConvertedStatsObj(min, baseObj);
> - Object maxValue = getConvertedStatsObj(max, baseObj);
> - Object predObj = getBaseObjectForComparison(baseObj, minValue);
> New code:
> + Object baseObj = predicate.getLiteral();
> + Object minValue = getBaseObjectForComparison(predicate.getType(), min);
> + Object maxValue = getBaseObjectForComparison(predicate.getType(), max);
> + Object predObj = getBaseObjectForComparison(predicate.getType(), baseObj);
> The values for min and max are of type String which contain as many
> characters as the CHAR column indicated. For example if the type is CHAR(10),
> and the row has value 1, the value of String min is "1 ";
> Before Hive 1.2, the method getConvertedStatsObj would call
> StringUtils.stripEnd(statsObj.toString(), null); which would remove the
> trailing spaces from min and max. Later in the compareToRange method, it was
> able to compare "1" with "1".
> In Hive 1.2 with the use getBaseObjectForComparison method, it simply returns
> obj.String if the data type is String, which means minValue and maxValue are
> still "1 ".
> As a result, the compareToRange method will return a wrong value
> ("1".compareTo("1 ") -9 instead of 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12236) LLAP: Prevent metadata queries from thrashing LLAP cache
[
https://issues.apache.org/jira/browse/HIVE-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034577#comment-15034577
]
Ashutosh Chauhan commented on HIVE-12236:
-
I dont understand either the problem or proposed patch. Queries like {{select
*}} don't launch job anyway, which apparently [~gopalv] is trying to optimize
for. Patch AFAICT tries to extend optimization for {{select col_list }} queries
for which select expressions are deemed inexpensive, which [~hagleitn] seems to
suggest is not safe to do. Totally at loss here : ) Suggestions?
> LLAP: Prevent metadata queries from thrashing LLAP cache
>
>
> Key: HIVE-12236
> URL: https://issues.apache.org/jira/browse/HIVE-12236
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-12236.WIP.patch
>
>
> Currently, metadata queries fired by BI tools tend to thrash LLAP's cache.
> Bypass the cache and process metadata queries directly from HiveServer2
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12538) After set spark related config, SparkSession never get reused
[ https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034578#comment-15034578 ] Jimmy Xiang commented on HIVE-12538: bq. Not quite follow.Is there anything special in operation conf for SparkSession? And when to set "isSparkConfigUpdated =false " ? We can set it to false for the session level conf only. So this flag in the operation level is totaly ignored, all the time. Things are a little tricky actually. If we use the session level conf, we could miss some non-spark-related settings in the operation level conf. If we use the operation level conf, we could miss some spark-related settings in the session level conf. Instead of just maintaining a isSparkConfigUpdated flag, probably, we should have a separate map to store such changed spark-related settings temporarily. This map can be reset upon SparkUtilities#getSparkSession() is invoked. > After set spark related config, SparkSession never get reused > - > > Key: HIVE-12538 > URL: https://issues.apache.org/jira/browse/HIVE-12538 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.3.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-12538.1.patch, HIVE-12538.patch > > > Hive on Spark yarn-cluster mode. > After setting "set spark.yarn.queue=QueueA;" , > run the query "select count(*) from test" 3 times and you will find 3 > different yarn applications. > Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING > & UNDEFINED state waiting for next work. > And if you submit one more "select count(*) from test" ,the third one will be > in FINISHED & SUCCEEDED state and a new yarn application will start up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-12557) NPE while removing entry in LRFU cache
[
https://issues.apache.org/jira/browse/HIVE-12557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-12557:
---
Assignee: Sergey Shelukhin
> NPE while removing entry in LRFU cache
> --
>
> Key: HIVE-12557
> URL: https://issues.apache.org/jira/browse/HIVE-12557
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
>
> {code}
> Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 6,
> vertexId=vertex_1448429572030_1851_5_00, diagnostics=[Task failed,
> taskId=task_1448429572030_1851_5_00_06, diagnostics=[TaskAttempt 0
> failed, info=[Error: Failure while running task:
> attempt_1448429572030_1851_5_00_06_0:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:348)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:74)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:352)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
> ... 14 more
> Caused by: java.io.IOException: java.io.IOException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
> ... 16 more
> Caused by: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:283)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:239)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:167)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:103)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
> ... 22 more
> Caused by: java.lang.NullPointerException
> at
>
[jira] [Resolved] (HIVE-12551) Fix several kryo exceptions in branch-1
[
https://issues.apache.org/jira/browse/HIVE-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran resolved HIVE-12551.
--
Resolution: Fixed
Fix Version/s: 1.3.0
Committed to branch-1
> Fix several kryo exceptions in branch-1
> ---
>
> Key: HIVE-12551
> URL: https://issues.apache.org/jira/browse/HIVE-12551
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Labels: serialization
> Fix For: 1.3.0
>
> Attachments: HIVE-12551.1.patch
>
>
> HIVE-11519, HIVE-12174 and the following exception are all caused by
> unregistered classes or serializers. HIVE-12175 should have fixed these
> issues for master branch.
> {code}
> Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.NullPointerException
> Serialization trace:
> chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> expr (org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor)
> childExpressions
> (org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterStringColumnBetween)
> conditionEvaluator
> (org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:367)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:276)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:1087)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:976)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:990)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:426)
> ... 27 more
> Caused by: java.lang.NullPointerException
> at java.util.Arrays$ArrayList.size(Arrays.java:3818)
> at java.util.AbstractList.add(AbstractList.java:108)
> at
[jira] [Commented] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034696#comment-15034696 ] Prasanth Jayachandran commented on HIVE-12558: -- >From the title i thought we aren't even doing that :). How about in console >under hive.exec.tez.print.summary? > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table names so that things like > cache hit ratio, etc., could be calculated for every table. > We need to output them to some metrics system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12537) RLEv2 doesn't seem to work
[
https://issues.apache.org/jira/browse/HIVE-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034675#comment-15034675
]
Hive QA commented on HIVE-12537:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12774984/HIVE-12537.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 9877 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.ql.io.orc.TestVectorOrcFile.testMemoryManagementV12
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6188/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6188/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6188/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12774984 - PreCommit-HIVE-TRUNK-Build
> RLEv2 doesn't seem to work
> --
>
> Key: HIVE-12537
> URL: https://issues.apache.org/jira/browse/HIVE-12537
> Project: Hive
> Issue Type: Bug
> Components: File Formats, ORC
>Affects Versions: 0.14.0, 1.0.1, 1.1.1, 1.3.0, 1.2.1, 2.0.0
>Reporter: Bogdan Raducanu
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Labels: orc, orcfile
> Attachments: HIVE-12537.1.patch, HIVE-12537.2.patch, Main.java,
> orcdump.txt
>
>
> Perhaps I'm doing something wrong or is actually working as expected.
> Putting 1 million constant int32 values produces an ORC file of 1MB.
> Surprisingly, 1 million consecutive ints produces a much smaller file.
> Code and FileDump attached.
> {code}
> ObjectInspector inspector =
> ObjectInspectorFactory.getReflectionObjectInspector(
> Integer.class,
> ObjectInspectorFactory.ObjectInspectorOptions.JAVA);
> Writer w = OrcFile.createWriter(new Path("/tmp/my.orc"),
> OrcFile.writerOptions(new Configuration())
> .compress(CompressionKind.NONE)
> .inspector(inspector)
>
> .encodingStrategy(OrcFile.EncodingStrategy.COMPRESSION)
> .version(OrcFile.Version.V_0_12)
> );
> for (int i = 0; i < 100; ++i) {
> w.addRow(123);
> }
> w.close();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034674#comment-15034674 ] Sergey Shelukhin commented on HIVE-11527: - Hi. Can you post the patch on https://reviews.apache.org/r/ (hive-git is the repo name)? Thanks. I will take a look shortly. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12515) Clean the SparkCounters related code after remove counter based stats collection[Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-12515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034649#comment-15034649
]
Hive QA commented on HIVE-12515:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12775041/HIVE-12515.2-spark.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 9862 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-bucket5.q-infer_bucket_sort_merge.q-and-1-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_memcheck
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/1017/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/1017/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-1017/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12775041 - PreCommit-HIVE-SPARK-Build
> Clean the SparkCounters related code after remove counter based stats
> collection[Spark Branch]
> --
>
> Key: HIVE-12515
> URL: https://issues.apache.org/jira/browse/HIVE-12515
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Chengxiang Li
>Assignee: Rui Li
> Attachments: HIVE-12515.1-spark.patch, HIVE-12515.2-spark.patch
>
>
> As SparkCounters is only used to collection stats, after HIVE-12411, we does
> not need it anymore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034756#comment-15034756 ] Sergey Shelukhin edited comment on HIVE-12558 at 12/1/15 10:36 PM: --- Summary in Tez print-out would be nice, perhaps it should be grouped by tables. However, it should also be easy to view metrics for past queries. AFAIK (judging by some JIRA I saw) Web UI only shows very recent queries. was (Author: sershe): Summary in Tez counters would be nice, perhaps it should be grouped by tables. However, it should also be easy to view metrics for past queries. AFAIK (judging by some JIRA I saw) Web UI only shows very recent queries. > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table names so that things like > cache hit ratio, etc., could be calculated for every table. > We need to output them to some metrics system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12537) RLEv2 doesn't seem to work
[
https://issues.apache.org/jira/browse/HIVE-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-12537:
-
Attachment: HIVE-12537.3.patch
> RLEv2 doesn't seem to work
> --
>
> Key: HIVE-12537
> URL: https://issues.apache.org/jira/browse/HIVE-12537
> Project: Hive
> Issue Type: Bug
> Components: File Formats, ORC
>Affects Versions: 0.14.0, 1.0.1, 1.1.1, 1.3.0, 1.2.1, 2.0.0
>Reporter: Bogdan Raducanu
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Labels: orc, orcfile
> Attachments: HIVE-12537.1.patch, HIVE-12537.2.patch,
> HIVE-12537.3.patch, Main.java, orcdump.txt
>
>
> Perhaps I'm doing something wrong or is actually working as expected.
> Putting 1 million constant int32 values produces an ORC file of 1MB.
> Surprisingly, 1 million consecutive ints produces a much smaller file.
> Code and FileDump attached.
> {code}
> ObjectInspector inspector =
> ObjectInspectorFactory.getReflectionObjectInspector(
> Integer.class,
> ObjectInspectorFactory.ObjectInspectorOptions.JAVA);
> Writer w = OrcFile.createWriter(new Path("/tmp/my.orc"),
> OrcFile.writerOptions(new Configuration())
> .compress(CompressionKind.NONE)
> .inspector(inspector)
>
> .encodingStrategy(OrcFile.EncodingStrategy.COMPRESSION)
> .version(OrcFile.Version.V_0_12)
> );
> for (int i = 0; i < 100; ++i) {
> w.addRow(123);
> }
> w.close();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12537) RLEv2 doesn't seem to work
[
https://issues.apache.org/jira/browse/HIVE-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034801#comment-15034801
]
Prasanth Jayachandran commented on HIVE-12537:
--
Fixed TestVectorOrcFile test failure. Other failures seems to be unrelated to
this change.
> RLEv2 doesn't seem to work
> --
>
> Key: HIVE-12537
> URL: https://issues.apache.org/jira/browse/HIVE-12537
> Project: Hive
> Issue Type: Bug
> Components: File Formats, ORC
>Affects Versions: 0.14.0, 1.0.1, 1.1.1, 1.3.0, 1.2.1, 2.0.0
>Reporter: Bogdan Raducanu
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Labels: orc, orcfile
> Attachments: HIVE-12537.1.patch, HIVE-12537.2.patch,
> HIVE-12537.3.patch, Main.java, orcdump.txt
>
>
> Perhaps I'm doing something wrong or is actually working as expected.
> Putting 1 million constant int32 values produces an ORC file of 1MB.
> Surprisingly, 1 million consecutive ints produces a much smaller file.
> Code and FileDump attached.
> {code}
> ObjectInspector inspector =
> ObjectInspectorFactory.getReflectionObjectInspector(
> Integer.class,
> ObjectInspectorFactory.ObjectInspectorOptions.JAVA);
> Writer w = OrcFile.createWriter(new Path("/tmp/my.orc"),
> OrcFile.writerOptions(new Configuration())
> .compress(CompressionKind.NONE)
> .inspector(inspector)
>
> .encodingStrategy(OrcFile.EncodingStrategy.COMPRESSION)
> .version(OrcFile.Version.V_0_12)
> );
> for (int i = 0; i < 100; ++i) {
> w.addRow(123);
> }
> w.close();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12236) LLAP: Prevent metadata queries from thrashing LLAP cache
[ https://issues.apache.org/jira/browse/HIVE-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12236: Attachment: HIVE-12236.patch Extended SimpleFetchOptimizer to optimize select * queries > LLAP: Prevent metadata queries from thrashing LLAP cache > > > Key: HIVE-12236 > URL: https://issues.apache.org/jira/browse/HIVE-12236 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 1.3.0, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-12236.WIP.patch, HIVE-12236.patch > > > Currently, metadata queries fired by BI tools tend to thrash LLAP's cache. > Bypass the cache and process metadata queries directly from HiveServer2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12556) Ctrl-C in beeline doesn't kill Tez query on HS2
[ https://issues.apache.org/jira/browse/HIVE-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034887#comment-15034887 ] Gunther Hagleitner commented on HIVE-12556: --- LGTM +1. It would be really nice not to have to maintain this SyncDagClient. ([~hitesh]) > Ctrl-C in beeline doesn't kill Tez query on HS2 > --- > > Key: HIVE-12556 > URL: https://issues.apache.org/jira/browse/HIVE-12556 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12556.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12560) Few tests hang forever
[ https://issues.apache.org/jira/browse/HIVE-12560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12560: - Description: TestHWISessionManager is hanging > Few tests hang forever > -- > > Key: HIVE-12560 > URL: https://issues.apache.org/jira/browse/HIVE-12560 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12560.patch > > > TestHWISessionManager is hanging -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12560) Few tests hang forever
[ https://issues.apache.org/jira/browse/HIVE-12560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034915#comment-15034915 ] Prasanth Jayachandran commented on HIVE-12560: -- +1 > Few tests hang forever > -- > > Key: HIVE-12560 > URL: https://issues.apache.org/jira/browse/HIVE-12560 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12560.patch > > > TestHWISessionManager is hanging -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12506) SHOW CREATE TABLE command creates a table that does not work for RCFile format
[
https://issues.apache.org/jira/browse/HIVE-12506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034942#comment-15034942
]
Jimmy Xiang commented on HIVE-12506:
+1
> SHOW CREATE TABLE command creates a table that does not work for RCFile format
> --
>
> Key: HIVE-12506
> URL: https://issues.apache.org/jira/browse/HIVE-12506
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.1.1
>Reporter: Eric Lin
>Assignee: Chaoyu Tang
> Attachments: HIVE-12506.patch, HIVE-12506.patch
>
>
> See the following test case:
> 1) Create a table with RCFile format:
> {code}
> DROP TABLE IF EXISTS test;
> CREATE TABLE test (a int) PARTITIONED BY (p int)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
> STORED AS RCFILE;
> {code}
> 2) run "DESC FORMATTED test"
> {code}
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe
> InputFormat: org.apache.hadoop.hive.ql.io.RCFileInputFormat
> OutputFormat: org.apache.hadoop.hive.ql.io.RCFileOutputFormat
> {code}
> shows that SerDe used is "ColumnarSerDe"
> 3) run "SHOW CREATE TABLE" and get the output:
> {code}
> CREATE TABLE `test`(
> `a` int)
> PARTITIONED BY (
> `p` int)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '|'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.RCFileInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.RCFileOutputFormat'
> LOCATION
> 'hdfs://node5.lab.cloudera.com:8020/user/hive/warehouse/case_78732.db/test'
> TBLPROPERTIES (
> 'transient_lastDdlTime'='1448343875')
> {code}
> Note that there is no mention of "ColumnarSerDe"
> 4) Drop the table and then create the table again using the output from 3)
> 5) Check the output of "DESC FORMATTED test"
> {code}
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> InputFormat: org.apache.hadoop.hive.ql.io.RCFileInputFormat
> OutputFormat: org.apache.hadoop.hive.ql.io.RCFileOutputFormat
> {code}
> The SerDe falls back to "LazySimpleSerDe", which is not correct.
> Any further query tries to INSERT or SELECT this table will fail with errors
> I suspect that we can't specify ROW FORMAT DELIMITED with ROW FORMAT SERDE at
> the same time at table creation, this causes confusion to end users as copy
> table structure using "SHOW CREATE TABLE" will not work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12557) NPE while removing entry in LRFU cache
[
https://issues.apache.org/jira/browse/HIVE-12557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034949#comment-15034949
]
Sergey Shelukhin commented on HIVE-12557:
-
Nm, I see an epic bug
> NPE while removing entry in LRFU cache
> --
>
> Key: HIVE-12557
> URL: https://issues.apache.org/jira/browse/HIVE-12557
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
>
> {code}
> Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 6,
> vertexId=vertex_1448429572030_1851_5_00, diagnostics=[Task failed,
> taskId=task_1448429572030_1851_5_00_06, diagnostics=[TaskAttempt 0
> failed, info=[Error: Failure while running task:
> attempt_1448429572030_1851_5_00_06_0:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:348)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:74)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:352)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
> ... 14 more
> Caused by: java.io.IOException: java.io.IOException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
> ... 16 more
> Caused by: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:283)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:239)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:167)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:103)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
> ... 22 more
> Caused by: java.lang.NullPointerException
> at
>
[jira] [Commented] (HIVE-12556) Ctrl-C in beeline doesn't kill Tez query on HS2
[ https://issues.apache.org/jira/browse/HIVE-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034769#comment-15034769 ] Sergey Shelukhin commented on HIVE-12556: - That uses reflection... I'd rather not use reflection. > Ctrl-C in beeline doesn't kill Tez query on HS2 > --- > > Key: HIVE-12556 > URL: https://issues.apache.org/jira/browse/HIVE-12556 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12556.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9642) Hive metastore client retries don't happen consistently for all api calls
[ https://issues.apache.org/jira/browse/HIVE-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-9642: Attachment: HIVE-9642.5.patch attaching same 5.patch file again to kick off precommit tests > Hive metastore client retries don't happen consistently for all api calls > - > > Key: HIVE-9642 > URL: https://issues.apache.org/jira/browse/HIVE-9642 > Project: Hive > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Xiaobing Zhou >Assignee: Daniel Dai > Attachments: HIVE-9642.1.patch, HIVE-9642.2.patch, HIVE-9642.3.patch, > HIVE-9642.4.patch, HIVE-9642.5.patch, HIVE-9642.5.patch > > > When org.apache.thrift.transport.TTransportException is thrown for issues > like socket timeout, the retry via RetryingMetaStoreClient happens only in > certain cases. > Retry happens for the getDatabase call in but not for getAllDatabases(). > The reason is RetryingMetaStoreClient checks for TTransportException being > the cause for InvocationTargetException. But in case of some calls such as > getAllDatabases in HiveMetastoreClient, all exceptions get wrapped in a > MetaException. We should remove this unnecessary wrapping of exceptions for > certain functions in HMC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12429) Switch default Hive authorization to SQLStandardAuth in 2.0
[ https://issues.apache.org/jira/browse/HIVE-12429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-12429: - Assignee: Daniel Dai (was: Alan Gates) > Switch default Hive authorization to SQLStandardAuth in 2.0 > --- > > Key: HIVE-12429 > URL: https://issues.apache.org/jira/browse/HIVE-12429 > Project: Hive > Issue Type: Task > Components: Authorization, Security >Affects Versions: 2.0.0 >Reporter: Alan Gates >Assignee: Daniel Dai > > Hive's default authorization is not real security, as it does not secure a > number of features and anyone can grant access to any object to any user. We > should switch the default o SQLStandardAuth, which provides real > authentication. > As this is a backwards incompatible change this was hard to do previously, > but 2.0 gives us a place to do this type of change. > By default authorization will still be off, as there are a few other things > to set when turning on authorization (such as the list of admin users). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12552) Wrong number of reducer estimation causing job to fail
[
https://issues.apache.org/jira/browse/HIVE-12552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-12552:
Attachment: HIVE-12552.2.patch
Addressed review comments from [~hagleitn]
With 2.0f, it was generating 1009 tasks and most of them were not getting
enough data; which could have been handled with less tasks. Got around 11-13%
improvement with less number of tasks in llap mode (attached images show
container mode for debugging purpose). Haven't changed bytes per reducer in my
run, which could bring down the number of reduce tasks.
> Wrong number of reducer estimation causing job to fail
> --
>
> Key: HIVE-12552
> URL: https://issues.apache.org/jira/browse/HIVE-12552
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: 6_plan.txt, HIVE-12552.1.patch, HIVE-12552.2.patch,
> With_max_partition_0.5_setting.png, with_default_setting.png
>
>
> {noformat}
> ], TaskAttempt 3 failed, info=[Error: Failure while running task:
> attempt_1448429572030_1812_1_03_29_3:java.lang.RuntimeException:
> java.lang.RuntimeException: Hive Runtime Error while closing operators:
> java.io.IOException: Illegal partition for 01 6c 6f 61 6e 20 61 63 63 6f 75
> 6e 74 00 01 80 1f e1 d7 ff (-1)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:348)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Hive Runtime Error while closing
> operators: java.io.IOException: Illegal partition for 01 6c 6f 61 6e 20 61 63
> 63 6f 75 6e 74 00 01 80 1f e1 d7 ff (-1)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:341)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: Illegal partition for 01 6c 6f 61 6e 20 61 63 63 6f 75
> 6e 74 00 01 80 1f e1 d7 ff (-1)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:402)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:852)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.writeSingleRow(VectorGroupByOperator.java:904)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.access$400(VectorGroupByOperator.java:59)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.flush(VectorGroupByOperator.java:469)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.close(VectorGroupByOperator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.closeOp(VectorGroupByOperator.java:950)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:656)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:670)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:670)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:318)
> ... 15 more
> Caused by: java.io.IOException: Illegal partition for 01 6c 6f 61 6e 20 61 63
> 63 6f 75 6e 74 00 01 80 1f e1 d7 ff (-1)
> at
> org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.collect(PipelinedSorter.java:379)
> at
> org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:357)
> at
>
[jira] [Updated] (HIVE-12560) Few tests hang forever
[ https://issues.apache.org/jira/browse/HIVE-12560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12560: Attachment: HIVE-12560.patch Ignore these test temporarily so that Hive QA dont wait for 2hours to timeout till we root cause it. > Few tests hang forever > -- > > Key: HIVE-12560 > URL: https://issues.apache.org/jira/browse/HIVE-12560 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12560.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12559) Vectorization on MR produces different results
[ https://issues.apache.org/jira/browse/HIVE-12559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034783#comment-15034783 ] Laljo John Pullokkaran commented on HIVE-12559: --- https://docs.oracle.com/cd/E19957-01/806-3568/ncg_math.html > Vectorization on MR produces different results > -- > > Key: HIVE-12559 > URL: https://issues.apache.org/jira/browse/HIVE-12559 > Project: Hive > Issue Type: Bug >Reporter: Laljo John Pullokkaran >Assignee: Matt McCline > > Vectorization on MR produces different results for semantically equivalent > queries. > SET hive.vectorized.execution.enabled=true; > SET hive.auto.convert.join=true; > SET hive.auto.convert.join.noconditionaltask=true; > SET hive.auto.convert.join.noconditionaltask.size=10; > SET hive.cbo.enable=false; > select sum(v1.cdouble) from alltypesorc v3 join alltypesorc v1 on > v1.csmallint=v3.csmallint join alltypesorc v2 on v1.ctinyint=v2.ctinyint; > -- Produces 6.065190932488167E11 > select sum(v1.cdouble) from alltypesorc v1 join alltypesorc v2 on > v1.ctinyint=v2.ctinyint join alltypesorc v3 on v1.csmallint=v3.csmallint; > -- Produces 6.065190932486892E11 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11371) Null pointer exception for nested table query when using ORC versus text
[ https://issues.apache.org/jira/browse/HIVE-11371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034812#comment-15034812 ] Prasanth Jayachandran commented on HIVE-11371: -- +1 > Null pointer exception for nested table query when using ORC versus text > > > Key: HIVE-11371 > URL: https://issues.apache.org/jira/browse/HIVE-11371 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 1.2.0 >Reporter: N Campbell >Assignee: Matt McCline > Attachments: HIVE-11371.01.patch, HIVE-11371.02.patch, TJOIN1, > TJOIN2, TJOIN3, TJOIN4 > > > Following query will fail if the file format is ORC > select tj1rnum, tj2rnum, tjoin3.rnum as rnumt3 from (select tjoin1.rnum > tj1rnum, tjoin2.rnum tj2rnum, tjoin2.c1 tj2c1 from tjoin1 left outer join > tjoin2 on tjoin1.c1 = tjoin2.c1 ) tj left outer join tjoin3 on tj2c1 = > tjoin3.c1 > aused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.vector.VectorCopyRow$LongCopyRow.copy(VectorCopyRow.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.VectorCopyRow.copyByReference(VectorCopyRow.java:260) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:238) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinOuterGenerateResultOperator.finishOuter(VectorMapJoinOuterGenerateResultOperator.java:495) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinOuterLongOperator.process(VectorMapJoinOuterLongOperator.java:430) > ... 22 more > ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 > killedTasks:0, Vertex vertex_1437788144883_0004_2_02 [Map 1] killed/failed > due to:null]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 > killedVertices:0 > SQLState: 08S01 > ErrorCode: 2 > getDatabaseProductNameApache Hive > getDatabaseProductVersion 1.2.1.2.3.0.0-2557 > getDriverName Hive JDBC > getDriverVersion 1.2.1.2.3.0.0-2557 > getDriverMajorVersion 1 > getDriverMinorVersion 2 > create table if not exists TJOIN1 (RNUM int , C1 int, C2 int) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc; > create table if not exists TJOIN2 (RNUM int , C1 int, C2 char(2)) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; > create table if not exists TJOIN3 (RNUM int , C1 int, C2 char(2)) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; > create table if not exists TJOIN4 (RNUM int , C1 int, C2 char(2)) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12557) NPE while removing entry in LRFU cache
[
https://issues.apache.org/jira/browse/HIVE-12557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034858#comment-15034858
]
Sergey Shelukhin commented on HIVE-12557:
-
Is this easy to repro?
> NPE while removing entry in LRFU cache
> --
>
> Key: HIVE-12557
> URL: https://issues.apache.org/jira/browse/HIVE-12557
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
>
> {code}
> Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 6,
> vertexId=vertex_1448429572030_1851_5_00, diagnostics=[Task failed,
> taskId=task_1448429572030_1851_5_00_06, diagnostics=[TaskAttempt 0
> failed, info=[Error: Failure while running task:
> attempt_1448429572030_1851_5_00_06_0:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:348)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:74)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:352)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
> ... 14 more
> Caused by: java.io.IOException: java.io.IOException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
> ... 16 more
> Caused by: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:283)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:239)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:167)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:103)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
> ... 22 more
> Caused by: java.lang.NullPointerException
> at
>
[jira] [Commented] (HIVE-12182) ALTER TABLE PARTITION COLUMN does not set partition column comments
[
https://issues.apache.org/jira/browse/HIVE-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034865#comment-15034865
]
Lefty Leverenz commented on HIVE-12182:
---
Did this make any syntax changes? If so, it needs a TODOC1.3 label and a
release note.
Here's where the documentation should go:
* [DDL -- Alter Column -- ChangeColumnName/Type/Position/Comment |
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ChangeColumnName/Type/Position/Comment]
> ALTER TABLE PARTITION COLUMN does not set partition column comments
> ---
>
> Key: HIVE-12182
> URL: https://issues.apache.org/jira/browse/HIVE-12182
> Project: Hive
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.1
>Reporter: Lenni Kuff
>Assignee: Naveen Gangam
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-12182-branch1.txt, HIVE-12182.2.patch,
> HIVE-12182.3.patch, HIVE-12182.patch
>
>
> ALTER TABLE PARTITION COLUMN does not set partition column comments. The
> syntax is accepted, but the COMMENT for the column is ignored.
> {code}
> 0: jdbc:hive2://localhost:1/default> create table part_test(i int comment
> 'HELLO') partitioned by (j int comment 'WORLD');
> No rows affected (0.104 seconds)
> 0: jdbc:hive2://localhost:1/default> describe part_test;
> +--+---+---+--+
> | col_name | data_type |comment|
> +--+---+---+--+
> | i| int | HELLO |
> | j| int | WORLD |
> | | NULL | NULL |
> | # Partition Information | NULL | NULL |
> | # col_name | data_type | comment |
> | | NULL | NULL |
> | j| int | WORLD |
> +--+---+---+--+
> 7 rows selected (0.109 seconds)
> 0: jdbc:hive2://localhost:1/default> alter table part_test partition
> column (j int comment 'WIDE');
> No rows affected (0.121 seconds)
> 0: jdbc:hive2://localhost:1/default> describe part_test;
> +--+---+---+--+
> | col_name | data_type |comment|
> +--+---+---+--+
> | i| int | HELLO |
> | j| int | |
> | | NULL | NULL |
> | # Partition Information | NULL | NULL |
> | # col_name | data_type | comment |
> | | NULL | NULL |
> | j| int | |
> +--+---+---+--+
> 7 rows selected (0.108 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12557) NPE while removing entry in LRFU cache
[
https://issues.apache.org/jira/browse/HIVE-12557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035039#comment-15035039
]
Prasanth Jayachandran commented on HIVE-12557:
--
Will be good to add test case. How does this fix the NPE?
> NPE while removing entry in LRFU cache
> --
>
> Key: HIVE-12557
> URL: https://issues.apache.org/jira/browse/HIVE-12557
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12557.patch
>
>
> {code}
> Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 6,
> vertexId=vertex_1448429572030_1851_5_00, diagnostics=[Task failed,
> taskId=task_1448429572030_1851_5_00_06, diagnostics=[TaskAttempt 0
> failed, info=[Error: Failure while running task:
> attempt_1448429572030_1851_5_00_06_0:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:348)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:74)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:352)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
> ... 14 more
> Caused by: java.io.IOException: java.io.IOException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
> ... 16 more
> Caused by: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:283)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:239)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:167)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:103)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
> ... 22 more
> Caused by:
[jira] [Commented] (HIVE-12469) Bump Commons-Collections dependency from 3.2.1 to 3.2.2. to address vulnerability
[
https://issues.apache.org/jira/browse/HIVE-12469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035041#comment-15035041
]
Hive QA commented on HIVE-12469:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12774385/HIVE-12469.2-branch1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 51 failed/errored test(s), 9206 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-groupby10.q-timestamp_comparison.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_cond_pushdown_unqual4.q-vectorization_16.q-union_remove_1.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-table_access_keys_stats.q-groupby_complex_types.q-vectorization_10.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-vector_distinct_2.q-load_dyn_part2.q-join35.q-and-12-more -
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_filters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avrocountemptytbl
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_rdd_cache
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_merge_multi_expressions
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_mixed_partition_formats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_temp_table_gb1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_fast_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_join_filters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_join_nulls
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_drop_partition
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_inner_join
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join_filters
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cross_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_groupby2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge10
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge11
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_empty
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union_fast_stats
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_join_filters
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_join_nulls
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_groupby2
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_stats_counter_partitioned
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_filters
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cross_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dynamic_rdd_cache
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_count_distinct
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-BRANCH_1-Build/6/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-BRANCH_1-Build/6/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-BRANCH_1-Build-6/
Messages:
{noformat}
[jira] [Commented] (HIVE-12557) NPE while removing entry in LRFU cache
[
https://issues.apache.org/jira/browse/HIVE-12557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035056#comment-15035056
]
Prasanth Jayachandran commented on HIVE-12557:
--
Will NPE be the only exception thrown. It will be good to catch any runtime
exceptions, debug log and rethrow. other than that +1
> NPE while removing entry in LRFU cache
> --
>
> Key: HIVE-12557
> URL: https://issues.apache.org/jira/browse/HIVE-12557
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12557.patch
>
>
> {code}
> Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 6,
> vertexId=vertex_1448429572030_1851_5_00, diagnostics=[Task failed,
> taskId=task_1448429572030_1851_5_00_06, diagnostics=[TaskAttempt 0
> failed, info=[Error: Failure while running task:
> attempt_1448429572030_1851_5_00_06_0:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:348)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:74)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:352)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
> ... 14 more
> Caused by: java.io.IOException: java.io.IOException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
> ... 16 more
> Caused by: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:283)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:239)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:167)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:103)
> at
>
[jira] [Commented] (HIVE-12554) Fix Spark branch build after merge [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035069#comment-15035069 ] Rui Li commented on HIVE-12554: --- Thanks Xuefu for taking care of this. > Fix Spark branch build after merge [Spark Branch] > - > > Key: HIVE-12554 > URL: https://issues.apache.org/jira/browse/HIVE-12554 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: spark-branch >Reporter: Xuefu Zhang >Assignee: Rui Li > Fix For: spark-branch > > Attachments: HIVE-12554.0-spark.patch > > > The previous merge from master broke spark branch build. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12491) Column Statistics: 3 attribute join on a 2-source table is off
[
https://issues.apache.org/jira/browse/HIVE-12491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-12491:
Attachment: HIVE-12491.3.patch
We can exploit semantic information about known udfs which have bounded NDVs.
> Column Statistics: 3 attribute join on a 2-source table is off
> --
>
> Key: HIVE-12491
> URL: https://issues.apache.org/jira/browse/HIVE-12491
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Gopal V
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-12491.2.patch, HIVE-12491.3.patch,
> HIVE-12491.WIP.patch, HIVE-12491.patch
>
>
> The eased out denominator has to detect duplicate row-stats from different
> attributes.
> {code}
> select account_id from customers c, customer_activation ca
> where c.customer_id = ca.customer_id
> and year(ca.dt) = year(c.dt) and month(ca.dt) = month(c.dt)
> and year(ca.dt) between year('2013-12-26') and year('2013-12-26')
> {code}
> {code}
> private Long getEasedOutDenominator(List distinctVals) {
> // Exponential back-off for NDVs.
> // 1) Descending order sort of NDVs
> // 2) denominator = NDV1 * (NDV2 ^ (1/2)) * (NDV3 ^ (1/4))) *
> Collections.sort(distinctVals, Collections.reverseOrder());
> long denom = distinctVals.get(0);
> for (int i = 1; i < distinctVals.size(); i++) {
> denom = (long) (denom * Math.pow(distinctVals.get(i), 1.0 / (1 <<
> i)));
> }
> return denom;
> }
> {code}
> This gets {{[8007986, 821974390, 821974390]}}, which is actually 3 columns 2
> of which are derived from the same column.
> {code}
> Reduce Output Operator (RS_12)
> key expressions: _col0 (type: bigint), year(_col2) (type: int),
> month(_col2) (type: int)
> sort order: +++
> Map-reduce partition columns: _col0 (type: bigint), year(_col2)
> (type: int), month(_col2) (type: int)
> value expressions: _col1 (type: bigint)
> Join Operator (JOIN_13)
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col0 (type: bigint), year(_col1) (type: int), month(_col1)
> (type: int)
> 1 _col0 (type: bigint), year(_col2) (type: int), month(_col2)
> (type: int)
> outputColumnNames: _col3
> {code}
> So the eased out denominator is off by a factor of 30,000 or so, causing OOMs
> in map-joins.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-12528) don't start HS2 Tez sessions in a single thread
[ https://issues.apache.org/jira/browse/HIVE-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-12528: --- Assignee: Sergey Shelukhin > don't start HS2 Tez sessions in a single thread > --- > > Key: HIVE-12528 > URL: https://issues.apache.org/jira/browse/HIVE-12528 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Starting sessions in parallel would improve the startup time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11371) Null pointer exception for nested table query when using ORC versus text
[ https://issues.apache.org/jira/browse/HIVE-11371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-11371: Attachment: HIVE-11371.03.patch Rebase and re-submit. > Null pointer exception for nested table query when using ORC versus text > > > Key: HIVE-11371 > URL: https://issues.apache.org/jira/browse/HIVE-11371 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 1.2.0 >Reporter: N Campbell >Assignee: Matt McCline > Attachments: HIVE-11371.01.patch, HIVE-11371.02.patch, > HIVE-11371.03.patch, TJOIN1, TJOIN2, TJOIN3, TJOIN4 > > > Following query will fail if the file format is ORC > select tj1rnum, tj2rnum, tjoin3.rnum as rnumt3 from (select tjoin1.rnum > tj1rnum, tjoin2.rnum tj2rnum, tjoin2.c1 tj2c1 from tjoin1 left outer join > tjoin2 on tjoin1.c1 = tjoin2.c1 ) tj left outer join tjoin3 on tj2c1 = > tjoin3.c1 > aused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.vector.VectorCopyRow$LongCopyRow.copy(VectorCopyRow.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.VectorCopyRow.copyByReference(VectorCopyRow.java:260) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:238) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinOuterGenerateResultOperator.finishOuter(VectorMapJoinOuterGenerateResultOperator.java:495) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinOuterLongOperator.process(VectorMapJoinOuterLongOperator.java:430) > ... 22 more > ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 > killedTasks:0, Vertex vertex_1437788144883_0004_2_02 [Map 1] killed/failed > due to:null]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 > killedVertices:0 > SQLState: 08S01 > ErrorCode: 2 > getDatabaseProductNameApache Hive > getDatabaseProductVersion 1.2.1.2.3.0.0-2557 > getDriverName Hive JDBC > getDriverVersion 1.2.1.2.3.0.0-2557 > getDriverMajorVersion 1 > getDriverMinorVersion 2 > create table if not exists TJOIN1 (RNUM int , C1 int, C2 int) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc; > create table if not exists TJOIN2 (RNUM int , C1 int, C2 char(2)) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; > create table if not exists TJOIN3 (RNUM int , C1 int, C2 char(2)) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; > create table if not exists TJOIN4 (RNUM int , C1 int, C2 char(2)) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11975) mssql scripts contains invalid 'GO' statement
[ https://issues.apache.org/jira/browse/HIVE-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-11975: Attachment: HIVE-11975.patch Patch attached. > mssql scripts contains invalid 'GO' statement > - > > Key: HIVE-11975 > URL: https://issues.apache.org/jira/browse/HIVE-11975 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.2.0, 1.1.0 >Reporter: Huan Huang >Assignee: Huan Huang > Attachments: HIVE-11975.patch > > > 006-HIVE-9456.mssql.sql script contains invalid 'GO' statement, caused hive > schematool to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-11975) mssql scripts contains invalid 'GO' statement
[ https://issues.apache.org/jira/browse/HIVE-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan reassigned HIVE-11975: --- Assignee: Sushanth Sowmyan (was: Huan Huang) > mssql scripts contains invalid 'GO' statement > - > > Key: HIVE-11975 > URL: https://issues.apache.org/jira/browse/HIVE-11975 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.2.0, 1.1.0 >Reporter: Huan Huang >Assignee: Sushanth Sowmyan > Attachments: HIVE-11975.patch > > > 006-HIVE-9456.mssql.sql script contains invalid 'GO' statement, caused hive > schematool to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12557) NPE while removing entry in LRFU cache
[
https://issues.apache.org/jira/browse/HIVE-12557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035047#comment-15035047
]
Sergey Shelukhin commented on HIVE-12557:
-
1) I don't actually know the precise cause - this seems to be a concurrency
issue where we try to remove the item from the list, and it's not the head
element yet doesn't have prev set.
2) I don't see any logical errors, but there's one place where the list is
operated on without taking the lock. So I suspect it causes the corruption.
3) It is impractical to add a test case, this is a concurrency issue.
> NPE while removing entry in LRFU cache
> --
>
> Key: HIVE-12557
> URL: https://issues.apache.org/jira/browse/HIVE-12557
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12557.patch
>
>
> {code}
> Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 6,
> vertexId=vertex_1448429572030_1851_5_00, diagnostics=[Task failed,
> taskId=task_1448429572030_1851_5_00_06, diagnostics=[TaskAttempt 0
> failed, info=[Error: Failure while running task:
> attempt_1448429572030_1851_5_00_06_0:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:348)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:74)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:352)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
> ... 14 more
> Caused by: java.io.IOException: java.io.IOException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
> ... 16 more
> Caused by: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:283)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:239)
> at
>
[jira] [Commented] (HIVE-12257) Enhance ORC FileDump utility to handle flush_length files and recovery
[
https://issues.apache.org/jira/browse/HIVE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035058#comment-15035058
]
Hive QA commented on HIVE-12257:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12774985/HIVE-12257.10.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6189/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6189/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6189/
Messages:
{noformat}
This message was trimmed, see log for full details
[loading
ZipFileIndexFileObject[/data/hive-ptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/2.6.0/hadoop-mapreduce-client-core-2.6.0.jar(org/apache/hadoop/mapreduce/lib/output/NullOutputFormat.class)]]
[loading
ZipFileIndexFileObject[/data/hive-ptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/2.6.0/hadoop-mapreduce-client-core-2.6.0.jar(org/apache/hadoop/mapreduce/security/token/delegation/DelegationTokenIdentifier.class)]]
[loading
ZipFileIndexFileObject[/data/hive-ptest/working/maven/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0.jar(org/apache/hadoop/util/Tool.class)]]
[loading
ZipFileIndexFileObject[/data/hive-ptest/working/maven/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0.jar(org/apache/hadoop/conf/Configurable.class)]]
[loading
ZipFileIndexFileObject[/usr/java/jdk1.7.0_45-cloudera/lib/ct.sym(META-INF/sym/rt.jar/java/lang/ClassNotFoundException.class)]]
[loading
ZipFileIndexFileObject[/data/hive-ptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/2.6.0/hadoop-mapreduce-client-core-2.6.0.jar(org/apache/hadoop/mapreduce/Mapper.class)]]
[loading
ZipFileIndexFileObject[/usr/java/jdk1.7.0_45-cloudera/lib/ct.sym(META-INF/sym/rt.jar/java/util/LinkedList.class)]]
[loading
ZipFileIndexFileObject[/usr/java/jdk1.7.0_45-cloudera/lib/ct.sym(META-INF/sym/rt.jar/java/util/concurrent/ExecutorService.class)]]
[loading
ZipFileIndexFileObject[/usr/java/jdk1.7.0_45-cloudera/lib/ct.sym(META-INF/sym/rt.jar/java/util/concurrent/Executors.class)]]
[loading
ZipFileIndexFileObject[/usr/java/jdk1.7.0_45-cloudera/lib/ct.sym(META-INF/sym/rt.jar/java/util/concurrent/TimeUnit.class)]]
[loading
ZipFileIndexFileObject[/usr/java/jdk1.7.0_45-cloudera/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Process.class)]]
[loading
ZipFileIndexFileObject[/data/hive-ptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/2.6.0/hadoop-mapreduce-client-core-2.6.0.jar(org/apache/hadoop/mapreduce/Mapper$Context.class)]]
[loading
ZipFileIndexFileObject[/usr/java/jdk1.7.0_45-cloudera/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/Target.class)]]
[loading
ZipFileIndexFileObject[/usr/java/jdk1.7.0_45-cloudera/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/ElementType.class)]]
[loading
ZipFileIndexFileObject[/usr/java/jdk1.7.0_45-cloudera/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/Retention.class)]]
[loading
ZipFileIndexFileObject[/usr/java/jdk1.7.0_45-cloudera/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/RetentionPolicy.class)]]
[loading
ZipFileIndexFileObject[/usr/java/jdk1.7.0_45-cloudera/lib/ct.sym(META-INF/sym/rt.jar/java/lang/annotation/Annotation.class)]]
[loading
ZipFileIndexFileObject[/data/hive-ptest/working/maven/org/apache/hadoop/hadoop-annotations/2.6.0/hadoop-annotations-2.6.0.jar(org/apache/hadoop/classification/InterfaceAudience.class)]]
[loading
ZipFileIndexFileObject[/data/hive-ptest/working/maven/org/apache/hadoop/hadoop-annotations/2.6.0/hadoop-annotations-2.6.0.jar(org/apache/hadoop/classification/InterfaceAudience$LimitedPrivate.class)]]
[loading
ZipFileIndexFileObject[/usr/java/jdk1.7.0_45-cloudera/lib/ct.sym(META-INF/sym/rt.jar/java/lang/SuppressWarnings.class)]]
[loading
ZipFileIndexFileObject[/usr/java/jdk1.7.0_45-cloudera/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Override.class)]]
[loading
ZipFileIndexFileObject[/data/hive-ptest/working/maven/com/sun/jersey/jersey-core/1.14/jersey-core-1.14.jar(javax/ws/rs/HttpMethod.class)]]
[loading
RegularFileObject[/data/hive-ptest/working/apache-github-source-source/hcatalog/webhcat/svr/target/classes/org/apache/hive/hcatalog/templeton/AppConfig$1.class]]
[loading
RegularFileObject[/data/hive-ptest/working/apache-github-source-source/hcatalog/webhcat/svr/target/classes/org/apache/hive/hcatalog/templeton/SecureProxySupport$3.class]]
[loading
RegularFileObject[/data/hive-ptest/working/apache-github-source-source/hcatalog/webhcat/svr/target/classes/org/apache/hive/hcatalog/templeton/SecureProxySupport$1.class]]
[loading
[jira] [Commented] (HIVE-12557) NPE while removing entry in LRFU cache
[
https://issues.apache.org/jira/browse/HIVE-12557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035068#comment-15035068
]
Sergey Shelukhin commented on HIVE-12557:
-
Hmm. debug log would be pretty useless :) It's an actual error so I error log.
It actually means LRFU state is corrupted so I was wondering if I should even
terminate the daemon.
And yeah the only thing it does there is bunch of assignments so only NPE is
expected. I guess I could add some preconditions too...
> NPE while removing entry in LRFU cache
> --
>
> Key: HIVE-12557
> URL: https://issues.apache.org/jira/browse/HIVE-12557
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12557.patch
>
>
> {code}
> Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 6,
> vertexId=vertex_1448429572030_1851_5_00, diagnostics=[Task failed,
> taskId=task_1448429572030_1851_5_00_06, diagnostics=[TaskAttempt 0
> failed, info=[Error: Failure while running task:
> attempt_1448429572030_1851_5_00_06_0:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:348)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:74)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:352)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
> ... 14 more
> Caused by: java.io.IOException: java.io.IOException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
> ... 16 more
> Caused by: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:283)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:239)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:167)
> at
>
[jira] [Commented] (HIVE-12557) NPE while removing entry in LRFU cache
[
https://issues.apache.org/jira/browse/HIVE-12557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035076#comment-15035076
]
Prasanth Jayachandran commented on HIVE-12557:
--
Somehow precondition didn't hit my head. It will be better to have
preconditions check instead of defensively catching NPE.
> NPE while removing entry in LRFU cache
> --
>
> Key: HIVE-12557
> URL: https://issues.apache.org/jira/browse/HIVE-12557
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12557.patch
>
>
> {code}
> Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 6,
> vertexId=vertex_1448429572030_1851_5_00, diagnostics=[Task failed,
> taskId=task_1448429572030_1851_5_00_06, diagnostics=[TaskAttempt 0
> failed, info=[Error: Failure while running task:
> attempt_1448429572030_1851_5_00_06_0:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:348)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:74)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:352)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
> ... 14 more
> Caused by: java.io.IOException: java.io.IOException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
> ... 16 more
> Caused by: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:283)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:239)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:167)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:103)
> at
>
[jira] [Updated] (HIVE-11372) join with between predicate comparing integer types returns no rows when ORC format used
[ https://issues.apache.org/jira/browse/HIVE-11372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-11372: -- Assignee: Matt McCline > join with between predicate comparing integer types returns no rows when ORC > format used > > > Key: HIVE-11372 > URL: https://issues.apache.org/jira/browse/HIVE-11372 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: N Campbell >Assignee: Matt McCline > Attachments: TINT, TSINT > > > getDatabaseProductNameApache Hive > getDatabaseProductVersion 1.2.1.2.3.0.0-2557 > getDriverName Hive JDBC > getDriverVersion 1.2.1.2.3.0.0-2557 > getDriverMajorVersion 1 > getDriverMinorVersion 2 > select tint.rnum, tsint.rnum from tint , tsint where tint.cint between > tsint.csint and tsint.csint > when ORC used no rows returned versus TEXT > create table if not exists TSINT ( RNUM int , CSINT smallint ) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; > create table if not exists TINT ( RNUM int , CINT int ) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11372) join with between predicate comparing integer types returns no rows when ORC format used
[ https://issues.apache.org/jira/browse/HIVE-11372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-11372: -- Summary: join with between predicate comparing integer types returns no rows when ORC format used (was: join with betwee predicate comparing integer types returns no rows when ORC format used) > join with between predicate comparing integer types returns no rows when ORC > format used > > > Key: HIVE-11372 > URL: https://issues.apache.org/jira/browse/HIVE-11372 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: N Campbell > Attachments: TINT, TSINT > > > getDatabaseProductNameApache Hive > getDatabaseProductVersion 1.2.1.2.3.0.0-2557 > getDriverName Hive JDBC > getDriverVersion 1.2.1.2.3.0.0-2557 > getDriverMajorVersion 1 > getDriverMinorVersion 2 > select tint.rnum, tsint.rnum from tint , tsint where tint.cint between > tsint.csint and tsint.csint > when ORC used no rows returned versus TEXT > create table if not exists TSINT ( RNUM int , CSINT smallint ) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; > create table if not exists TINT ( RNUM int , CINT int ) > -- ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS orc ; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12236) Enable SimpleFetchOptimizer for more query types
[ https://issues.apache.org/jira/browse/HIVE-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12236: Attachment: HIVE-12236.2.patch With tests. > Enable SimpleFetchOptimizer for more query types > > > Key: HIVE-12236 > URL: https://issues.apache.org/jira/browse/HIVE-12236 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 1.3.0, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-12236.2.patch, HIVE-12236.WIP.patch, > HIVE-12236.patch > > > Currently, metadata queries fired by BI tools tend to thrash LLAP's cache. > Bypass the cache and process metadata queries directly from HiveServer2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12528) don't start HS2 Tez sessions in a single thread
[ https://issues.apache.org/jira/browse/HIVE-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12528: Attachment: HIVE-12528.patch [~sseth] [~hagleitn] can you take a look? Note that this will start multiple tez apps in parallel (using different TezClient-s). > don't start HS2 Tez sessions in a single thread > --- > > Key: HIVE-12528 > URL: https://issues.apache.org/jira/browse/HIVE-12528 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12528.patch > > > Starting sessions in parallel would improve the startup time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12561) Add sentry jars to HS2 & HMS classpath
[ https://issues.apache.org/jira/browse/HIVE-12561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-12561: --- Attachment: HIVE-12561.patch > Add sentry jars to HS2 & HMS classpath > -- > > Key: HIVE-12561 > URL: https://issues.apache.org/jira/browse/HIVE-12561 > Project: Hive > Issue Type: Improvement >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal >Priority: Minor > Attachments: HIVE-12561.patch > > > If SENTRY_HOME is either set or can be detected at conventional location, add > relevant jars to HS2 and HMS classpath. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11241) Database prefix does not work properly if table has same name
[
https://issues.apache.org/jira/browse/HIVE-11241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035014#comment-15035014
]
Ferdinand Xu commented on HIVE-11241:
-
I have resolved it as duplicated. Thank you for pointing it out.
> Database prefix does not work properly if table has same name
> -
>
> Key: HIVE-11241
> URL: https://issues.apache.org/jira/browse/HIVE-11241
> Project: Hive
> Issue Type: Bug
> Components: Database/Schema
>Reporter: Johndee Burks
>Assignee: Ferdinand Xu
> Fix For: 2.0.0
>
> Attachments: HIVE-11241.patch
>
>
> If you do the following it will fail:
> {code}
> 0: jdbc:hive2://cdh54-1.test.com:1/defaul> create database test4;
> No rows affected (0.881 seconds)
> 0: jdbc:hive2://cdh54-1.test.com:1/defaul> use test4;
> No rows affected (0.1 seconds)
> 0: jdbc:hive2://cdh54-1.test.com:1/defaul> create table test4 (c1
> char(200));
> No rows affected (0.306 seconds)
> 0: jdbc:hive2://cdh54-1.test.com:1/defaul> desc test4.test4;
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. cannot find field test4 from
> [0:c1] (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12557) NPE while removing entry in LRFU cache
[
https://issues.apache.org/jira/browse/HIVE-12557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-12557:
Attachment: HIVE-12557.patch
Most of the patch is debug logging for potential future NPEs.
As for the bug, there's a spot where we use a synchronized block on re-entrant
lock instead of actually locking it. That might cause some problems 0_o
> NPE while removing entry in LRFU cache
> --
>
> Key: HIVE-12557
> URL: https://issues.apache.org/jira/browse/HIVE-12557
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12557.patch
>
>
> {code}
> Error while processing statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 6,
> vertexId=vertex_1448429572030_1851_5_00, diagnostics=[Task failed,
> taskId=task_1448429572030_1851_5_00_06, diagnostics=[TaskAttempt 0
> failed, info=[Error: Failure while running task:
> attempt_1448429572030_1851_5_00_06_0:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:348)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:74)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:352)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
> ... 14 more
> Caused by: java.io.IOException: java.io.IOException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
> ... 16 more
> Caused by: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:283)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:239)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:167)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:103)
> at
>
[jira] [Updated] (HIVE-11975) mssql scripts contains invalid 'GO' statement
[ https://issues.apache.org/jira/browse/HIVE-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-11975: Description: 006-HIVE-9456.mssql.sql script contains invalid 'GO' statement, caused hive schematool to fail. NO PRECOMMIT TESTS (since this does not change hive code, and affects only mssql, which pretests do not run against) was:006-HIVE-9456.mssql.sql script contains invalid 'GO' statement, caused hive schematool to fail. > mssql scripts contains invalid 'GO' statement > - > > Key: HIVE-11975 > URL: https://issues.apache.org/jira/browse/HIVE-11975 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.0, 1.3.0, 2.0.0 >Reporter: Huan Huang >Assignee: Sushanth Sowmyan > Attachments: HIVE-11975.patch > > > 006-HIVE-9456.mssql.sql script contains invalid 'GO' statement, caused hive > schematool to fail. > NO PRECOMMIT TESTS (since this does not change hive code, and affects only > mssql, which pretests do not run against) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11975) mssql scripts contains invalid 'GO' statement
[ https://issues.apache.org/jira/browse/HIVE-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035086#comment-15035086 ] Thejas M Nair commented on HIVE-11975: -- +1 > mssql scripts contains invalid 'GO' statement > - > > Key: HIVE-11975 > URL: https://issues.apache.org/jira/browse/HIVE-11975 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.0, 1.3.0, 2.0.0 >Reporter: Huan Huang >Assignee: Sushanth Sowmyan > Attachments: HIVE-11975.patch > > > 006-HIVE-9456.mssql.sql script contains invalid 'GO' statement, caused hive > schematool to fail. > NO PRECOMMIT TESTS (since this does not change hive code, and affects only > mssql, which pretests do not run against) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12236) Enable SimpleFetchOptimizer for more query types
[ https://issues.apache.org/jira/browse/HIVE-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12236: Summary: Enable SimpleFetchOptimizer for more query types (was: LLAP: Prevent metadata queries from thrashing LLAP cache) > Enable SimpleFetchOptimizer for more query types > > > Key: HIVE-12236 > URL: https://issues.apache.org/jira/browse/HIVE-12236 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 1.3.0, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-12236.WIP.patch, HIVE-12236.patch > > > Currently, metadata queries fired by BI tools tend to thrash LLAP's cache. > Bypass the cache and process metadata queries directly from HiveServer2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12445) Tracking of completed dags is a slow memory leak
[ https://issues.apache.org/jira/browse/HIVE-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035197#comment-15035197 ] Sergey Shelukhin commented on HIVE-12445: - [~sseth] ping? > Tracking of completed dags is a slow memory leak > > > Key: HIVE-12445 > URL: https://issues.apache.org/jira/browse/HIVE-12445 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.0.0 >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Attachments: HIVE-12445.patch > > > LLAP daemons track completed DAGs, but never clean up these structures. This > is primarily to disallow out of order executions. Evaluate whether that can > be avoided - otherwise this structure needs to be cleaned up with a delay. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12532) LLAP Cache: Uncompressed data cache has NPE
[
https://issues.apache.org/jira/browse/HIVE-12532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-12532:
Attachment: HIVE-12532.01.patch
The same patch for HiveQA.
> LLAP Cache: Uncompressed data cache has NPE
> ---
>
> Key: HIVE-12532
> URL: https://issues.apache.org/jira/browse/HIVE-12532
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.0.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12435.patch, HIVE-12532.01.patch
>
>
> {code}
> 2015-11-26 08:28:45,232
> [TezTaskRunner_attempt_1448429572030_0255_2_02_19_2(attempt_1448429572030_0255_2_02_19_2)]
> WARN org.apache.tez.runtime.LogicalIOProcessorRuntimeTask: Ignoring
> exception when closing input a(cleanup). Exception class=java.io.IOException,
> message=java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:283)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.close(LlapInputFormat.java:275)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doClose(HiveRecordReader.java:50)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.close(HiveContextAwareRecordReader.java:104)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.close(TezGroupedSplitsInputFormat.java:177)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.close(MRReaderMapred.java:96)
> at org.apache.tez.mapreduce.input.MRInput.close(MRInput.java:559)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.cleanup(LogicalIOProcessorRuntimeTask.java:872)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:104)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.preReadUncompressedStream(EncodedReaderImpl.java:795)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:320)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:413)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:194)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:191)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:191)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:74)
> ... 5 more
> {code}
> Not clear if current.next can set it to null before the continue;
> {code}
> assert partOffset <= current.getOffset();
> if (partOffset == current.getOffset() && current instanceof CacheChunk)
> {
> // We assume cache chunks would always match the way we read, so
> check and skip it.
> assert current.getOffset() == partOffset && current.getEnd() ==
> partEnd;
> lastUncompressed = (CacheChunk)current;
> current = current.next;
> continue;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HIVE-12338) Add webui to HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-12338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang reopened HIVE-12338: > Add webui to HiveServer2 > > > Key: HIVE-12338 > URL: https://issues.apache.org/jira/browse/HIVE-12338 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12338.1.patch, HIVE-12338.2.patch, > HIVE-12338.3.patch, HIVE-12338.4.patch, hs2-conf.png, hs2-logs.png, > hs2-metrics.png, hs2-webui.png > > > A web ui for HiveServer2 can show some useful information such as: > > 1. Sessions, > 2. Queries that are executing on the HS2, their states, starting time, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12338) Add webui to HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-12338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034073#comment-15034073 ] Jimmy Xiang commented on HIVE-12338: Sure. Just re-opened it. Let's keep it open till all sub-tasks are resolved. > Add webui to HiveServer2 > > > Key: HIVE-12338 > URL: https://issues.apache.org/jira/browse/HIVE-12338 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12338.1.patch, HIVE-12338.2.patch, > HIVE-12338.3.patch, HIVE-12338.4.patch, hs2-conf.png, hs2-logs.png, > hs2-metrics.png, hs2-webui.png > > > A web ui for HiveServer2 can show some useful information such as: > > 1. Sessions, > 2. Queries that are executing on the HS2, their states, starting time, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12483) Fix precommit Spark test branch
[ https://issues.apache.org/jira/browse/HIVE-12483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034105#comment-15034105 ] Xuefu Zhang commented on HIVE-12483: Yeah. it's a little tricky. I did a merge last week, and in the first test run after the merge in HIVE-12515 some of these failures went away. Right now we are fixing the build (HIVE-12554), and we should have another test run coming out soon. At that time, we can probably confirm what the issue is gone. Thanks for investigating this. > Fix precommit Spark test branch > --- > > Key: HIVE-12483 > URL: https://issues.apache.org/jira/browse/HIVE-12483 > Project: Hive > Issue Type: Task >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-12483.1-spark.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12381) analyze table compute stats for table with special characters will wipe out all the table stats
[
https://issues.apache.org/jira/browse/HIVE-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034033#comment-15034033
]
Ashutosh Chauhan commented on HIVE-12381:
-
+1
> analyze table compute stats for table with special characters will wipe out
> all the table stats
> ---
>
> Key: HIVE-12381
> URL: https://issues.apache.org/jira/browse/HIVE-12381
> Project: Hive
> Issue Type: Bug
> Environment:
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-12381.01.patch, HIVE-12381.02.patch,
> HIVE-12381.03.patch, HIVE-12381.04.patch, HIVE-12381.05.patch
>
>
> repo:
> {code}
> drop table `t//`;
> create table `t//` (col string);
> insert into `t//` values(1);
> insert into `t//` values(null);
> analyze table `t//` compute statistics;
> explain select * from `t//`;
> {code}
> The result
> {code}
> Statistics: Num rows: 1 Data size: 5 Basic stats: COMPLETE Column stats: NONE
> {code}
> is wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12529) HiveTxnManager.acquireLocks() should not block forever
[ https://issues.apache.org/jira/browse/HIVE-12529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-12529: -- Attachment: HIVE-12529.4.patch [~alangates] could you review please > HiveTxnManager.acquireLocks() should not block forever > -- > > Key: HIVE-12529 > URL: https://issues.apache.org/jira/browse/HIVE-12529 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-12529.2.patch, HIVE-12529.4.patch, HIVE-12529.patch > > > Currently, in DbTxnManager this method will block until all competing locks > have gone away. > This is not appropriate for all clients. There should be a way to specify a > max-wait-time. > It will throw an exception on timeout (given how current method signature is > written). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12483) Fix precommit Spark test branch
[ https://issues.apache.org/jira/browse/HIVE-12483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034108#comment-15034108 ] Sergio Peña commented on HIVE-12483: Thanks. I will set this task as resolved. > Fix precommit Spark test branch > --- > > Key: HIVE-12483 > URL: https://issues.apache.org/jira/browse/HIVE-12483 > Project: Hive > Issue Type: Task >Reporter: Sergio Peña >Assignee: Sergio Peña > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12483) Fix precommit Spark test branch
[ https://issues.apache.org/jira/browse/HIVE-12483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-12483: --- Attachment: (was: HIVE-12483.1-spark.patch) > Fix precommit Spark test branch > --- > > Key: HIVE-12483 > URL: https://issues.apache.org/jira/browse/HIVE-12483 > Project: Hive > Issue Type: Task >Reporter: Sergio Peña >Assignee: Sergio Peña > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12554) Fix Spark branch build after merge [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034176#comment-15034176
]
Hive QA commented on HIVE-12554:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12775038/HIVE-12554.0-spark.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9866 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/1016/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/1016/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-1016/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12775038 - PreCommit-HIVE-SPARK-Build
> Fix Spark branch build after merge [Spark Branch]
> -
>
> Key: HIVE-12554
> URL: https://issues.apache.org/jira/browse/HIVE-12554
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: spark-branch
>Reporter: Xuefu Zhang
>Assignee: Rui Li
> Attachments: HIVE-12554.0-spark.patch
>
>
> The previous merge from master broke spark branch build.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12020) Revert log4j2 xml configuration to properties based configuration
[
https://issues.apache.org/jira/browse/HIVE-12020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034239#comment-15034239
]
Hive QA commented on HIVE-12020:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12774983/HIVE-12020.4.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 9854 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-script_pipe.q-tez_smb_empty.q-mapjoin_decimal.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6187/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6187/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6187/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12774983 - PreCommit-HIVE-TRUNK-Build
> Revert log4j2 xml configuration to properties based configuration
> -
>
> Key: HIVE-12020
> URL: https://issues.apache.org/jira/browse/HIVE-12020
> Project: Hive
> Issue Type: Sub-task
> Components: Logging
>Affects Versions: 2.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12020.1.patch, HIVE-12020.2.patch,
> HIVE-12020.3.patch, HIVE-12020.4.patch
>
>
> Log4j 2.4 release brought back properties based configuration. We should
> revert XML based configuration and use properties based configuration instead
> (less verbose and will be similar to old log4j properties).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12491) Column Statistics: 3 attribute join on a 2-source table is off
[
https://issues.apache.org/jira/browse/HIVE-12491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-12491:
Attachment: HIVE-12491.5.patch
> Column Statistics: 3 attribute join on a 2-source table is off
> --
>
> Key: HIVE-12491
> URL: https://issues.apache.org/jira/browse/HIVE-12491
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Gopal V
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-12491.2.patch, HIVE-12491.3.patch,
> HIVE-12491.4.patch, HIVE-12491.5.patch, HIVE-12491.WIP.patch, HIVE-12491.patch
>
>
> The eased out denominator has to detect duplicate row-stats from different
> attributes.
> {code}
> select account_id from customers c, customer_activation ca
> where c.customer_id = ca.customer_id
> and year(ca.dt) = year(c.dt) and month(ca.dt) = month(c.dt)
> and year(ca.dt) between year('2013-12-26') and year('2013-12-26')
> {code}
> {code}
> private Long getEasedOutDenominator(List distinctVals) {
> // Exponential back-off for NDVs.
> // 1) Descending order sort of NDVs
> // 2) denominator = NDV1 * (NDV2 ^ (1/2)) * (NDV3 ^ (1/4))) *
> Collections.sort(distinctVals, Collections.reverseOrder());
> long denom = distinctVals.get(0);
> for (int i = 1; i < distinctVals.size(); i++) {
> denom = (long) (denom * Math.pow(distinctVals.get(i), 1.0 / (1 <<
> i)));
> }
> return denom;
> }
> {code}
> This gets {{[8007986, 821974390, 821974390]}}, which is actually 3 columns 2
> of which are derived from the same column.
> {code}
> Reduce Output Operator (RS_12)
> key expressions: _col0 (type: bigint), year(_col2) (type: int),
> month(_col2) (type: int)
> sort order: +++
> Map-reduce partition columns: _col0 (type: bigint), year(_col2)
> (type: int), month(_col2) (type: int)
> value expressions: _col1 (type: bigint)
> Join Operator (JOIN_13)
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col0 (type: bigint), year(_col1) (type: int), month(_col1)
> (type: int)
> 1 _col0 (type: bigint), year(_col2) (type: int), month(_col2)
> (type: int)
> outputColumnNames: _col3
> {code}
> So the eased out denominator is off by a factor of 30,000 or so, causing OOMs
> in map-joins.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12529) HiveTxnManager.acquireLocks() should not block forever
[
https://issues.apache.org/jira/browse/HIVE-12529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035405#comment-15035405
]
Hive QA commented on HIVE-12529:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12775077/HIVE-12529.4.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9870 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6193/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6193/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6193/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12775077 - PreCommit-HIVE-TRUNK-Build
> HiveTxnManager.acquireLocks() should not block forever
> --
>
> Key: HIVE-12529
> URL: https://issues.apache.org/jira/browse/HIVE-12529
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-12529.2.patch, HIVE-12529.4.patch, HIVE-12529.patch
>
>
> Currently, in DbTxnManager this method will block until all competing locks
> have gone away.
> This is not appropriate for all clients. There should be a way to specify a
> max-wait-time.
> It will throw an exception on timeout (given how current method signature is
> written).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12538) After set spark related config, SparkSession never get reused
[ https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-12538: - Attachment: HIVE-12538.3.patch Using ConcurrentHashMap instead of HashMap. > After set spark related config, SparkSession never get reused > - > > Key: HIVE-12538 > URL: https://issues.apache.org/jira/browse/HIVE-12538 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.3.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-12538.1.patch, HIVE-12538.2.patch, > HIVE-12538.3.patch, HIVE-12538.patch > > > Hive on Spark yarn-cluster mode. > After setting "set spark.yarn.queue=QueueA;" , > run the query "select count(*) from test" 3 times and you will find 3 > different yarn applications. > Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING > & UNDEFINED state waiting for next work. > And if you submit one more "select count(*) from test" ,the third one will be > in FINISHED & SUCCEEDED state and a new yarn application will start up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12475) Parquet schema evolution within array<struct<>> doesn't work
[
https://issues.apache.org/jira/browse/HIVE-12475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035327#comment-15035327
]
Hive QA commented on HIVE-12475:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12775067/HIVE-12475.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9869 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6192/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6192/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6192/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12775067 - PreCommit-HIVE-TRUNK-Build
> Parquet schema evolution within array> doesn't work
>
>
> Key: HIVE-12475
> URL: https://issues.apache.org/jira/browse/HIVE-12475
> Project: Hive
> Issue Type: Bug
> Components: File Formats
>Affects Versions: 1.1.0
>Reporter: Mohammad Kamrul Islam
>Assignee: Mohammad Kamrul Islam
> Attachments: HIVE-12475.1.patch
>
>
> If we create a table with type array>, and later added a field in
> the struct, we got the following exception.
> The following SQL statements would recreate the error:
> {quote}
> CREATE TABLE pq_test (f1 array>) STORED AS PARQUET;
> INSERT INTO TABLE pq_test select array(named_struct("c1",1,"c2",2)) FROM tmp
> LIMIT 2;
> SELECT * from pq_test;
> ALTER TABLE pq_test REPLACE COLUMNS (f1
> array>); //* cc
> SELECT * from pq_test;
> {quote}
> Exception:
> {quote}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
> at
> org.apache.hadoop.hive.ql.io.parquet.serde.ArrayWritableObjectInspector.getStructFieldData(ArrayWritableObjectInspector.java:142)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:363)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:316)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:199)
> at
> org.apache.hadoop.hive.serde2.DelimitedJSONSerDe.serializeField(DelimitedJSONSerDe.java:61)
> at
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.doSerialize(LazySimpleSerDe.java:236)
> at
> org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.serialize(AbstractEncodingAwareSerDe.java:55)
> at
> org.apache.hadoop.hive.ql.exec.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:71)
> at
> org.apache.hadoop.hive.ql.exec.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:40)
> at
> org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:89)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12538) After set spark related config, SparkSession never get reused
[ https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-12538: - Attachment: HIVE-12538.2.patch Addressing issues mentioned in comments. > After set spark related config, SparkSession never get reused > - > > Key: HIVE-12538 > URL: https://issues.apache.org/jira/browse/HIVE-12538 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.3.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-12538.1.patch, HIVE-12538.2.patch, HIVE-12538.patch > > > Hive on Spark yarn-cluster mode. > After setting "set spark.yarn.queue=QueueA;" , > run the query "select count(*) from test" 3 times and you will find 3 > different yarn applications. > Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING > & UNDEFINED state waiting for next work. > And if you submit one more "select count(*) from test" ,the third one will be > in FINISHED & SUCCEEDED state and a new yarn application will start up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12338) Add webui to HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-12338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035424#comment-15035424 ] Lefty Leverenz commented on HIVE-12338: --- Do you expect the sub-tasks to be resolved in time to include this in the 2.0.0 release? (If not, let's remove the TODOC2.0 label and Fix Version.) > Add webui to HiveServer2 > > > Key: HIVE-12338 > URL: https://issues.apache.org/jira/browse/HIVE-12338 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12338.1.patch, HIVE-12338.2.patch, > HIVE-12338.3.patch, HIVE-12338.4.patch, hs2-conf.png, hs2-logs.png, > hs2-metrics.png, hs2-webui.png > > > A web ui for HiveServer2 can show some useful information such as: > > 1. Sessions, > 2. Queries that are executing on the HS2, their states, starting time, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12491) Column Statistics: 3 attribute join on a 2-source table is off
[
https://issues.apache.org/jira/browse/HIVE-12491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-12491:
Attachment: HIVE-12491.4.patch
> Column Statistics: 3 attribute join on a 2-source table is off
> --
>
> Key: HIVE-12491
> URL: https://issues.apache.org/jira/browse/HIVE-12491
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Gopal V
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-12491.2.patch, HIVE-12491.3.patch,
> HIVE-12491.4.patch, HIVE-12491.WIP.patch, HIVE-12491.patch
>
>
> The eased out denominator has to detect duplicate row-stats from different
> attributes.
> {code}
> select account_id from customers c, customer_activation ca
> where c.customer_id = ca.customer_id
> and year(ca.dt) = year(c.dt) and month(ca.dt) = month(c.dt)
> and year(ca.dt) between year('2013-12-26') and year('2013-12-26')
> {code}
> {code}
> private Long getEasedOutDenominator(List distinctVals) {
> // Exponential back-off for NDVs.
> // 1) Descending order sort of NDVs
> // 2) denominator = NDV1 * (NDV2 ^ (1/2)) * (NDV3 ^ (1/4))) *
> Collections.sort(distinctVals, Collections.reverseOrder());
> long denom = distinctVals.get(0);
> for (int i = 1; i < distinctVals.size(); i++) {
> denom = (long) (denom * Math.pow(distinctVals.get(i), 1.0 / (1 <<
> i)));
> }
> return denom;
> }
> {code}
> This gets {{[8007986, 821974390, 821974390]}}, which is actually 3 columns 2
> of which are derived from the same column.
> {code}
> Reduce Output Operator (RS_12)
> key expressions: _col0 (type: bigint), year(_col2) (type: int),
> month(_col2) (type: int)
> sort order: +++
> Map-reduce partition columns: _col0 (type: bigint), year(_col2)
> (type: int), month(_col2) (type: int)
> value expressions: _col1 (type: bigint)
> Join Operator (JOIN_13)
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col0 (type: bigint), year(_col1) (type: int), month(_col1)
> (type: int)
> 1 _col0 (type: bigint), year(_col2) (type: int), month(_col2)
> (type: int)
> outputColumnNames: _col3
> {code}
> So the eased out denominator is off by a factor of 30,000 or so, causing OOMs
> in map-joins.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11775) Implement limit push down through union all in CBO
[
https://issues.apache.org/jira/browse/HIVE-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035245#comment-15035245
]
Hive QA commented on HIVE-11775:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12775064/HIVE-11775.06.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9870 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6191/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6191/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6191/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12775064 - PreCommit-HIVE-TRUNK-Build
> Implement limit push down through union all in CBO
> --
>
> Key: HIVE-11775
> URL: https://issues.apache.org/jira/browse/HIVE-11775
> Project: Hive
> Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-11775.01.patch, HIVE-11775.02.patch,
> HIVE-11775.03.patch, HIVE-11775.04.patch, HIVE-11775.05.patch,
> HIVE-11775.06.patch
>
>
> Enlightened by HIVE-11684 (Kudos to [~jcamachorodriguez]), we can actually
> push limit down through union all, which reduces the intermediate number of
> rows in union branches.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11358: Attachment: HIVE-11358.05.patch > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch, > HIVE-11358.03.patch, HIVE-11358.04.patch, HIVE-11358.05.patch, > HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11358: Attachment: HIVE-11358.04.patch Addressed the latest feedback... > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch, > HIVE-11358.03.patch, HIVE-11358.04.patch, HIVE-11358.05.patch, > HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11358: Attachment: (was: HIVE-11358.04.patch) > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.02.patch, > HIVE-11358.03.patch, HIVE-11358.04.patch, HIVE-11358.05.patch, > HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11531) Add mysql-style LIMIT support to Hive, or improve ROW_NUMBER performance-wise
[ https://issues.apache.org/jira/browse/HIVE-11531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035243#comment-15035243 ] Hui Zheng commented on HIVE-11531: -- Yes, I have correct it in HIVE-11531.03.patch . But it seems it will take so long time to be tested by Hive QA. > Add mysql-style LIMIT support to Hive, or improve ROW_NUMBER performance-wise > - > > Key: HIVE-11531 > URL: https://issues.apache.org/jira/browse/HIVE-11531 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Sergey Shelukhin >Assignee: Hui Zheng > Attachments: HIVE-11531.02.patch, HIVE-11531.03.patch, > HIVE-11531.WIP.1.patch, HIVE-11531.WIP.2.patch, HIVE-11531.patch > > > For any UIs that involve pagination, it is useful to issue queries in the > form SELECT ... LIMIT X,Y where X,Y are coordinates inside the result to be > paginated (which can be extremely large by itself). At present, ROW_NUMBER > can be used to achieve this effect, but optimizations for LIMIT such as TopN > in ReduceSink do not apply to ROW_NUMBER. We can add first class support for > "skip" to existing limit, or improve ROW_NUMBER for better performance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12471) Secure HS2 web UI with SSL
[ https://issues.apache.org/jira/browse/HIVE-12471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-12471: --- Attachment: HIVE-12471.1.patch This patch changed the HttpServer construction to use builder which is flexible. It addes 3 configurations for web UI SSL support. By default, it is disabled. Tested web UI with and without SSL, both work as expected. > Secure HS2 web UI with SSL > -- > > Key: HIVE-12471 > URL: https://issues.apache.org/jira/browse/HIVE-12471 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: HIVE-12471.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11312) ORC format: where clause with CHAR data type not returning any rows
[
https://issues.apache.org/jira/browse/HIVE-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034373#comment-15034373
]
Prasanth Jayachandran commented on HIVE-11312:
--
[~sershe] Can you take a look since Gopal is out?
> ORC format: where clause with CHAR data type not returning any rows
> ---
>
> Key: HIVE-11312
> URL: https://issues.apache.org/jira/browse/HIVE-11312
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.0, 1.3.0, 1.2.1, 2.0.0
>Reporter: Thomas Friedrich
>Assignee: Prasanth Jayachandran
> Labels: orc
> Attachments: HIVE-11312.1.patch, HIVE-11312.2.patch,
> HIVE-11312.3.patch
>
>
> Test case:
> Setup:
> create table orc_test( col1 string, col2 char(10)) stored as orc
> tblproperties ("orc.compress"="NONE");
> insert into orc_test values ('val1', '1');
> Query:
> select * from orc_test where col2='1';
> Query returns no row.
> Problem is introduced with HIVE-10286, class RecordReaderImpl.java, method
> evaluatePredicateRange.
> Old code:
> - Object baseObj = predicate.getLiteral(PredicateLeaf.FileFormat.ORC);
> - Object minValue = getConvertedStatsObj(min, baseObj);
> - Object maxValue = getConvertedStatsObj(max, baseObj);
> - Object predObj = getBaseObjectForComparison(baseObj, minValue);
> New code:
> + Object baseObj = predicate.getLiteral();
> + Object minValue = getBaseObjectForComparison(predicate.getType(), min);
> + Object maxValue = getBaseObjectForComparison(predicate.getType(), max);
> + Object predObj = getBaseObjectForComparison(predicate.getType(), baseObj);
> The values for min and max are of type String which contain as many
> characters as the CHAR column indicated. For example if the type is CHAR(10),
> and the row has value 1, the value of String min is "1 ";
> Before Hive 1.2, the method getConvertedStatsObj would call
> StringUtils.stripEnd(statsObj.toString(), null); which would remove the
> trailing spaces from min and max. Later in the compareToRange method, it was
> able to compare "1" with "1".
> In Hive 1.2 with the use getBaseObjectForComparison method, it simply returns
> obj.String if the data type is String, which means minValue and maxValue are
> still "1 ".
> As a result, the compareToRange method will return a wrong value
> ("1".compareTo("1 ") -9 instead of 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11312) ORC format: where clause with CHAR data type not returning any rows
[
https://issues.apache.org/jira/browse/HIVE-11312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034393#comment-15034393
]
Sergey Shelukhin commented on HIVE-11312:
-
Is {noformat}
olumnType.startsWith(serdeConstants.CHAR_TYPE_NAME) {noformat} a proper way to
check the type? Otherwise looks good.
> ORC format: where clause with CHAR data type not returning any rows
> ---
>
> Key: HIVE-11312
> URL: https://issues.apache.org/jira/browse/HIVE-11312
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.0, 1.3.0, 1.2.1, 2.0.0
>Reporter: Thomas Friedrich
>Assignee: Prasanth Jayachandran
> Labels: orc
> Attachments: HIVE-11312.1.patch, HIVE-11312.2.patch,
> HIVE-11312.3.patch
>
>
> Test case:
> Setup:
> create table orc_test( col1 string, col2 char(10)) stored as orc
> tblproperties ("orc.compress"="NONE");
> insert into orc_test values ('val1', '1');
> Query:
> select * from orc_test where col2='1';
> Query returns no row.
> Problem is introduced with HIVE-10286, class RecordReaderImpl.java, method
> evaluatePredicateRange.
> Old code:
> - Object baseObj = predicate.getLiteral(PredicateLeaf.FileFormat.ORC);
> - Object minValue = getConvertedStatsObj(min, baseObj);
> - Object maxValue = getConvertedStatsObj(max, baseObj);
> - Object predObj = getBaseObjectForComparison(baseObj, minValue);
> New code:
> + Object baseObj = predicate.getLiteral();
> + Object minValue = getBaseObjectForComparison(predicate.getType(), min);
> + Object maxValue = getBaseObjectForComparison(predicate.getType(), max);
> + Object predObj = getBaseObjectForComparison(predicate.getType(), baseObj);
> The values for min and max are of type String which contain as many
> characters as the CHAR column indicated. For example if the type is CHAR(10),
> and the row has value 1, the value of String min is "1 ";
> Before Hive 1.2, the method getConvertedStatsObj would call
> StringUtils.stripEnd(statsObj.toString(), null); which would remove the
> trailing spaces from min and max. Later in the compareToRange method, it was
> able to compare "1" with "1".
> In Hive 1.2 with the use getBaseObjectForComparison method, it simply returns
> obj.String if the data type is String, which means minValue and maxValue are
> still "1 ".
> As a result, the compareToRange method will return a wrong value
> ("1".compareTo("1 ") -9 instead of 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12491) Column Statistics: 3 attribute join on a 2-source table is off
[
https://issues.apache.org/jira/browse/HIVE-12491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-12491:
Attachment: HIVE-12491.2.patch
Addressed comments and little bit of refactoring in StatsRuleProcFactory (no
logic change there) for better readability.
> Column Statistics: 3 attribute join on a 2-source table is off
> --
>
> Key: HIVE-12491
> URL: https://issues.apache.org/jira/browse/HIVE-12491
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Gopal V
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-12491.2.patch, HIVE-12491.WIP.patch,
> HIVE-12491.patch
>
>
> The eased out denominator has to detect duplicate row-stats from different
> attributes.
> {code}
> select account_id from customers c, customer_activation ca
> where c.customer_id = ca.customer_id
> and year(ca.dt) = year(c.dt) and month(ca.dt) = month(c.dt)
> and year(ca.dt) between year('2013-12-26') and year('2013-12-26')
> {code}
> {code}
> private Long getEasedOutDenominator(List distinctVals) {
> // Exponential back-off for NDVs.
> // 1) Descending order sort of NDVs
> // 2) denominator = NDV1 * (NDV2 ^ (1/2)) * (NDV3 ^ (1/4))) *
> Collections.sort(distinctVals, Collections.reverseOrder());
> long denom = distinctVals.get(0);
> for (int i = 1; i < distinctVals.size(); i++) {
> denom = (long) (denom * Math.pow(distinctVals.get(i), 1.0 / (1 <<
> i)));
> }
> return denom;
> }
> {code}
> This gets {{[8007986, 821974390, 821974390]}}, which is actually 3 columns 2
> of which are derived from the same column.
> {code}
> Reduce Output Operator (RS_12)
> key expressions: _col0 (type: bigint), year(_col2) (type: int),
> month(_col2) (type: int)
> sort order: +++
> Map-reduce partition columns: _col0 (type: bigint), year(_col2)
> (type: int), month(_col2) (type: int)
> value expressions: _col1 (type: bigint)
> Join Operator (JOIN_13)
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col0 (type: bigint), year(_col1) (type: int), month(_col1)
> (type: int)
> 1 _col0 (type: bigint), year(_col2) (type: int), month(_col2)
> (type: int)
> outputColumnNames: _col3
> {code}
> So the eased out denominator is off by a factor of 30,000 or so, causing OOMs
> in map-joins.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12552) Wrong number of reducer estimation causing job to fail
[
https://issues.apache.org/jira/browse/HIVE-12552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034348#comment-15034348
]
Gunther Hagleitner commented on HIVE-12552:
---
That was done for min but not for max. The implicit assumption was that max >
min and max > 1, min < 1. Can you elaborate how a max of 0.5 is actually
helping (instead of for instance changing bytes per reducer)?
Otherwise patch looks good. Would be nice though if you use the same style for
both min and max (i.e.: nested Math.max for both or two statements for both).
> Wrong number of reducer estimation causing job to fail
> --
>
> Key: HIVE-12552
> URL: https://issues.apache.org/jira/browse/HIVE-12552
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: 6_plan.txt, HIVE-12552.1.patch,
> With_max_partition_0.5_setting.png, with_default_setting.png
>
>
> {noformat}
> ], TaskAttempt 3 failed, info=[Error: Failure while running task:
> attempt_1448429572030_1812_1_03_29_3:java.lang.RuntimeException:
> java.lang.RuntimeException: Hive Runtime Error while closing operators:
> java.io.IOException: Illegal partition for 01 6c 6f 61 6e 20 61 63 63 6f 75
> 6e 74 00 01 80 1f e1 d7 ff (-1)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:348)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Hive Runtime Error while closing
> operators: java.io.IOException: Illegal partition for 01 6c 6f 61 6e 20 61 63
> 63 6f 75 6e 74 00 01 80 1f e1 d7 ff (-1)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:341)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: Illegal partition for 01 6c 6f 61 6e 20 61 63 63 6f 75
> 6e 74 00 01 80 1f e1 d7 ff (-1)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:402)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:852)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.writeSingleRow(VectorGroupByOperator.java:904)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.access$400(VectorGroupByOperator.java:59)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.flush(VectorGroupByOperator.java:469)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.close(VectorGroupByOperator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.closeOp(VectorGroupByOperator.java:950)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:656)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:670)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:670)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:318)
> ... 15 more
> Caused by: java.io.IOException: Illegal partition for 01 6c 6f 61 6e 20 61 63
> 63 6f 75 6e 74 00 01 80 1f e1 d7 ff (-1)
> at
> org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.collect(PipelinedSorter.java:379)
> at
> org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:357)
> at
> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:163)
> at
>
[jira] [Commented] (HIVE-12471) Secure HS2 web UI with SSL
[ https://issues.apache.org/jira/browse/HIVE-12471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034345#comment-15034345 ] Jimmy Xiang commented on HIVE-12471: The patch is on RB: https://reviews.apache.org/r/40833/ > Secure HS2 web UI with SSL > -- > > Key: HIVE-12471 > URL: https://issues.apache.org/jira/browse/HIVE-12471 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: HIVE-12471.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
