[jira] [Commented] (HIVE-12045) ClassNotFound for GenericUDF in "select distinct..." query (Hive on Spark)
[
https://issues.apache.org/jira/browse/HIVE-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007774#comment-15007774
]
Rui Li commented on HIVE-12045:
---
Thanks Xuefu. I'll try with master.
> ClassNotFound for GenericUDF in "select distinct..." query (Hive on Spark)
> --
>
> Key: HIVE-12045
> URL: https://issues.apache.org/jira/browse/HIVE-12045
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Environment: Cloudera QuickStart VM - CDH5.4.2
> beeline
>Reporter: Zsolt Tóth
>Assignee: Rui Li
> Attachments: HIVE-12045.1-spark.patch, HIVE-12045.2-spark.patch,
> example.jar, genUDF.patch, hive.log.gz
>
>
> If I execute the following query in beeline, I get ClassNotFoundException for
> the UDF class.
> {code}
> drop function myGenericUdf;
> create function myGenericUdf as 'org.example.myGenericUdf' using jar
> 'hdfs:///tmp/myudf.jar';
> select distinct myGenericUdf(1,2,1) from mytable;
> {code}
> In my example, myGenericUdf just looks for the 1st argument's value in the
> others and returns the index. I don't think this is related to the actual
> GenericUDF function.
> Note that:
> "select myGenericUdf(1,2,1) from mytable;" succeeds
> If I use the non-generic implementation of the same UDF, the select distinct
> call succeeds.
> StackTrace:
> {code}
> 15/10/06 05:20:25 ERROR exec.Utilities: Failed to load plan:
> hdfs://quickstart.cloudera:8020/tmp/hive/hive/f9de3f09-c12d-4528-9ee6-1f12932a14ae/hive_2015-10-06_05-20-07_438_6519207588897968406-20/-mr-10003/27cd7226-3e22-46f4-bddd-fb8fd4aa4b8d/map.xml:
> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
> class: org.example.myGenericUDF
> Serialization trace:
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.GroupByOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
> class: org.example.myGenericUDF
> Serialization trace:
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.GroupByOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
>
[jira] [Updated] (HIVE-12430) Remove remaining reference to the hadoop-2 profile
[ https://issues.apache.org/jira/browse/HIVE-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-12430: -- Attachment: HIVE-12430.1.patch Simple patch to fix this. [~sershe], [~gopalv] - please review. > Remove remaining reference to the hadoop-2 profile > -- > > Key: HIVE-12430 > URL: https://issues.apache.org/jira/browse/HIVE-12430 > Project: Hive > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-12430.1.patch > > > A reference to the hadoop-2 profile still exists in packaging/pom.xml. Leads > to llap-server not being built correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11777) implement an option to have single ETL strategy for multiple directories
[ https://issues.apache.org/jira/browse/HIVE-11777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11777: Attachment: HIVE-11777.06.patch Added a test. > implement an option to have single ETL strategy for multiple directories > > > Key: HIVE-11777 > URL: https://issues.apache.org/jira/browse/HIVE-11777 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11777.01.patch, HIVE-11777.02.patch, > HIVE-11777.03.patch, HIVE-11777.04.patch, HIVE-11777.05.patch, > HIVE-11777.06.patch, HIVE-11777.patch > > > In case of metastore footer PPD we don't want to call PPD call with all > attendant SARG, MS and HBase overhead for each directory. If we wait for some > time (10ms? some fraction of inputs?) we can do one call without losing > overall perf. > For now make it time based. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12319) Remove HadoopShims::getHadoopConfNames()
[
https://issues.apache.org/jira/browse/HIVE-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006350#comment-15006350
]
Hive QA commented on HIVE-12319:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12772444/HIVE-12319.1.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9744 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6046/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6046/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6046/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12772444 - PreCommit-HIVE-TRUNK-Build
> Remove HadoopShims::getHadoopConfNames()
>
>
> Key: HIVE-12319
> URL: https://issues.apache.org/jira/browse/HIVE-12319
> Project: Hive
> Issue Type: Improvement
> Components: Shims
>Affects Versions: 2.0.0
>Reporter: Ashutosh Chauhan
>Assignee: Aleksei Statkevich
> Attachments: HIVE-12319.1.patch, HIVE-12319.patch
>
>
> It was introduced in HIVE-6159 It has served its purpose now that we support
> only Hadoop 2.x line
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8396) Hive CliDriver command splitting can be broken when comments are present
[
https://issues.apache.org/jira/browse/HIVE-8396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006544#comment-15006544
]
Elliot West commented on HIVE-8396:
---
I notice that the command line input behaves differently compare with script
exectution with the {{-f}} option (0.14.0):
h4. Case 1: CLI entry fails
{code}
hive (default)> -- hello
> set x=1;
FAILED: ParseException line 2:4 missing KW_ROLE at 'x' near 'x'
line 2:5 missing EOF at '=' near 'x'
{code}
h4. Case 2: Script execution succeeds
{code}
[ewest@host]$ cat x.hql
-- hello
set x=1;
set x;
[ewest@host]$ hive -f x.hql
x=1
{code}
> Hive CliDriver command splitting can be broken when comments are present
>
>
> Key: HIVE-8396
> URL: https://issues.apache.org/jira/browse/HIVE-8396
> Project: Hive
> Issue Type: Bug
> Components: Parser, Query Processor
>Affects Versions: 0.14.0
>Reporter: Sergey Shelukhin
>
> {noformat}
> -- SORT_QUERY_RESULTS
> set hive.cbo.enable=true;
> ... commands ...
> {noformat}
> causes
> {noformat}
> 2014-10-07 18:55:57,193 ERROR ql.Driver (SessionState.java:printError(825)) -
> FAILED: ParseException line 2:4 missing KW_ROLE at 'hive' near 'hive'
> {noformat}
> If the comment is moved after the command it works.
> I noticed this earlier when I comment out parts of some random q file for
> debugging purposes, and it starts failing. This is annoying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12271) Add metrics around HS2 query execution and job submission for Hive
[
https://issues.apache.org/jira/browse/HIVE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008216#comment-15008216
]
Hive QA commented on HIVE-12271:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12772561/HIVE-12271.2.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9785 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6053/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6053/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6053/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12772561 - PreCommit-HIVE-TRUNK-Build
> Add metrics around HS2 query execution and job submission for Hive
> ---
>
> Key: HIVE-12271
> URL: https://issues.apache.org/jira/browse/HIVE-12271
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Lenni Kuff
>Assignee: Szehon Ho
> Attachments: HIVE-12271.2.patch, HIVE-12271.patch
>
>
> We should add more metrics around query execution. Specifically:
> * Number of in-use worker threads
> * Number of in-use async threads
> * Number of queries waiting for compilation
> * Stats for query planning / compilation time
> * Stats for total job submission time
> * Others?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12431) Cancel queries after configurable timeout waiting on compilation
[ https://issues.apache.org/jira/browse/HIVE-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lenni Kuff updated HIVE-12431: -- Assignee: (was: Vaibhav Gumashta) > Cancel queries after configurable timeout waiting on compilation > > > Key: HIVE-12431 > URL: https://issues.apache.org/jira/browse/HIVE-12431 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Query Processor >Affects Versions: 1.2.1 >Reporter: Lenni Kuff > > To help with HiveServer2 scalability, it would be useful to allow users to > configure a timeout value for queries waiting to be compiled. If the timeout > value is reached then the query would abort. One option to achieve this would > be to update the compile lock to use a try-lock with the timeout value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12417) Support for exclamation mark missing in regexp
[
https://issues.apache.org/jira/browse/HIVE-12417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008159#comment-15008159
]
Olaf Flebbe commented on HIVE-12417:
Oops, the negation operator is broken now, somehow. Canceling patch
> Support for exclamation mark missing in regexp
> --
>
> Key: HIVE-12417
> URL: https://issues.apache.org/jira/browse/HIVE-12417
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Olaf Flebbe
>Assignee: Olaf Flebbe
> Attachments: HIVE-12417.1.patch
>
>
> with HIVE-6013 gets support for regular expressions. However, die ! character
> is valid, too. It is needed for expressions like
> {code}
> set hive.support.quoted.identifiers = none;
> select `^(?!donotuseme).*$` from table;
> {code}
> which is the idiom to select all but column {{donotuseme}} .
> See http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for
> a reference of supported chars in Java regexp.
> The patch simply fixes the lexer to support '!' as REGEX char. And does
> simply work.
> Please review.
> If you like to have an iTest for it, I beg you to help me. I tried several
> days on a different issue to figure out how it is supposed to work and failed
> miserably.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12413) Default mode for hive.mapred.mode should be strict
[ https://issues.apache.org/jira/browse/HIVE-12413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12413: Attachment: HIVE-12413.1.patch > Default mode for hive.mapred.mode should be strict > -- > > Key: HIVE-12413 > URL: https://issues.apache.org/jira/browse/HIVE-12413 > Project: Hive > Issue Type: Task > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12413.1.patch, HIVE-12413.patch > > > Non-strict mode allows some questionable semantics and questionable > operations. Its better that user makes a conscious choice to enable such a > behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
