[jira] [Commented] (HIVE-12196) NPE when converting bad timestamp value
[
https://issues.apache.org/jira/browse/HIVE-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006904#comment-15006904
]
Ryan Blue commented on HIVE-12196:
--
Thanks for fixign this, Aihua!
> NPE when converting bad timestamp value
> ---
>
> Key: HIVE-12196
> URL: https://issues.apache.org/jira/browse/HIVE-12196
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 1.1.1
>Reporter: Ryan Blue
>Assignee: Aihua Xu
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-12196.patch
>
>
> When I convert a timestamp value that is slightly wrong, the result is a NPE.
> Other queries correctly reject the timestamp:
> {code}
> hive> select from_utc_timestamp('2015-04-11-12:24:34.535', 'UTC');
> FAILED: NullPointerException null
> hive> select TIMESTAMP '2015-04-11-12:24:34.535';
> FAILED: SemanticException Unable to convert time literal
> '2015-04-11-12:24:34.535' to time value.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-12196) NPE when converting bad timestamp value
[
https://issues.apache.org/jira/browse/HIVE-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu resolved HIVE-12196.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Thanks Chaoyu for reviewing the code.
> NPE when converting bad timestamp value
> ---
>
> Key: HIVE-12196
> URL: https://issues.apache.org/jira/browse/HIVE-12196
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 1.1.1
>Reporter: Ryan Blue
>Assignee: Aihua Xu
> Fix For: 2.0.0
>
> Attachments: HIVE-12196.patch
>
>
> When I convert a timestamp value that is slightly wrong, the result is a NPE.
> Other queries correctly reject the timestamp:
> {code}
> hive> select from_utc_timestamp('2015-04-11-12:24:34.535', 'UTC');
> FAILED: NullPointerException null
> hive> select TIMESTAMP '2015-04-11-12:24:34.535';
> FAILED: SemanticException Unable to convert time literal
> '2015-04-11-12:24:34.535' to time value.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12196) NPE when converting bad timestamp value
[
https://issues.apache.org/jira/browse/HIVE-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-12196:
Fix Version/s: 1.3.0
> NPE when converting bad timestamp value
> ---
>
> Key: HIVE-12196
> URL: https://issues.apache.org/jira/browse/HIVE-12196
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 1.1.1
>Reporter: Ryan Blue
>Assignee: Aihua Xu
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-12196.patch
>
>
> When I convert a timestamp value that is slightly wrong, the result is a NPE.
> Other queries correctly reject the timestamp:
> {code}
> hive> select from_utc_timestamp('2015-04-11-12:24:34.535', 'UTC');
> FAILED: NullPointerException null
> hive> select TIMESTAMP '2015-04-11-12:24:34.535';
> FAILED: SemanticException Unable to convert time literal
> '2015-04-11-12:24:34.535' to time value.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6113) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006600#comment-15006600 ] Oleksiy Sayankin commented on HIVE-6113: ROOT-CAUSE: Bug http://www.datanucleus.org/servlet/jira/browse/NUCRDBMS-755 in DataNucleus Store RDBMS of version 3.2.9 SOLUTION: Use DataNucleus Store RDBMS of higher version where fix of NUCRDBMS-755 is provided. Version change summary: datanucleus-api-jdo 3.2.6 ---> 4.2.1 datanucleus-core3.2.10 ---> 4.1.6 datanucleus-rdbms 3.2.9 ---> 4.1.7 > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > -- > > Key: HIVE-6113 > URL: https://issues.apache.org/jira/browse/HIVE-6113 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 0.12.0 > Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0 >Reporter: William Stone >Priority: Critical > Labels: HiveMetaStoreClient, metastore, unable_instantiate > Attachments: HIVE-6113.patch > > > When I exccute SQL "use fdm; desc formatted fdm.tableName;" in python, throw > Error as followed. > but when I tryit again , It will success. > 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - > org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143) > at > org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128) > at > org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:197) > Caused by: java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) > at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383) > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139) > ... 20 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) > ... 25 more > Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes > to datastore > NestedThrowables: > java.sql.BatchUpdateException: Duplicate entry 'default' for key > 'UNIQUE_DATABASE' > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) > at > org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) > at >
[jira] [Updated] (HIVE-6113) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-6113: --- Attachment: HIVE-6113.patch > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > -- > > Key: HIVE-6113 > URL: https://issues.apache.org/jira/browse/HIVE-6113 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 0.12.0 > Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0 >Reporter: William Stone >Priority: Critical > Labels: HiveMetaStoreClient, metastore, unable_instantiate > Attachments: HIVE-6113.patch > > > When I exccute SQL "use fdm; desc formatted fdm.tableName;" in python, throw > Error as followed. > but when I tryit again , It will success. > 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - > org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143) > at > org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128) > at > org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:197) > Caused by: java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) > at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383) > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139) > ... 20 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) > ... 25 more > Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes > to datastore > NestedThrowables: > java.sql.BatchUpdateException: Duplicate entry 'default' for key > 'UNIQUE_DATABASE' > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) > at > org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) > at > org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:358) > at > org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:404) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >
[jira] [Commented] (HIVE-11304) Migrate to Log4j2 from Log4j 1.x
[ https://issues.apache.org/jira/browse/HIVE-11304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006789#comment-15006789 ] Xuefu Zhang commented on HIVE-11304: Thanks for looking at it. I will retry and post my new findings. > Migrate to Log4j2 from Log4j 1.x > > > Key: HIVE-11304 > URL: https://issues.apache.org/jira/browse/HIVE-11304 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Labels: TODOC2.0, incompatibleChange > Fix For: 2.0.0 > > Attachments: HIVE-11304.10.patch, HIVE-11304.11.patch, > HIVE-11304.2.patch, HIVE-11304.3.patch, HIVE-11304.4.patch, > HIVE-11304.5.patch, HIVE-11304.6.patch, HIVE-11304.7.patch, > HIVE-11304.8.patch, HIVE-11304.9.patch, HIVE-11304.patch > > > Log4J2 has some great benefits and can benefit hive significantly. Some > notable features include > 1) Performance (parametrized logging, performance when logging is disabled > etc.) More details can be found here > https://logging.apache.org/log4j/2.x/performance.html > 2) RoutingAppender - Route logs to different log files based on MDC context > (useful for HS2, LLAP etc.) > 3) Asynchronous logging > This is an umbrella jira to track changes related to Log4j2 migration. > Log4J1 EOL - > https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12196) NPE when converting bad timestamp value
[
https://issues.apache.org/jira/browse/HIVE-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006814#comment-15006814
]
Aihua Xu commented on HIVE-12196:
-
Committed to branch-1 as well.
> NPE when converting bad timestamp value
> ---
>
> Key: HIVE-12196
> URL: https://issues.apache.org/jira/browse/HIVE-12196
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 1.1.1
>Reporter: Ryan Blue
>Assignee: Aihua Xu
> Fix For: 2.0.0
>
> Attachments: HIVE-12196.patch
>
>
> When I convert a timestamp value that is slightly wrong, the result is a NPE.
> Other queries correctly reject the timestamp:
> {code}
> hive> select from_utc_timestamp('2015-04-11-12:24:34.535', 'UTC');
> FAILED: NullPointerException null
> hive> select TIMESTAMP '2015-04-11-12:24:34.535';
> FAILED: SemanticException Unable to convert time literal
> '2015-04-11-12:24:34.535' to time value.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12402) Split hive.root.logger separately to make it compatible with log4j1.x
[ https://issues.apache.org/jira/browse/HIVE-12402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006924#comment-15006924 ] Lefty Leverenz commented on HIVE-12402: --- Doc note: This should be documented in the logging section of Getting Started. * [Getting Started -- Hive Logging | https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-HiveLogging] > Split hive.root.logger separately to make it compatible with log4j1.x > - > > Key: HIVE-12402 > URL: https://issues.apache.org/jira/browse/HIVE-12402 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12402.patch > > > With new Log4j2.x specifying logger name and log level together will not work. > With old logger following will work > --hiveconf hive.root.logger=DEBUG,console > But with new logger we should specify logger and level separately > --hiveconf hive.root.logger=console --hiveconf hive.log.level=DEBUG > We can do this change internally for users still using the old configs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12196) NPE when converting bad timestamp value
[
https://issues.apache.org/jira/browse/HIVE-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006799#comment-15006799
]
Aihua Xu commented on HIVE-12196:
-
Committed to master branch.
> NPE when converting bad timestamp value
> ---
>
> Key: HIVE-12196
> URL: https://issues.apache.org/jira/browse/HIVE-12196
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 1.1.1
>Reporter: Ryan Blue
>Assignee: Aihua Xu
> Fix For: 2.0.0
>
> Attachments: HIVE-12196.patch
>
>
> When I convert a timestamp value that is slightly wrong, the result is a NPE.
> Other queries correctly reject the timestamp:
> {code}
> hive> select from_utc_timestamp('2015-04-11-12:24:34.535', 'UTC');
> FAILED: NullPointerException null
> hive> select TIMESTAMP '2015-04-11-12:24:34.535';
> FAILED: SemanticException Unable to convert time literal
> '2015-04-11-12:24:34.535' to time value.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12384) Union Operator may produce incorrect result on TEZ
[
https://issues.apache.org/jira/browse/HIVE-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006782#comment-15006782
]
Hive QA commented on HIVE-12384:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12772451/HIVE-12384.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9783 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_unionDistinct_1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6048/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6048/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6048/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12772451 - PreCommit-HIVE-TRUNK-Build
> Union Operator may produce incorrect result on TEZ
> --
>
> Key: HIVE-12384
> URL: https://issues.apache.org/jira/browse/HIVE-12384
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 0.14.0, 1.0.0, 1.1.0, 1.0.1, 1.2.1
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-12384.1.patch, HIVE-12384.2.patch
>
>
> Union queries may produce incorrect result on TEZ.
> TEZ removes union op, thus might loose the implicit cast in union op.
> Reproduction test case:
> set hive.cbo.enable=false;
> set hive.execution.engine=tez;
> select (x/sum(x) over()) as y from(select cast(1 as decimal(10,0)) as x
> from (select * from src limit 2)s1 union all select cast(1 as decimal(10,0))
> x from (select * from src limit 2) s2 union all select '1' x from
> (select * from src limit 2) s3)u order by y;
> select (x/sum(x) over()) as y from(select cast(1 as decimal(10,0)) as x from
> (select * from src limit 2)s1 union all select cast(1 as decimal(10,0)) x
> from (select * from src limit 2) s2 union all select cast (null as string) x
> from (select * from src limit 2) s3)u order by y;
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12402) Split hive.root.logger separately to make it compatible with log4j1.x
[ https://issues.apache.org/jira/browse/HIVE-12402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-12402: -- Labels: TODOC2.0 (was: ) > Split hive.root.logger separately to make it compatible with log4j1.x > - > > Key: HIVE-12402 > URL: https://issues.apache.org/jira/browse/HIVE-12402 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12402.patch > > > With new Log4j2.x specifying logger name and log level together will not work. > With old logger following will work > --hiveconf hive.root.logger=DEBUG,console > But with new logger we should specify logger and level separately > --hiveconf hive.root.logger=console --hiveconf hive.log.level=DEBUG > We can do this change internally for users still using the old configs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8396) Hive CliDriver command splitting can be broken when comments are present
[
https://issues.apache.org/jira/browse/HIVE-8396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006640#comment-15006640
]
Elliot West commented on HIVE-8396:
---
Worth noting that there is a complementary symmetric set of behaviours that are
also not in agreement:
h4. CLI entry
{code}
hive (default)> set z="
> --hello
> ";
hive (default)> set z;
z="
--hello
"
{code}
h4. Script execution
{code}
[ewest@host]$ cat z.hql
set z="
-- hello
";
set z;
[ewest@host]$ hive -f z.hql
z="
"
{code}
> Hive CliDriver command splitting can be broken when comments are present
>
>
> Key: HIVE-8396
> URL: https://issues.apache.org/jira/browse/HIVE-8396
> Project: Hive
> Issue Type: Bug
> Components: Parser, Query Processor
>Affects Versions: 0.14.0
>Reporter: Sergey Shelukhin
>
> {noformat}
> -- SORT_QUERY_RESULTS
> set hive.cbo.enable=true;
> ... commands ...
> {noformat}
> causes
> {noformat}
> 2014-10-07 18:55:57,193 ERROR ql.Driver (SessionState.java:printError(825)) -
> FAILED: ParseException line 2:4 missing KW_ROLE at 'hive' near 'hive'
> {noformat}
> If the comment is moved after the command it works.
> I noticed this earlier when I comment out parts of some random q file for
> debugging purposes, and it starts failing. This is annoying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12417) Support for exclamation mark missing in regexp
[
https://issues.apache.org/jira/browse/HIVE-12417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Olaf Flebbe updated HIVE-12417:
---
Attachment: HIVE-12417.1.patch
> Support for exclamation mark missing in regexp
> --
>
> Key: HIVE-12417
> URL: https://issues.apache.org/jira/browse/HIVE-12417
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Olaf Flebbe
>Assignee: Olaf Flebbe
> Attachments: HIVE-12417.1.patch
>
>
> with HIVE-6013 gets support for regular expressions. However, die ! character
> is valid, too. It is needed for expressions like
> {code}
> set hive.support.quoted.identifiers = none;
> select '^(?!donotuseme).*$ from table;
> {code}
> which is the idiom to select all but column {{donotuseme}} .
> See http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for
> a reference of supported chars in Java regexp.
> The patch simply fixes the lexer to support '!' as REGEX char. And does
> simply work.
> Please review.
> If you like to have an iTest for it, I beg you to help me. I tried several
> days on a different issue to figure out how it is supposed to work and failed
> miserably.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12417) Support for exclamation mark missing in regexp
[
https://issues.apache.org/jira/browse/HIVE-12417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Olaf Flebbe updated HIVE-12417:
---
Description:
with HIVE-6013 gets support for regular expressions. However, die ! character
is valid, too. It is needed for expressions like
{code}
set hive.support.quoted.identifiers = none;
select `^(?!donotuseme).*$` from table;
{code}
which is the idiom to select all but column {{donotuseme}} .
See http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for a
reference of supported chars in Java regexp.
The patch simply fixes the lexer to support '!' as REGEX char. And does simply
work.
Please review.
If you like to have an iTest for it, I beg you to help me. I tried several days
on a different issue to figure out how it is supposed to work and failed
miserably.
was:
with HIVE-6013 gets support for regular expressions. However, die ! character
is valid, too. It is needed for expressions like
{code}
set hive.support.quoted.identifiers = none;
select '^(?!donotuseme).*$ from table;
{code}
which is the idiom to select all but column {{donotuseme}} .
See http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for a
reference of supported chars in Java regexp.
The patch simply fixes the lexer to support '!' as REGEX char. And does simply
work.
Please review.
If you like to have an iTest for it, I beg you to help me. I tried several days
on a different issue to figure out how it is supposed to work and failed
miserably.
> Support for exclamation mark missing in regexp
> --
>
> Key: HIVE-12417
> URL: https://issues.apache.org/jira/browse/HIVE-12417
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Olaf Flebbe
>Assignee: Olaf Flebbe
> Attachments: HIVE-12417.1.patch
>
>
> with HIVE-6013 gets support for regular expressions. However, die ! character
> is valid, too. It is needed for expressions like
> {code}
> set hive.support.quoted.identifiers = none;
> select `^(?!donotuseme).*$` from table;
> {code}
> which is the idiom to select all but column {{donotuseme}} .
> See http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for
> a reference of supported chars in Java regexp.
> The patch simply fixes the lexer to support '!' as REGEX char. And does
> simply work.
> Please review.
> If you like to have an iTest for it, I beg you to help me. I tried several
> days on a different issue to figure out how it is supposed to work and failed
> miserably.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6113) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006620#comment-15006620 ] Oleksiy Sayankin commented on HIVE-6113: For review https://reviews.apache.org/r/40344/ > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > -- > > Key: HIVE-6113 > URL: https://issues.apache.org/jira/browse/HIVE-6113 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.0.0, 1.2.1 > Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0 >Reporter: William Stone >Priority: Critical > Labels: HiveMetaStoreClient, metastore, unable_instantiate > Fix For: 1.2.1 > > Attachments: HIVE-6113.patch > > > When I exccute SQL "use fdm; desc formatted fdm.tableName;" in python, throw > Error as followed. > but when I tryit again , It will success. > 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - > org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143) > at > org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128) > at > org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:197) > Caused by: java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) > at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383) > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139) > ... 20 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) > ... 25 more > Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes > to datastore > NestedThrowables: > java.sql.BatchUpdateException: Duplicate entry 'default' for key > 'UNIQUE_DATABASE' > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) > at > org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) > at > org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:358) > at > org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:404) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at >
[jira] [Commented] (HIVE-11110) Reorder applyPreJoinOrderingTransforms, add NotNULL/FilterMerge rules, improve Filter selectivity estimation
[
https://issues.apache.org/jira/browse/HIVE-0?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006579#comment-15006579
]
Jesus Camacho Rodriguez commented on HIVE-0:
+1
> Reorder applyPreJoinOrderingTransforms, add NotNULL/FilterMerge rules,
> improve Filter selectivity estimation
>
>
> Key: HIVE-0
> URL: https://issues.apache.org/jira/browse/HIVE-0
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-0-10.patch, HIVE-0-11.patch,
> HIVE-0-12.patch, HIVE-0-branch-1.2.patch, HIVE-0.1.patch,
> HIVE-0.13.patch, HIVE-0.14.patch, HIVE-0.15.patch,
> HIVE-0.16.patch, HIVE-0.17.patch, HIVE-0.18.patch,
> HIVE-0.19.patch, HIVE-0.2.patch, HIVE-0.20.patch,
> HIVE-0.21.patch, HIVE-0.22.patch, HIVE-0.23.patch,
> HIVE-0.24.patch, HIVE-0.4.patch, HIVE-0.5.patch,
> HIVE-0.6.patch, HIVE-0.7.patch, HIVE-0.8.patch,
> HIVE-0.9.patch, HIVE-0.91.patch, HIVE-0.92.patch, HIVE-0.patch
>
>
> Query
> {code}
> select count(*)
> from store_sales
> ,store_returns
> ,date_dim d1
> ,date_dim d2
> where d1.d_quarter_name = '2000Q1'
>and d1.d_date_sk = ss_sold_date_sk
>and ss_customer_sk = sr_customer_sk
>and ss_item_sk = sr_item_sk
>and ss_ticket_number = sr_ticket_number
>and sr_returned_date_sk = d2.d_date_sk
>and d2.d_quarter_name in ('2000Q1','2000Q2','2000Q3’);
> {code}
> The store_sales table is partitioned on ss_sold_date_sk, which is also used
> in a join clause. The join clause should add a filter “filterExpr:
> ss_sold_date_sk is not null”, which should get pushed the MetaStore when
> fetching the stats. Currently this is not done in CBO planning, which results
> in the stats from __HIVE_DEFAULT_PARTITION__ to be fetched and considered in
> the optimization phase. In particular, this increases the NDV for the join
> columns and may result in wrong planning.
> Including HiveJoinAddNotNullRule in the optimization phase solves this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8396) Hive CliDriver command splitting can be broken when comments are present
[
https://issues.apache.org/jira/browse/HIVE-8396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006685#comment-15006685
]
Elliot West commented on HIVE-8396:
---
Beeline does this. See {{org.apache.hive.beeline.BeeLine.dispatch(String)}},
specifically where {{isComment(line)}} is called.
> Hive CliDriver command splitting can be broken when comments are present
>
>
> Key: HIVE-8396
> URL: https://issues.apache.org/jira/browse/HIVE-8396
> Project: Hive
> Issue Type: Bug
> Components: Parser, Query Processor
>Affects Versions: 0.14.0
>Reporter: Sergey Shelukhin
>
> {noformat}
> -- SORT_QUERY_RESULTS
> set hive.cbo.enable=true;
> ... commands ...
> {noformat}
> causes
> {noformat}
> 2014-10-07 18:55:57,193 ERROR ql.Driver (SessionState.java:printError(825)) -
> FAILED: ParseException line 2:4 missing KW_ROLE at 'hive' near 'hive'
> {noformat}
> If the comment is moved after the command it works.
> I noticed this earlier when I comment out parts of some random q file for
> debugging purposes, and it starts failing. This is annoying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11488) Add sessionId and queryId info to HS2 log
[ https://issues.apache.org/jira/browse/HIVE-11488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006990#comment-15006990 ] Aihua Xu commented on HIVE-11488: - TestHWISessionManager unit test passed locally. Seems the other tests are not related. > Add sessionId and queryId info to HS2 log > - > > Key: HIVE-11488 > URL: https://issues.apache.org/jira/browse/HIVE-11488 > Project: Hive > Issue Type: New Feature > Components: Logging >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11488.2.patch, HIVE-11488.3.patch, HIVE-11488.patch > > > Session is critical for a multi-user system like Hive. Currently Hive doesn't > log seessionId to the log file, which sometimes make debugging and analysis > difficult when multiple activities are going on at the same time and the log > from different sessions are mixed together. > Currently, Hive already has the sessionId saved in SessionState and also > there is another sessionId in SessionHandle (Seems not used and I'm still > looking to understand it). Generally we should have one sessionId from the > beginning in the client side and server side. Seems we have some work on that > side first. > The sessionId then can be added to log4j supported mapped diagnostic context > (MDC) and can be configured to output to log file through the log4j property. > MDC is per thread, so we need to add sessionId to the HS2 main thread and > then it will be inherited by the child threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12017) Do not disable CBO by default when number of joins in a query is equal or less than 1
[ https://issues.apache.org/jira/browse/HIVE-12017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007037#comment-15007037 ] Jesus Camacho Rodriguez commented on HIVE-12017: [~ashutoshc], I was waiting for a QA run. I updated the patch with the latest q files updates, and the RB entry with the code changes as well. Thanks > Do not disable CBO by default when number of joins in a query is equal or > less than 1 > - > > Key: HIVE-12017 > URL: https://issues.apache.org/jira/browse/HIVE-12017 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12017.01.patch, HIVE-12017.02.patch, > HIVE-12017.03.patch, HIVE-12017.04.patch, HIVE-12017.05.patch, > HIVE-12017.06.patch, HIVE-12017.07.patch, HIVE-12017.08.patch, > HIVE-12017.09.patch, HIVE-12017.10.patch > > > Instead, we could disable some parts of CBO that are not relevant if the > query contains 1 or 0 joins. Implementation should be able to define easily > other query patterns for which we might disable some parts of CBO (in case we > want to do it in the future). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6113) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-6113: --- Target Version/s: 1.2.2 (was: 1.2.1) > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > -- > > Key: HIVE-6113 > URL: https://issues.apache.org/jira/browse/HIVE-6113 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.0.0, 1.2.1 > Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0 >Reporter: William Stone >Priority: Critical > Labels: HiveMetaStoreClient, metastore, unable_instantiate > Attachments: HIVE-6113.patch > > > When I exccute SQL "use fdm; desc formatted fdm.tableName;" in python, throw > Error as followed. > but when I tryit again , It will success. > 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - > org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143) > at > org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128) > at > org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:197) > Caused by: java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) > at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383) > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139) > ... 20 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) > ... 25 more > Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes > to datastore > NestedThrowables: > java.sql.BatchUpdateException: Duplicate entry 'default' for key > 'UNIQUE_DATABASE' > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) > at > org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) > at > org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:358) > at > org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:404) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at >
[jira] [Commented] (HIVE-6113) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006993#comment-15006993 ] Sushanth Sowmyan commented on HIVE-6113: @[~osayankin] : I know [~sershe] will be pleased with this proposal, he's been suggesting the same for other reasons. :) That said, whenever we do a major version bump of DN, we should do a verification to ensure that we continue to work correctly. Have you verified the elements in http://www.datanucleus.org/products/accessplatform_4_2/migration.html to see if we won't be affected adversely? @[~eliac] : Very interesting analysis. Could you point me to where you see the following: > If a table is deleted from the DB during this operation, > DatabaseMetaData.getColumns will throw an exception. >This exception is interpreted by Hive to mean that the "default" Hive database >doesn't exist. Because I do recollect similar sounding issues where DN would report a null return when we tried to do a getDatabase, which reads equivalently to a case of a NoSuchObjectException from us, rather than throwing a JDOException when there was an underlying db issue. If this is the same issue and we have a trace of where that happens, this solves a lot more for us, hopefully. > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > -- > > Key: HIVE-6113 > URL: https://issues.apache.org/jira/browse/HIVE-6113 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.0.0, 1.2.1 > Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0 >Reporter: William Stone >Assignee: Oleksiy Sayankin >Priority: Critical > Labels: HiveMetaStoreClient, metastore, unable_instantiate > Attachments: HIVE-6113.patch > > > When I exccute SQL "use fdm; desc formatted fdm.tableName;" in python, throw > Error as followed. > but when I tryit again , It will success. > 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - > org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143) > at > org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128) > at > org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:197) > Caused by: java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) > at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383) > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139) > ... 20 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at
[jira] [Commented] (HIVE-11488) Add sessionId and queryId info to HS2 log
[ https://issues.apache.org/jira/browse/HIVE-11488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007014#comment-15007014 ] Aihua Xu commented on HIVE-11488: - Committed to master branch. Thanks Szehon for reviewing the code. We probably need to add the doc for this. > Add sessionId and queryId info to HS2 log > - > > Key: HIVE-11488 > URL: https://issues.apache.org/jira/browse/HIVE-11488 > Project: Hive > Issue Type: New Feature > Components: Logging >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11488.2.patch, HIVE-11488.3.patch, HIVE-11488.patch > > > Session is critical for a multi-user system like Hive. Currently Hive doesn't > log seessionId to the log file, which sometimes make debugging and analysis > difficult when multiple activities are going on at the same time and the log > from different sessions are mixed together. > Currently, Hive already has the sessionId saved in SessionState and also > there is another sessionId in SessionHandle (Seems not used and I'm still > looking to understand it). Generally we should have one sessionId from the > beginning in the client side and server side. Seems we have some work on that > side first. > The sessionId then can be added to log4j supported mapped diagnostic context > (MDC) and can be configured to output to log file through the log4j property. > MDC is per thread, so we need to add sessionId to the HS2 main thread and > then it will be inherited by the child threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12017) Do not disable CBO by default when number of joins in a query is equal or less than 1
[ https://issues.apache.org/jira/browse/HIVE-12017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-12017: --- Attachment: HIVE-12017.10.patch > Do not disable CBO by default when number of joins in a query is equal or > less than 1 > - > > Key: HIVE-12017 > URL: https://issues.apache.org/jira/browse/HIVE-12017 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12017.01.patch, HIVE-12017.02.patch, > HIVE-12017.03.patch, HIVE-12017.04.patch, HIVE-12017.05.patch, > HIVE-12017.06.patch, HIVE-12017.07.patch, HIVE-12017.08.patch, > HIVE-12017.09.patch, HIVE-12017.10.patch > > > Instead, we could disable some parts of CBO that are not relevant if the > query contains 1 or 0 joins. Implementation should be able to define easily > other query patterns for which we might disable some parts of CBO (in case we > want to do it in the future). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9780) Add another level of explain for RDBMS audience
[ https://issues.apache.org/jira/browse/HIVE-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007080#comment-15007080 ] Carter Shanklin commented on HIVE-9780: --- This is so much better than the old style explain plans we should consider making it the default for Hive 2.0. > Add another level of explain for RDBMS audience > --- > > Key: HIVE-9780 > URL: https://issues.apache.org/jira/browse/HIVE-9780 > Project: Hive > Issue Type: New Feature > Components: Diagnosability >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong >Priority: Minor > Fix For: 1.2.0 > > Attachments: HIVE-9780.01.patch, HIVE-9780.02.patch, > HIVE-9780.03.patch, HIVE-9780.04.patch, HIVE-9780.05.patch, > HIVE-9780.06.patch, HIVE-9780.07.patch, HIVE-9780.08.patch, > HIVE-9780.09.patch, HIVE-9780.10.patch > > > Current Hive Explain (default) is targeted at MR Audience. We need a new > level of explain plan to be targeted at RDBMS audience. The explain requires > these: > 1) The focus needs to be on what part of the query is being executed rather > than internals of the engines > 2) There needs to be a clearly readable tree of operations > 3) Examples - Table scan should mention the table being scanned, the Sarg, > the size of table and expected cardinality after the Sarg'ed read. The join > should mention the table being joined with and the join condition. The > aggregate should mention the columns in the group-by. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6113) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007081#comment-15007081 ] Sergey Shelukhin commented on HIVE-6113: DN version should be upgraded in 2.0 and 1.3 too if upgrading in 1.2.2. Also the release note is needed. Otherwise this makes sense to me. > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > -- > > Key: HIVE-6113 > URL: https://issues.apache.org/jira/browse/HIVE-6113 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.0.0, 1.2.1 > Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0 >Reporter: William Stone >Assignee: Oleksiy Sayankin >Priority: Critical > Labels: HiveMetaStoreClient, metastore, unable_instantiate > Attachments: HIVE-6113.patch > > > When I exccute SQL "use fdm; desc formatted fdm.tableName;" in python, throw > Error as followed. > but when I tryit again , It will success. > 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - > org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143) > at > org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128) > at > org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:197) > Caused by: java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) > at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383) > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139) > ... 20 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) > ... 25 more > Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes > to datastore > NestedThrowables: > java.sql.BatchUpdateException: Duplicate entry 'default' for key > 'UNIQUE_DATABASE' > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) > at > org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) > at > org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:358) > at > org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:404) > at
[jira] [Updated] (HIVE-11684) Implement limit pushdown through outer join in CBO
[ https://issues.apache.org/jira/browse/HIVE-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-11684: --- Target Version/s: 2.0.0 > Implement limit pushdown through outer join in CBO > -- > > Key: HIVE-11684 > URL: https://issues.apache.org/jira/browse/HIVE-11684 > Project: Hive > Issue Type: New Feature > Components: CBO >Affects Versions: 2.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-11684.01.patch, HIVE-11684.02.patch, > HIVE-11684.03.patch, HIVE-11684.04.patch, HIVE-11684.05.patch, > HIVE-11684.07.patch, HIVE-11684.08.patch, HIVE-11684.09.patch, > HIVE-11684.10.patch, HIVE-11684.11.patch, HIVE-11684.12.patch, > HIVE-11684.12.patch, HIVE-11684.14.patch, HIVE-11684.15.patch, > HIVE-11684.16.patch, HIVE-11684.17.patch, HIVE-11684.18.patch, > HIVE-11684.19.patch, HIVE-11684.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6113) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-6113: --- Fix Version/s: (was: 1.2.1) > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > -- > > Key: HIVE-6113 > URL: https://issues.apache.org/jira/browse/HIVE-6113 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.0.0, 1.2.1 > Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0 >Reporter: William Stone >Priority: Critical > Labels: HiveMetaStoreClient, metastore, unable_instantiate > Attachments: HIVE-6113.patch > > > When I exccute SQL "use fdm; desc formatted fdm.tableName;" in python, throw > Error as followed. > but when I tryit again , It will success. > 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - > org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143) > at > org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128) > at > org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:197) > Caused by: java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) > at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383) > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139) > ... 20 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) > ... 25 more > Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes > to datastore > NestedThrowables: > java.sql.BatchUpdateException: Duplicate entry 'default' for key > 'UNIQUE_DATABASE' > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) > at > org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) > at > org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:358) > at > org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:404) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at >
[jira] [Updated] (HIVE-6113) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-6113: --- Assignee: Oleksiy Sayankin > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > -- > > Key: HIVE-6113 > URL: https://issues.apache.org/jira/browse/HIVE-6113 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.0.0, 1.2.1 > Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0 >Reporter: William Stone >Assignee: Oleksiy Sayankin >Priority: Critical > Labels: HiveMetaStoreClient, metastore, unable_instantiate > Attachments: HIVE-6113.patch > > > When I exccute SQL "use fdm; desc formatted fdm.tableName;" in python, throw > Error as followed. > but when I tryit again , It will success. > 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - > org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143) > at > org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128) > at > org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:197) > Caused by: java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) > at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383) > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139) > ... 20 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) > ... 25 more > Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes > to datastore > NestedThrowables: > java.sql.BatchUpdateException: Duplicate entry 'default' for key > 'UNIQUE_DATABASE' > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) > at > org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) > at > org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:358) > at > org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:404) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at >
[jira] [Updated] (HIVE-6113) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-6113: --- Target Version/s: 1.3.0, 2.0.0, 1.2.2 (was: 1.2.2) > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > -- > > Key: HIVE-6113 > URL: https://issues.apache.org/jira/browse/HIVE-6113 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.0.0, 1.2.1 > Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0 >Reporter: William Stone >Assignee: Oleksiy Sayankin >Priority: Critical > Labels: HiveMetaStoreClient, metastore, unable_instantiate > Attachments: HIVE-6113.patch > > > When I exccute SQL "use fdm; desc formatted fdm.tableName;" in python, throw > Error as followed. > but when I tryit again , It will success. > 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - > org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143) > at > org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128) > at > org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:197) > Caused by: java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) > at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383) > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139) > ... 20 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) > ... 25 more > Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes > to datastore > NestedThrowables: > java.sql.BatchUpdateException: Duplicate entry 'default' for key > 'UNIQUE_DATABASE' > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) > at > org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) > at > org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:358) > at > org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:404) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at
[jira] [Commented] (HIVE-6113) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006999#comment-15006999 ] Sushanth Sowmyan commented on HIVE-6113: A couple of other gardening notes: +cc [~ashutoshc]/[~sershe] as they would also be interested in this issue. Also, I removed the fix-version of 1.2.1 from this bug, fix version is marked by a committer to denote what versions a patch has already been patched into. Target version is the field used to request what version the requestor wants the patch to go into, and for that, it must be unreleased versions. Thus, since 1.2.1 has already been released, I've updated Target version to 1.2.2. > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > -- > > Key: HIVE-6113 > URL: https://issues.apache.org/jira/browse/HIVE-6113 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.0.0, 1.2.1 > Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0 >Reporter: William Stone >Assignee: Oleksiy Sayankin >Priority: Critical > Labels: HiveMetaStoreClient, metastore, unable_instantiate > Attachments: HIVE-6113.patch > > > When I exccute SQL "use fdm; desc formatted fdm.tableName;" in python, throw > Error as followed. > but when I tryit again , It will success. > 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - > org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143) > at > org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128) > at > org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:197) > Caused by: java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) > at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383) > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139) > ... 20 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) > ... 25 more > Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes > to datastore > NestedThrowables: > java.sql.BatchUpdateException: Duplicate entry 'default' for key > 'UNIQUE_DATABASE' > at >
[jira] [Updated] (HIVE-12341) LLAP: add security to daemon protocol endpoint (excluding shuffle)
[ https://issues.apache.org/jira/browse/HIVE-12341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12341: Attachment: HIVE-12341.01.patch Fixing a small NPE in unsecure setup > LLAP: add security to daemon protocol endpoint (excluding shuffle) > -- > > Key: HIVE-12341 > URL: https://issues.apache.org/jira/browse/HIVE-12341 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12341.01.patch, HIVE-12341.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12341) LLAP: add security to daemon protocol endpoint (excluding shuffle)
[ https://issues.apache.org/jira/browse/HIVE-12341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12341: Attachment: (was: HIVE-12341.WIP.patch) > LLAP: add security to daemon protocol endpoint (excluding shuffle) > -- > > Key: HIVE-12341 > URL: https://issues.apache.org/jira/browse/HIVE-12341 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12341.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12420) WebHCat server throws NPE when you run command with -d user.name.
[
https://issues.apache.org/jira/browse/HIVE-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-12420:
--
Assignee: (was: Eugene Koifman)
> WebHCat server throws NPE when you run command with -d user.name.
> -
>
> Key: HIVE-12420
> URL: https://issues.apache.org/jira/browse/HIVE-12420
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
>Affects Versions: 2.0.0
>Reporter: Takahiko Saito
>
> When you run with '-d user.name', it failed with:
> {noformat}
> [hrt_qa@os-r6-bccslu-hive-1-r-5 ~]$ curl -s -d user.name=hrt_qa -d
> execute="drop table if exists templetontest_tab2;"
> http://os-r6-bccslu-hive-1-r-3.novalocal:20111/templeton/v1/ddl
>
>
>
> Error 500 Server Error
>
>
> HTTP ERROR: 500
> Problem accessing /templeton/v1/ddl. Reason:
> Server Error
> Powered by Jetty://
>
>
> {noformat}
> server log shows:
> {noformat}
> WARN | 16 Nov 2015 19:48:22,738 | org.eclipse.jetty.servlet.ServletHandler |
> /templeton/v1/ddl
> java.lang.NullPointerException
> at
> org.apache.http.client.utils.URLEncodedUtils.parse(URLEncodedUtils.java:235)
> ~[hive-jdbc-1.2.1.2.3.5.0-13-standalone.jar:1.2.1.2.3.5.0-13]
> at
> org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.getUserName(PseudoAuthenticationHandler.java:143)
> ~[hadoop-auth-2.6.0.jar:?]
> at
> org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:179)
> ~[hadoop-auth-2.6.0.jar:?]
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:507)
> ~[hadoop-auth-2.6.0.jar:?]
> at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:88)
> ~[hadoop-hdfs-2.7.1.2.3.5.0-13.jar:?]
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
> ~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at org.eclipse.jetty.server.Server.handle(Server.java:349)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:925)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
> WARN | 16 Nov 2015 19:48:22,738 | org.eclipse.jetty.servlet.ServletHandler |
> /templeton/v1/ddl
> java.lang.NullPointerException
> at
> org.apache.http.client.utils.URLEncodedUtils.parse(URLEncodedUtils.java:235)
> ~[hive-jdbc-1.2.1.2.3.5.0-13-standalone.jar:1.2.1.2.3.5.0-13]
> at
>
[jira] [Commented] (HIVE-12384) Union Operator may produce incorrect result on TEZ
[ https://issues.apache.org/jira/browse/HIVE-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007373#comment-15007373 ] Laljo John Pullokkaran commented on HIVE-12384: --- Filed HIVE-12423 to remove union operator. Decoupling it from current patch. > Union Operator may produce incorrect result on TEZ > -- > > Key: HIVE-12384 > URL: https://issues.apache.org/jira/browse/HIVE-12384 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 0.14.0, 1.0.0, 1.1.0, 1.0.1, 1.2.1 >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Attachments: HIVE-12384.1.patch, HIVE-12384.2.patch, > HIVE-12384.3.patch > > > Union queries may produce incorrect result on TEZ. > TEZ removes union op, thus might loose the implicit cast in union op. > Reproduction test case: > set hive.cbo.enable=false; > set hive.execution.engine=tez; > select (x/sum(x) over()) as y from(select cast(1 as decimal(10,0)) as x > from (select * from src limit 2)s1 union all select cast(1 as decimal(10,0)) > x from (select * from src limit 2) s2 union all select '1' x from > (select * from src limit 2) s3)u order by y; > select (x/sum(x) over()) as y from(select cast(1 as decimal(10,0)) as x from > (select * from src limit 2)s1 union all select cast(1 as decimal(10,0)) x > from (select * from src limit 2) s2 union all select cast (null as string) x > from (select * from src limit 2) s3)u order by y; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12175) Upgrade Kryo version to 3.0.x
[
https://issues.apache.org/jira/browse/HIVE-12175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-12175:
-
Attachment: HIVE-12175.4.patch
> Upgrade Kryo version to 3.0.x
> -
>
> Key: HIVE-12175
> URL: https://issues.apache.org/jira/browse/HIVE-12175
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 2.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12175.1.patch, HIVE-12175.2.patch,
> HIVE-12175.3.patch, HIVE-12175.3.patch, HIVE-12175.4.patch
>
>
> Current version of kryo (2.22) has some issue (refer exception below and in
> HIVE-12174) with serializing ArrayLists generated using Arrays.asList(). We
> need to either replace all occurrences of Arrays.asList() or change the
> current StdInstantiatorStrategy. This issue is fixed in later versions and
> kryo community recommends using DefaultInstantiatorStrategy with fallback to
> StdInstantiatorStrategy. More discussion about this issue is here
> https://github.com/EsotericSoftware/kryo/issues/216. Alternatively, custom
> serilization/deserilization class can be provided for Arrays.asList.
> Also, kryo 3.0 introduced unsafe based serialization which claims to have
> much better performance for certain types of serialization.
> Exception:
> {code}
> Caused by: java.lang.NullPointerException
> at java.util.Arrays$ArrayList.size(Arrays.java:2847)
> at java.util.AbstractList.add(AbstractList.java:108)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> ... 57 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11981) ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)
[ https://issues.apache.org/jira/browse/HIVE-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007490#comment-15007490 ] Prasanth Jayachandran commented on HIVE-11981: -- There is only one unaddressed comment in Vectorizer.java. Other than that patch looks good to me. +1 > ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized) > -- > > Key: HIVE-11981 > URL: https://issues.apache.org/jira/browse/HIVE-11981 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-11981.01.patch, HIVE-11981.02.patch, > HIVE-11981.03.patch, HIVE-11981.05.patch, HIVE-11981.06.patch, > HIVE-11981.07.patch, HIVE-11981.08.patch, HIVE-11981.09.patch, > HIVE-11981.091.patch, HIVE-11981.092.patch, HIVE-11981.093.patch, > HIVE-11981.094.patch, HIVE-11981.095.patch, HIVE-11981.096.patch, > HIVE-11981.097.patch, HIVE-11981.098.patch, HIVE-11981.099.patch, > HIVE-11981.0991.patch, ORC Schema Evolution Issues.docx > > > High priority issues with schema evolution for the ORC file format. > Schema evolution here is limited to adding new columns and a few cases of > column type-widening (e.g. int to bigint). > Renaming columns, deleting column, moving columns and other schema evolution > were not pursued due to lack of importance and lack of time. Also, it > appears a much more sophisticated metadata would be needed to support them. > The biggest issues for users have been adding new columns for ACID table > (HIVE-11421 Support Schema evolution for ACID tables) and vectorization > (HIVE-10598 Vectorization borks when column is added to table). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12017) Do not disable CBO by default when number of joins in a query is equal or less than 1
[
https://issues.apache.org/jira/browse/HIVE-12017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007492#comment-15007492
]
Hive QA commented on HIVE-12017:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12772543/HIVE-12017.10.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9783 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6050/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6050/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6050/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12772543 - PreCommit-HIVE-TRUNK-Build
> Do not disable CBO by default when number of joins in a query is equal or
> less than 1
> -
>
> Key: HIVE-12017
> URL: https://issues.apache.org/jira/browse/HIVE-12017
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 2.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-12017.01.patch, HIVE-12017.02.patch,
> HIVE-12017.03.patch, HIVE-12017.04.patch, HIVE-12017.05.patch,
> HIVE-12017.06.patch, HIVE-12017.07.patch, HIVE-12017.08.patch,
> HIVE-12017.09.patch, HIVE-12017.10.patch
>
>
> Instead, we could disable some parts of CBO that are not relevant if the
> query contains 1 or 0 joins. Implementation should be able to define easily
> other query patterns for which we might disable some parts of CBO (in case we
> want to do it in the future).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12426) SessionState.get().getHiveOperation() is unreliable in HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-12426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] André Kelpe updated HIVE-12426: --- Description: SessionState.get().getHiveOperation() is supposed to allow easy access to the current kind of operation. The information however is incorrect in the HiveServer2 case, since the setting of the value is happening in a different thread than the code than the actual user session is in. Since this is a ThreadLocal there is no way to retrieve the correct value, if it is set by a different thread. (was: SessionState.get().getHiveOperation() is supposed to allow easy access to the current kind of operation. The information however is incorrect in the HiveServer2 case, since the setting of the value is happening in a different thread than the code that the actual user session is in. Since this is a ThreadLocal there is no way to retrieve the correct value, if it is set by a different thread.) > SessionState.get().getHiveOperation() is unreliable in HiveServer2 > -- > > Key: HIVE-12426 > URL: https://issues.apache.org/jira/browse/HIVE-12426 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.2.1 >Reporter: André Kelpe >Assignee: Vaibhav Gumashta > > SessionState.get().getHiveOperation() is supposed to allow easy access to the > current kind of operation. The information however is incorrect in the > HiveServer2 case, since the setting of the value is happening in a different > thread than the code than the actual user session is in. Since this is a > ThreadLocal there is no way to retrieve the correct value, if it is set by a > different thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007381#comment-15007381 ] Siddharth Seth commented on HIVE-11358: --- Is there a way to group the LLAP variables separately in HiveConf ? The mechanics for validation and parameter access offered by HiveConf are definitely useful. At the same time, it would be useful to separate out LLAP parameters (and within that separate out client / daemon config properties at a later point). llap-daemon-site still exists - but there will be confusion about what goes into this, vs what goes into hive-site.xml. Does the daemon require a hive-site while starting up ? With YARN and HDFS - a perpetual source of confusion has been separation of client vs server configs. This ends up with servers and clients using the same config files (sane path), and the same property names. As a result - knowing whether a parameter change affects the server / client is confusing, also where a change is required is confusing. Separating the configs within HiveConf would be useful for something like this. Functionally, the patch looks good. > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12271) Add metrics around HS2 query execution and job submission for Hive
[ https://issues.apache.org/jira/browse/HIVE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007438#comment-15007438 ] Szehon Ho commented on HIVE-12271: -- You mean in LegacyMetrics right? Yea I tried removing it, but it failed some of the existing tests, seemed they had designed so the scope stays forever (they have an API called reopen scope), so I put it back. Thanks > Add metrics around HS2 query execution and job submission for Hive > --- > > Key: HIVE-12271 > URL: https://issues.apache.org/jira/browse/HIVE-12271 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 1.2.1 >Reporter: Lenni Kuff >Assignee: Szehon Ho > Attachments: HIVE-12271.2.patch, HIVE-12271.patch > > > We should add more metrics around query execution. Specifically: > * Number of in-use worker threads > * Number of in-use async threads > * Number of queries waiting for compilation > * Stats for query planning / compilation time > * Stats for total job submission time > * Others? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11948) Investigate TxnHandler and CompactionTxnHandler to see where we improve concurrency
[
https://issues.apache.org/jira/browse/HIVE-11948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007498#comment-15007498
]
Alan Gates commented on HIVE-11948:
---
In TxnHandler, around line 495:
bq. Why doesn't this get a txnid as parameter? The caller should either know
the txnid or know there isn't one. Either way getTxnIdFromLockId() will not be
needed. This would be a Thrift change.
We should file a JIRA for that. Same goes for comment at line 501. We might
just want to file an umbrella JIRA saying "take care of TODOs in TxnHandler and
CompactionTxnHandler" and then we can file JIRAs for individual ones.
TxnHandler, line 522:
{code}
if (txnid > 0) {
heartbeatTxn(dbConn, txnid);
}
else {
heartbeatLock(dbConn, extLockId);
}
{code}
Previously the code was:
{code}
heartbeatLock(dbConn, extLockId);
...
if (txnid > 0) heartbeatTxn(dbConn, txnid);
{code}
You've changed the logic so that locks will only be heartbeat if there is no
transaction. I don't think that's what you want.
TxnHander unlock(), around line 581, you moved the check that a lock is
associated with a txn below the failure detection. Are you depending on the db
constraints to catch that the lock entry can't be deleted because a txn it is
associated with still exists? If so, that should be commented. If not, this
is a logical error as we want to make sure never to unlock a lock associated
with a txn.
TxnHandler.getRequiredIsolationLevel(), line 2270
{code}
if(dbProduct == null) {
Connection tmp = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
determineDatabaseProduct(tmp);
closeDbConn(tmp);
}
{code}
We should modify determineDatabaseProduct to accept null for the connection and
create its own rather than repeating this logic anytime we don't have a
connection.
> Investigate TxnHandler and CompactionTxnHandler to see where we improve
> concurrency
> ---
>
> Key: HIVE-11948
> URL: https://issues.apache.org/jira/browse/HIVE-11948
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-11948.3.patch, HIVE-11948.4.patch,
> HIVE-11948.5.patch, HIVE-11948.6.patch, HIVE-11948.7.patch, HIVE-11948.patch
>
>
> at least some operations (or parts of operations) can run at READ_COMMITTED.
> CompactionTxnHandler.setRunAs()
> CompactionTxnHandler.findNextToCompact()
> if update stmt includes cq_state = '" + INITIATED_STATE + "'" in WHERE clause
> and logic to look for "next" candidate
> CompactionTxnHandler.markCompacted()
> perhaps add cq_state=WORKING_STATE in Where clause (mostly as an extra
> consistency check)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12424) Make use of Kryo's Object-to-Object deep copy
[ https://issues.apache.org/jira/browse/HIVE-12424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12424: - Summary: Make use of Kryo's Object-to-Object deep copy (was: Make use Kryo Object-to-Object deep copy) > Make use of Kryo's Object-to-Object deep copy > - > > Key: HIVE-12424 > URL: https://issues.apache.org/jira/browse/HIVE-12424 > Project: Hive > Issue Type: Sub-task > Components: Serializers/Deserializers >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > > Currently, plan serialization and operator tree serialization uses Object -> > bytes -> Object approach for deep copy. It also uses ByteArrayOutputStream as > intermediate buffer whose write method is synchronized. Similarly read from > ByteArrayInputStream is also synchronized. Also Utilities.clonePlan() creates > a new HiveConf object that scans through conf directories and adds site.xml > which is an expensive operation. All these can be avoided using Kryo's Object > -> Object deep copy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6113) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
[
https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007158#comment-15007158
]
Hive QA commented on HIVE-6113:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12772479/HIVE-6113.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 898 failed/errored test(s), 5641 tests
executed
*Failed tests:*
{noformat}
TestFilterHooks - did not produce a TEST-*.xml file
TestHS2ImpersonationWithRemoteMS - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMetastoreAuthorizationProvider - did not produce a TEST-*.xml file
TestPartitionNameWhitelistValidation - did not produce a TEST-*.xml file
TestRemoteHiveMetaStore - did not produce a TEST-*.xml file
TestSetUGIOnBothClientServer - did not produce a TEST-*.xml file
TestSetUGIOnOnlyClient - did not produce a TEST-*.xml file
TestSetUGIOnOnlyServer - did not produce a TEST-*.xml file
TestStorageBasedMetastoreAuthorizationDrops - did not produce a TEST-*.xml file
TestStorageBasedMetastoreAuthorizationReads - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.initializationError
org.apache.hadoop.hive.cli.TestCliDriverMethods.testProcessSelectDatabase
org.apache.hadoop.hive.cli.TestCliDriverMethods.testQuit
org.apache.hadoop.hive.cli.TestCliDriverMethods.testRun
org.apache.hadoop.hive.cli.TestCliDriverMethods.testprocessInitFiles
org.apache.hadoop.hive.cli.TestCliSessionState.testgetDbName
org.apache.hadoop.hive.cli.TestCompareCliDriver.initializationError
org.apache.hadoop.hive.cli.TestContribCliDriver.initializationError
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.initializationError
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.initializationError
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_external_table_ppd
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_external_table_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_map_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_map_queries_prefix
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_storage_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_handler_bulk
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_handler_snapshot
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_joins
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_null_first_col
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_join
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_key_range
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_pushdown
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_scan_params
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_single_sourced_multi_insert
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats3
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_timestamp
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_timestamp_format
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_ppd_key_ranges
org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_cascade_dbdrop
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_cascade_dbdrop_hadoop20
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_generatehfiles_require_family_path
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.initializationError
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.initializationError
org.apache.hadoop.hive.cli.TestMinimrCliDriver.initializationError
org.apache.hadoop.hive.cli.TestNegativeCliDriver.initializationError
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.initializationError
org.apache.hadoop.hive.cli.TestSparkCliDriver.initializationError
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.initializationError
org.apache.hadoop.hive.hooks.TestHs2Hooks.org.apache.hadoop.hive.hooks.TestHs2Hooks
org.apache.hadoop.hive.metastore.TestAdminUser.testCreateAdminNAddUser
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInEmbed.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInEmbed
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.testCreateRole
[jira] [Updated] (HIVE-12384) Union Operator may produce incorrect result on TEZ
[ https://issues.apache.org/jira/browse/HIVE-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-12384: -- Attachment: HIVE-12384.3.patch > Union Operator may produce incorrect result on TEZ > -- > > Key: HIVE-12384 > URL: https://issues.apache.org/jira/browse/HIVE-12384 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 0.14.0, 1.0.0, 1.1.0, 1.0.1, 1.2.1 >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Attachments: HIVE-12384.1.patch, HIVE-12384.2.patch, > HIVE-12384.3.patch > > > Union queries may produce incorrect result on TEZ. > TEZ removes union op, thus might loose the implicit cast in union op. > Reproduction test case: > set hive.cbo.enable=false; > set hive.execution.engine=tez; > select (x/sum(x) over()) as y from(select cast(1 as decimal(10,0)) as x > from (select * from src limit 2)s1 union all select cast(1 as decimal(10,0)) > x from (select * from src limit 2) s2 union all select '1' x from > (select * from src limit 2) s3)u order by y; > select (x/sum(x) over()) as y from(select cast(1 as decimal(10,0)) as x from > (select * from src limit 2)s1 union all select cast(1 as decimal(10,0)) x > from (select * from src limit 2) s2 union all select cast (null as string) x > from (select * from src limit 2) s3)u order by y; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11358: Attachment: HIVE-11358.01.patch Fixed the names (with fallback), fixed the setting, added descriptions (some will be documented by someone else in a separate patch). > LLAP: move LlapConfiguration into HiveConf > -- > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-12419) hive.log.trace.id needs to be whitelisted
[ https://issues.apache.org/jira/browse/HIVE-12419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepesh Khandelwal resolved HIVE-12419. --- Resolution: Not A Problem Looks like common/src/java/org/apache/hadoop/hive/conf/HiveConf.java already has safe regex for hive.log.* so should work without any issues. > hive.log.trace.id needs to be whitelisted > - > > Key: HIVE-12419 > URL: https://issues.apache.org/jira/browse/HIVE-12419 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Deepesh Khandelwal >Assignee: Deepesh Khandelwal > Fix For: 2.0.0 > > > HIVE-12249 introduces hive.log.trace.id as part of improving logging for hive > queries. The property needs to be added to SQL Std Auth whitelisted > properties list to be usable with HiveServer2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6113) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007204#comment-15007204 ] Ashutosh Chauhan commented on HIVE-6113: 2.0 is a good time to do this. Lets try to get it in. Web page pointed out by Sushanth does list few items we need to be wary of. > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > -- > > Key: HIVE-6113 > URL: https://issues.apache.org/jira/browse/HIVE-6113 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.0.0, 1.2.1 > Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0 >Reporter: William Stone >Assignee: Oleksiy Sayankin >Priority: Critical > Labels: HiveMetaStoreClient, metastore, unable_instantiate > Attachments: HIVE-6113.patch > > > When I exccute SQL "use fdm; desc formatted fdm.tableName;" in python, throw > Error as followed. > but when I tryit again , It will success. > 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - > org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143) > at > org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128) > at > org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:197) > Caused by: java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) > at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383) > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139) > ... 20 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) > ... 25 more > Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes > to datastore > NestedThrowables: > java.sql.BatchUpdateException: Duplicate entry 'default' for key > 'UNIQUE_DATABASE' > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) > at > org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) > at > org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:358) > at > org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:404) > at
[jira] [Updated] (HIVE-12420) WebHCat server throws NPE when you run command with -d user.name.
[
https://issues.apache.org/jira/browse/HIVE-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-12420:
--
Summary: WebHCat server throws NPE when you run command with -d user.name.
(was: Uploaded image for project: 'Bug DB' Bug DBBUG-48161 WebHCat server
throws NPE when you run command with -d user.name.)
> WebHCat server throws NPE when you run command with -d user.name.
> -
>
> Key: HIVE-12420
> URL: https://issues.apache.org/jira/browse/HIVE-12420
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
>Affects Versions: 2.0.0
>Reporter: Takahiko Saito
>Assignee: Eugene Koifman
>
> When you run with '-d user.name', it failed with:
> {noformat}
> [hrt_qa@os-r6-bccslu-hive-1-r-5 ~]$ curl -s -d user.name=hrt_qa -d
> execute="drop table if exists templetontest_tab2;"
> http://os-r6-bccslu-hive-1-r-3.novalocal:20111/templeton/v1/ddl
>
>
>
> Error 500 Server Error
>
>
> HTTP ERROR: 500
> Problem accessing /templeton/v1/ddl. Reason:
> Server Error
> Powered by Jetty://
>
>
> {noformat}
> server log shows:
> {noformat}
> WARN | 16 Nov 2015 19:48:22,738 | org.eclipse.jetty.servlet.ServletHandler |
> /templeton/v1/ddl
> java.lang.NullPointerException
> at
> org.apache.http.client.utils.URLEncodedUtils.parse(URLEncodedUtils.java:235)
> ~[hive-jdbc-1.2.1.2.3.5.0-13-standalone.jar:1.2.1.2.3.5.0-13]
> at
> org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.getUserName(PseudoAuthenticationHandler.java:143)
> ~[hadoop-auth-2.6.0.jar:?]
> at
> org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:179)
> ~[hadoop-auth-2.6.0.jar:?]
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:507)
> ~[hadoop-auth-2.6.0.jar:?]
> at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:88)
> ~[hadoop-hdfs-2.7.1.2.3.5.0-13.jar:?]
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
> ~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at org.eclipse.jetty.server.Server.handle(Server.java:349)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:925)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
> WARN | 16 Nov 2015 19:48:22,738 | org.eclipse.jetty.servlet.ServletHandler |
> /templeton/v1/ddl
>
[jira] [Updated] (HIVE-12271) Add metrics around HS2 query execution and job submission for Hive
[ https://issues.apache.org/jira/browse/HIVE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-12271: - Attachment: HIVE-12271.2.patch Rebase patch. > Add metrics around HS2 query execution and job submission for Hive > --- > > Key: HIVE-12271 > URL: https://issues.apache.org/jira/browse/HIVE-12271 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 1.2.1 >Reporter: Lenni Kuff >Assignee: Szehon Ho > Attachments: HIVE-12271.2.patch, HIVE-12271.patch > > > We should add more metrics around query execution. Specifically: > * Number of in-use worker threads > * Number of in-use async threads > * Number of queries waiting for compilation > * Stats for query planning / compilation time > * Stats for total job submission time > * Others? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12271) Add metrics around HS2 query execution and job submission for Hive
[ https://issues.apache.org/jira/browse/HIVE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007163#comment-15007163 ] Szehon Ho commented on HIVE-12271: -- Tests failures do not seem related. [~jxiang] do you have time to help review? > Add metrics around HS2 query execution and job submission for Hive > --- > > Key: HIVE-12271 > URL: https://issues.apache.org/jira/browse/HIVE-12271 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 1.2.1 >Reporter: Lenni Kuff >Assignee: Szehon Ho > Attachments: HIVE-12271.2.patch, HIVE-12271.patch > > > We should add more metrics around query execution. Specifically: > * Number of in-use worker threads > * Number of in-use async threads > * Number of queries waiting for compilation > * Stats for query planning / compilation time > * Stats for total job submission time > * Others? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-11036) Race condition in DataNucleus makes Metastore to hang
[ https://issues.apache.org/jira/browse/HIVE-11036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-11036. - Resolution: Duplicate Marking this as duplicate, since HIVE-6113 has more discussion. > Race condition in DataNucleus makes Metastore to hang > - > > Key: HIVE-11036 > URL: https://issues.apache.org/jira/browse/HIVE-11036 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Takahiko Saito >Assignee: Ashutosh Chauhan > Attachments: HIVE-11036.patch > > > Under moderate to high concurrent query workload Metastore gets deadlocked in > DataNucleus -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12384) Union Operator may produce incorrect result on TEZ
[ https://issues.apache.org/jira/browse/HIVE-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007230#comment-15007230 ] Ashutosh Chauhan commented on HIVE-12384: - I think better fix is to always insert a select operator (whenever there is a need for casting) regardless of execution engine type. We should get rid of Union operator at runtime altogether (since its not doing any useful computation). Planning needs to be as independent of run time execution engine as much as possible. > Union Operator may produce incorrect result on TEZ > -- > > Key: HIVE-12384 > URL: https://issues.apache.org/jira/browse/HIVE-12384 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 0.14.0, 1.0.0, 1.1.0, 1.0.1, 1.2.1 >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Attachments: HIVE-12384.1.patch, HIVE-12384.2.patch, > HIVE-12384.3.patch > > > Union queries may produce incorrect result on TEZ. > TEZ removes union op, thus might loose the implicit cast in union op. > Reproduction test case: > set hive.cbo.enable=false; > set hive.execution.engine=tez; > select (x/sum(x) over()) as y from(select cast(1 as decimal(10,0)) as x > from (select * from src limit 2)s1 union all select cast(1 as decimal(10,0)) > x from (select * from src limit 2) s2 union all select '1' x from > (select * from src limit 2) s3)u order by y; > select (x/sum(x) over()) as y from(select cast(1 as decimal(10,0)) as x from > (select * from src limit 2)s1 union all select cast(1 as decimal(10,0)) x > from (select * from src limit 2) s2 union all select cast (null as string) x > from (select * from src limit 2) s3)u order by y; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11358: Summary: LLAP: move LlapConfiguration into HiveConf and document the settings (was: LLAP: move LlapConfiguration into HiveConf) > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11684) Implement limit pushdown through outer join in CBO
[ https://issues.apache.org/jira/browse/HIVE-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-11684: --- Attachment: HIVE-11684.20.patch > Implement limit pushdown through outer join in CBO > -- > > Key: HIVE-11684 > URL: https://issues.apache.org/jira/browse/HIVE-11684 > Project: Hive > Issue Type: New Feature > Components: CBO >Affects Versions: 2.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-11684.01.patch, HIVE-11684.02.patch, > HIVE-11684.03.patch, HIVE-11684.04.patch, HIVE-11684.05.patch, > HIVE-11684.07.patch, HIVE-11684.08.patch, HIVE-11684.09.patch, > HIVE-11684.10.patch, HIVE-11684.11.patch, HIVE-11684.12.patch, > HIVE-11684.12.patch, HIVE-11684.14.patch, HIVE-11684.15.patch, > HIVE-11684.16.patch, HIVE-11684.17.patch, HIVE-11684.18.patch, > HIVE-11684.19.patch, HIVE-11684.20.patch, HIVE-11684.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11358: Attachment: HIVE-11358.01.patch Fix the tests > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.01.patch, > HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12271) Add metrics around HS2 query execution and job submission for Hive
[ https://issues.apache.org/jira/browse/HIVE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007305#comment-15007305 ] Jimmy Xiang commented on HIVE-12271: In one endStoredScope method, you removed the named scope, while in another, you didn't. Is this intentional? Other than that, the patch looks good to me. > Add metrics around HS2 query execution and job submission for Hive > --- > > Key: HIVE-12271 > URL: https://issues.apache.org/jira/browse/HIVE-12271 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 1.2.1 >Reporter: Lenni Kuff >Assignee: Szehon Ho > Attachments: HIVE-12271.2.patch, HIVE-12271.patch > > > We should add more metrics around query execution. Specifically: > * Number of in-use worker threads > * Number of in-use async threads > * Number of queries waiting for compilation > * Stats for query planning / compilation time > * Stats for total job submission time > * Others? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007304#comment-15007304 ] Sergey Shelukhin edited comment on HIVE-11358 at 11/16/15 8:58 PM: --- Fixed the tests was (Author: sershe): Fix the tests > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.01.patch, > HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12341) LLAP: add security to daemon protocol endpoint (excluding shuffle)
[ https://issues.apache.org/jira/browse/HIVE-12341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12341: Attachment: (was: HIVE-12341.WIP.nogen.patch) > LLAP: add security to daemon protocol endpoint (excluding shuffle) > -- > > Key: HIVE-12341 > URL: https://issues.apache.org/jira/browse/HIVE-12341 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12341.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11358: Attachment: HIVE-11358.01.patch > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11358: Attachment: (was: HIVE-11358.01.patch) > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12384) Union Operator may produce incorrect result on TEZ
[ https://issues.apache.org/jira/browse/HIVE-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007290#comment-15007290 ] Ashutosh Chauhan commented on HIVE-12384: - Which penalty you are thinking about ? Fix I am envisioning is one in which we get rid of Union operator altogether. Casting (if any needed) is done by Select Operator. > Union Operator may produce incorrect result on TEZ > -- > > Key: HIVE-12384 > URL: https://issues.apache.org/jira/browse/HIVE-12384 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 0.14.0, 1.0.0, 1.1.0, 1.0.1, 1.2.1 >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Attachments: HIVE-12384.1.patch, HIVE-12384.2.patch, > HIVE-12384.3.patch > > > Union queries may produce incorrect result on TEZ. > TEZ removes union op, thus might loose the implicit cast in union op. > Reproduction test case: > set hive.cbo.enable=false; > set hive.execution.engine=tez; > select (x/sum(x) over()) as y from(select cast(1 as decimal(10,0)) as x > from (select * from src limit 2)s1 union all select cast(1 as decimal(10,0)) > x from (select * from src limit 2) s2 union all select '1' x from > (select * from src limit 2) s3)u order by y; > select (x/sum(x) over()) as y from(select cast(1 as decimal(10,0)) as x from > (select * from src limit 2)s1 union all select cast(1 as decimal(10,0)) x > from (select * from src limit 2) s2 union all select cast (null as string) x > from (select * from src limit 2) s3)u order by y; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12384) Union Operator may produce incorrect result on TEZ
[ https://issues.apache.org/jira/browse/HIVE-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007254#comment-15007254 ] Laljo John Pullokkaran commented on HIVE-12384: --- The counter argument is this: We already have an operator that handles type casting in MR. So why penalize MR? > Union Operator may produce incorrect result on TEZ > -- > > Key: HIVE-12384 > URL: https://issues.apache.org/jira/browse/HIVE-12384 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 0.14.0, 1.0.0, 1.1.0, 1.0.1, 1.2.1 >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Attachments: HIVE-12384.1.patch, HIVE-12384.2.patch, > HIVE-12384.3.patch > > > Union queries may produce incorrect result on TEZ. > TEZ removes union op, thus might loose the implicit cast in union op. > Reproduction test case: > set hive.cbo.enable=false; > set hive.execution.engine=tez; > select (x/sum(x) over()) as y from(select cast(1 as decimal(10,0)) as x > from (select * from src limit 2)s1 union all select cast(1 as decimal(10,0)) > x from (select * from src limit 2) s2 union all select '1' x from > (select * from src limit 2) s3)u order by y; > select (x/sum(x) over()) as y from(select cast(1 as decimal(10,0)) as x from > (select * from src limit 2)s1 union all select cast(1 as decimal(10,0)) x > from (select * from src limit 2) s2 union all select cast (null as string) x > from (select * from src limit 2) s3)u order by y; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11358: Attachment: (was: HIVE-11358.01.patch) > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12384) Union Operator may produce incorrect result on TEZ
[ https://issues.apache.org/jira/browse/HIVE-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007328#comment-15007328 ] Laljo John Pullokkaran commented on HIVE-12384: --- Penalty is the cost of additional select (which in case of MR) is not required. Not to harp on this, this cost shouldn't be that significant. > Union Operator may produce incorrect result on TEZ > -- > > Key: HIVE-12384 > URL: https://issues.apache.org/jira/browse/HIVE-12384 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 0.14.0, 1.0.0, 1.1.0, 1.0.1, 1.2.1 >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Attachments: HIVE-12384.1.patch, HIVE-12384.2.patch, > HIVE-12384.3.patch > > > Union queries may produce incorrect result on TEZ. > TEZ removes union op, thus might loose the implicit cast in union op. > Reproduction test case: > set hive.cbo.enable=false; > set hive.execution.engine=tez; > select (x/sum(x) over()) as y from(select cast(1 as decimal(10,0)) as x > from (select * from src limit 2)s1 union all select cast(1 as decimal(10,0)) > x from (select * from src limit 2) s2 union all select '1' x from > (select * from src limit 2) s3)u order by y; > select (x/sum(x) over()) as y from(select cast(1 as decimal(10,0)) as x from > (select * from src limit 2)s1 union all select cast(1 as decimal(10,0)) x > from (select * from src limit 2) s2 union all select cast (null as string) x > from (select * from src limit 2) s3)u order by y; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12271) Add metrics around HS2 query execution and job submission for Hive
[ https://issues.apache.org/jira/browse/HIVE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007599#comment-15007599 ] Jimmy Xiang commented on HIVE-12271: I see. Thanks for the explanation. +1 > Add metrics around HS2 query execution and job submission for Hive > --- > > Key: HIVE-12271 > URL: https://issues.apache.org/jira/browse/HIVE-12271 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 1.2.1 >Reporter: Lenni Kuff >Assignee: Szehon Ho > Attachments: HIVE-12271.2.patch, HIVE-12271.patch > > > We should add more metrics around query execution. Specifically: > * Number of in-use worker threads > * Number of in-use async threads > * Number of queries waiting for compilation > * Stats for query planning / compilation time > * Stats for total job submission time > * Others? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12427) Improve HiveServer2 *DBC ResultSet performance
[ https://issues.apache.org/jira/browse/HIVE-12427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-12427: Summary: Improve HiveServer2 *DBC ResultSet performance (was: Improve HiveServer2 JDBC ResultSet performance) > Improve HiveServer2 *DBC ResultSet performance > -- > > Key: HIVE-12427 > URL: https://issues.apache.org/jira/browse/HIVE-12427 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, ODBC >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12427) Improve HiveServer2 *DBC ResultSet performance
[ https://issues.apache.org/jira/browse/HIVE-12427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007621#comment-15007621 ] Vaibhav Gumashta commented on HIVE-12427: - We'll add the details of the proposed changes shortly. > Improve HiveServer2 *DBC ResultSet performance > -- > > Key: HIVE-12427 > URL: https://issues.apache.org/jira/browse/HIVE-12427 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, ODBC >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12049) Provide an option to write serialized thrift objects in final tasks
[ https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-12049: Affects Version/s: (was: 1.2.0) > Provide an option to write serialized thrift objects in final tasks > --- > > Key: HIVE-12049 > URL: https://issues.apache.org/jira/browse/HIVE-12049 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Rohit Dholakia >Assignee: Rohit Dholakia > > For each fetch request to HiveServer2, we pay the penalty of deserializing > the row objects and translating them into a different representation suitable > for the RPC transfer. In a moderate to high concurrency scenarios, this can > result in significant CPU and memory wastage. By having each task write the > appropriate thrift objects to the output files, HiveServer2 can simply stream > a batch of rows on the wire without incurring any of the additional cost of > deserialization and translation. > This can be implemented by writing a new SerDe, which the FileSinkOperator > can use to write thrift formatted row batches to the output file. Using the > pluggable property of the hive.query.result.fileformat, we can set it to use > SequenceFile and write a batch of thrift formatted rows as a value blob. The > FetchTask can now simply read the blob and send it over the wire. On the > client side, the *DBC driver can read the blob and since it is already > formatted in the way it expects, it can continue building the ResultSet the > way it does in the current implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12049) Provide an option to write serialized thrift objects in final tasks
[ https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-12049: Description: For each fetch request to HiveServer2, we pay the penalty of deserializing the row objects and translating them into a different representation suitable for the RPC transfer. In a moderate to high concurrency scenarios, this can result in significant CPU and memory wastage. By having each task write the appropriate thrift objects to the output files, HiveServer2 can simply stream a batch of rows on the wire without incurring any of the additional cost of deserialization and translation. This can be implemented by writing a new SerDe, which the FileSinkOperator can use to write thrift formatted row batches to the output file. Using the pluggable property of the hive.query.result.fileformat, we can set it to use SequenceFile and write a batch of thrift formatted rows as a value blob. The FetchTask can now simply read the blob and send it over the wire. On the client side, the *DBC driver can read the blob and since it is already formatted in the way it expects, it can continue building the ResultSet the way it does in the current implementation. > Provide an option to write serialized thrift objects in final tasks > --- > > Key: HIVE-12049 > URL: https://issues.apache.org/jira/browse/HIVE-12049 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Rohit Dholakia >Assignee: Rohit Dholakia > > For each fetch request to HiveServer2, we pay the penalty of deserializing > the row objects and translating them into a different representation suitable > for the RPC transfer. In a moderate to high concurrency scenarios, this can > result in significant CPU and memory wastage. By having each task write the > appropriate thrift objects to the output files, HiveServer2 can simply stream > a batch of rows on the wire without incurring any of the additional cost of > deserialization and translation. > This can be implemented by writing a new SerDe, which the FileSinkOperator > can use to write thrift formatted row batches to the output file. Using the > pluggable property of the hive.query.result.fileformat, we can set it to use > SequenceFile and write a batch of thrift formatted rows as a value blob. The > FetchTask can now simply read the blob and send it over the wire. On the > client side, the *DBC driver can read the blob and since it is already > formatted in the way it expects, it can continue building the ResultSet the > way it does in the current implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12427) Improve HiveServer2 *DBC ResultSet performance
[ https://issues.apache.org/jira/browse/HIVE-12427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007641#comment-15007641 ] Vaibhav Gumashta commented on HIVE-12427: - Added description in subtasks. > Improve HiveServer2 *DBC ResultSet performance > -- > > Key: HIVE-12427 > URL: https://issues.apache.org/jira/browse/HIVE-12427 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, ODBC >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-12428) Provide an option for HiveServer2 to stream serialized thrift results when they are available
[ https://issues.apache.org/jira/browse/HIVE-12428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta reassigned HIVE-12428: --- Assignee: Vaibhav Gumashta > Provide an option for HiveServer2 to stream serialized thrift results when > they are available > - > > Key: HIVE-12428 > URL: https://issues.apache.org/jira/browse/HIVE-12428 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > > Once HIVE-12049 is complete, we will have an option on HS2 side to avoid > paying the deserialization penalty and simple stream batches of rows on the > wire which the *DBC driver can appropriately interpret. This can be done by > changing the FetchOperator such that when it has the thrift formatted row > sets available, it uses the fastpath. For queries where tasks don't write > thrift rowsets, the current implementation will suffice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12049) Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-12049:
Description:
For each fetch request to HiveServer2, we pay the penalty of deserializing the
row objects and translating them into a different representation suitable for
the RPC transfer. In a moderate to high concurrency scenarios, this can result
in significant CPU and memory wastage. By having each task write the
appropriate thrift objects to the output files, HiveServer2 can simply stream a
batch of rows on the wire without incurring any of the additional cost of
deserialization and translation.
This can be implemented by writing a new SerDe, which the FileSinkOperator can
use to write thrift formatted row batches to the output file. Using the
pluggable property of the {{hive.query.result.fileformat}}, we can set it to
use SequenceFile and write a batch of thrift formatted rows as a value blob.
The FetchTask can now simply read the blob and send it over the wire. On the
client side, the *DBC driver can read the blob and since it is already
formatted in the way it expects, it can continue building the ResultSet the way
it does in the current implementation.
was:
For each fetch request to HiveServer2, we pay the penalty of deserializing the
row objects and translating them into a different representation suitable for
the RPC transfer. In a moderate to high concurrency scenarios, this can result
in significant CPU and memory wastage. By having each task write the
appropriate thrift objects to the output files, HiveServer2 can simply stream a
batch of rows on the wire without incurring any of the additional cost of
deserialization and translation.
This can be implemented by writing a new SerDe, which the FileSinkOperator can
use to write thrift formatted row batches to the output file. Using the
pluggable property of the hive.query.result.fileformat, we can set it to use
SequenceFile and write a batch of thrift formatted rows as a value blob. The
FetchTask can now simply read the blob and send it over the wire. On the client
side, the *DBC driver can read the blob and since it is already formatted in
the way it expects, it can continue building the ResultSet the way it does in
the current implementation.
> Provide an option to write serialized thrift objects in final tasks
> ---
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can read the blob and since it is already
> formatted in the way it expects, it can continue building the ResultSet the
> way it does in the current implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12422) LLAP: add security to Web UI endpoint
[ https://issues.apache.org/jira/browse/HIVE-12422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12422: Attachment: HIVE-12422.patch > LLAP: add security to Web UI endpoint > - > > Key: HIVE-12422 > URL: https://issues.apache.org/jira/browse/HIVE-12422 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12422.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12422) LLAP: add security to Web UI endpoint
[ https://issues.apache.org/jira/browse/HIVE-12422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12422: Description: NO PRECOMMIT TESTS > LLAP: add security to Web UI endpoint > - > > Key: HIVE-12422 > URL: https://issues.apache.org/jira/browse/HIVE-12422 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12422.patch > > > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11358) LLAP: move LlapConfiguration into HiveConf and document the settings
[ https://issues.apache.org/jira/browse/HIVE-11358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007709#comment-15007709 ] Sergey Shelukhin commented on HIVE-11358: - Hive has settings for several servers, in particular metastore and HS2, already, and they use the same config. Existing LLAP settings for IO also live there. HS2, CLI and metastore configs also have settings that are used in 2 or all 3 components. I am not sure there can be any way to separate them that is actually helpful. Users won't read the source to determine which config to use. One question is, do we need a separate daemon config, or should we just use hive-site.xml for everything? We can add a tool to generate configs for each scope by adding some scope setting to HiveConf vars, but it should be done in separate JIRA. > LLAP: move LlapConfiguration into HiveConf and document the settings > > > Key: HIVE-11358 > URL: https://issues.apache.org/jira/browse/HIVE-11358 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11358.01.patch, HIVE-11358.patch > > > Hive uses HiveConf for configuration. LlapConfiguration should be replaced > with parameters in HiveConf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12049) Provide an option to write serialized thrift objects in final tasks
[ https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-12049: Description: (was: As of now, a ColumnBasedSet is created from a RowSet and several new objects are created for this conversion in the hive-service layer. With this JIRA, the aim is to change the ListSinkOperator.process(row, tag) function so that the return object from the fetcher.convert() function will be converted to a thrift-compliant ColumnBasedSet so that significant amount of thrift object creation can be removed from the hive-service layer. ) > Provide an option to write serialized thrift objects in final tasks > --- > > Key: HIVE-12049 > URL: https://issues.apache.org/jira/browse/HIVE-12049 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 1.2.0 >Reporter: Rohit Dholakia >Assignee: Rohit Dholakia > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12049) Provide an option to write serialized thrift objects in final tasks
[ https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-12049: Summary: Provide an option to write serialized thrift objects in final tasks (was: Create ColumnBasedSets in hive-exec Driver instead of hive-service layer) > Provide an option to write serialized thrift objects in final tasks > --- > > Key: HIVE-12049 > URL: https://issues.apache.org/jira/browse/HIVE-12049 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 1.2.0 >Reporter: Rohit Dholakia >Assignee: Rohit Dholakia > > As of now, a ColumnBasedSet is created from a RowSet and several new objects > are created for this conversion in the hive-service layer. With this JIRA, > the aim is to change the ListSinkOperator.process(row, tag) function so that > the return object from the fetcher.convert() function will be converted to a > thrift-compliant ColumnBasedSet so that significant amount of thrift object > creation can be removed from the hive-service layer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12045) ClassNotFound for GenericUDF in "select distinct..." query (Hive on Spark)
[
https://issues.apache.org/jira/browse/HIVE-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007630#comment-15007630
]
Xuefu Zhang commented on HIVE-12045:
[~lirui], it seems that hive.log is generated using master. Could you migrate
your work on master instead? Spark branch seems having some test-related
issues. Thanks.
> ClassNotFound for GenericUDF in "select distinct..." query (Hive on Spark)
> --
>
> Key: HIVE-12045
> URL: https://issues.apache.org/jira/browse/HIVE-12045
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Environment: Cloudera QuickStart VM - CDH5.4.2
> beeline
>Reporter: Zsolt Tóth
>Assignee: Rui Li
> Attachments: HIVE-12045.1-spark.patch, HIVE-12045.2-spark.patch,
> example.jar, genUDF.patch, hive.log.gz
>
>
> If I execute the following query in beeline, I get ClassNotFoundException for
> the UDF class.
> {code}
> drop function myGenericUdf;
> create function myGenericUdf as 'org.example.myGenericUdf' using jar
> 'hdfs:///tmp/myudf.jar';
> select distinct myGenericUdf(1,2,1) from mytable;
> {code}
> In my example, myGenericUdf just looks for the 1st argument's value in the
> others and returns the index. I don't think this is related to the actual
> GenericUDF function.
> Note that:
> "select myGenericUdf(1,2,1) from mytable;" succeeds
> If I use the non-generic implementation of the same UDF, the select distinct
> call succeeds.
> StackTrace:
> {code}
> 15/10/06 05:20:25 ERROR exec.Utilities: Failed to load plan:
> hdfs://quickstart.cloudera:8020/tmp/hive/hive/f9de3f09-c12d-4528-9ee6-1f12932a14ae/hive_2015-10-06_05-20-07_438_6519207588897968406-20/-mr-10003/27cd7226-3e22-46f4-bddd-fb8fd4aa4b8d/map.xml:
> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
> class: org.example.myGenericUDF
> Serialization trace:
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.GroupByOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
> class: org.example.myGenericUDF
> Serialization trace:
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.GroupByOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
>
[jira] [Commented] (HIVE-12424) Make use of Kryo's Object-to-Object deep copy
[ https://issues.apache.org/jira/browse/HIVE-12424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007710#comment-15007710 ] Prasanth Jayachandran commented on HIVE-12424: -- This issue is currently blocked by https://github.com/EsotericSoftware/kryo/issues/148 which copies transient fields as well. maven jars has not reflected the changes yet. > Make use of Kryo's Object-to-Object deep copy > - > > Key: HIVE-12424 > URL: https://issues.apache.org/jira/browse/HIVE-12424 > Project: Hive > Issue Type: Sub-task > Components: Serializers/Deserializers >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > > Currently, plan serialization and operator tree serialization uses Object -> > bytes -> Object approach for deep copy. It also uses ByteArrayOutputStream as > intermediate buffer whose write method is synchronized. Similarly read from > ByteArrayInputStream is also synchronized. Also Utilities.clonePlan() creates > a new HiveConf object that scans through conf directories and adds site.xml > which is an expensive operation. All these can be avoided using Kryo's Object > -> Object deep copy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11304) Migrate to Log4j2 from Log4j 1.x
[ https://issues.apache.org/jira/browse/HIVE-11304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007636#comment-15007636 ] Xuefu Zhang commented on HIVE-11304: I retried and confirm what you observed. I don't know why I didn't get it first. Thanks. > Migrate to Log4j2 from Log4j 1.x > > > Key: HIVE-11304 > URL: https://issues.apache.org/jira/browse/HIVE-11304 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Labels: TODOC2.0, incompatibleChange > Fix For: 2.0.0 > > Attachments: HIVE-11304.10.patch, HIVE-11304.11.patch, > HIVE-11304.2.patch, HIVE-11304.3.patch, HIVE-11304.4.patch, > HIVE-11304.5.patch, HIVE-11304.6.patch, HIVE-11304.7.patch, > HIVE-11304.8.patch, HIVE-11304.9.patch, HIVE-11304.patch > > > Log4J2 has some great benefits and can benefit hive significantly. Some > notable features include > 1) Performance (parametrized logging, performance when logging is disabled > etc.) More details can be found here > https://logging.apache.org/log4j/2.x/performance.html > 2) RoutingAppender - Route logs to different log files based on MDC context > (useful for HS2, LLAP etc.) > 3) Asynchronous logging > This is an umbrella jira to track changes related to Log4j2 migration. > Log4J1 EOL - > https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-11304) Migrate to Log4j2 from Log4j 1.x
[ https://issues.apache.org/jira/browse/HIVE-11304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007636#comment-15007636 ] Xuefu Zhang edited comment on HIVE-11304 at 11/16/15 11:40 PM: --- I retried and confirmed what you observed. I don't know why I didn't get it the first time. Thanks. was (Author: xuefuz): I retried and confirm what you observed. I don't know why I didn't get it first. Thanks. > Migrate to Log4j2 from Log4j 1.x > > > Key: HIVE-11304 > URL: https://issues.apache.org/jira/browse/HIVE-11304 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Labels: TODOC2.0, incompatibleChange > Fix For: 2.0.0 > > Attachments: HIVE-11304.10.patch, HIVE-11304.11.patch, > HIVE-11304.2.patch, HIVE-11304.3.patch, HIVE-11304.4.patch, > HIVE-11304.5.patch, HIVE-11304.6.patch, HIVE-11304.7.patch, > HIVE-11304.8.patch, HIVE-11304.9.patch, HIVE-11304.patch > > > Log4J2 has some great benefits and can benefit hive significantly. Some > notable features include > 1) Performance (parametrized logging, performance when logging is disabled > etc.) More details can be found here > https://logging.apache.org/log4j/2.x/performance.html > 2) RoutingAppender - Route logs to different log files based on MDC context > (useful for HS2, LLAP etc.) > 3) Asynchronous logging > This is an umbrella jira to track changes related to Log4j2 migration. > Log4J1 EOL - > https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12427) Improve HiveServer2 *DBC ResultSet performance
[ https://issues.apache.org/jira/browse/HIVE-12427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-12427: Assignee: (was: Vaibhav Gumashta) > Improve HiveServer2 *DBC ResultSet performance > -- > > Key: HIVE-12427 > URL: https://issues.apache.org/jira/browse/HIVE-12427 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, ODBC >Reporter: Vaibhav Gumashta > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12422) LLAP: add security to Web UI endpoint
[ https://issues.apache.org/jira/browse/HIVE-12422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007684#comment-15007684 ] Sergey Shelukhin commented on HIVE-12422: - The patch is on top of HIVE-12341. The default option is to use the daemon keytab. In the setup with HTTP user, the keytab and other settings can be set in daemon config (filters, auth type, principal and keytab are the necessary minimum). > LLAP: add security to Web UI endpoint > - > > Key: HIVE-12422 > URL: https://issues.apache.org/jira/browse/HIVE-12422 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12422.patch > > > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8396) Hive CliDriver command splitting can be broken when comments are present
[
https://issues.apache.org/jira/browse/HIVE-8396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006564#comment-15006564
]
Elliot West commented on HIVE-8396:
---
I note that in {{org.apache.hadoop.hive.cli.CliDriver}} the following code
strips full line comments from the input script:
{code}
public int processReader(BufferedReader r) throws IOException {
String line;
StringBuilder qsb = new StringBuilder();
while ((line = r.readLine()) != null) {
// Skipping through comments
if (! line.startsWith("--")) {
qsb.append(line + "\n");
}
}
return (processLine(qsb.toString()));
}
{code}
Can we not also do this in the shell input processor? Also, what does
{{beeline}} do?
> Hive CliDriver command splitting can be broken when comments are present
>
>
> Key: HIVE-8396
> URL: https://issues.apache.org/jira/browse/HIVE-8396
> Project: Hive
> Issue Type: Bug
> Components: Parser, Query Processor
>Affects Versions: 0.14.0
>Reporter: Sergey Shelukhin
>
> {noformat}
> -- SORT_QUERY_RESULTS
> set hive.cbo.enable=true;
> ... commands ...
> {noformat}
> causes
> {noformat}
> 2014-10-07 18:55:57,193 ERROR ql.Driver (SessionState.java:printError(825)) -
> FAILED: ParseException line 2:4 missing KW_ROLE at 'hive' near 'hive'
> {noformat}
> If the comment is moved after the command it works.
> I noticed this earlier when I comment out parts of some random q file for
> debugging purposes, and it starts failing. This is annoying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11110) Reorder applyPreJoinOrderingTransforms, add NotNULL/FilterMerge rules, improve Filter selectivity estimation
[
https://issues.apache.org/jira/browse/HIVE-0?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006570#comment-15006570
]
Hive QA commented on HIVE-0:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12772449/HIVE-0.24.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 9765 tests executed
*Failed tests:*
{noformat}
TestCliDriver-script_pipe.q-orc_split_elimination.q-groupby_join_pushdown.q-and-12-more
- did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6047/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6047/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6047/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12772449 - PreCommit-HIVE-TRUNK-Build
> Reorder applyPreJoinOrderingTransforms, add NotNULL/FilterMerge rules,
> improve Filter selectivity estimation
>
>
> Key: HIVE-0
> URL: https://issues.apache.org/jira/browse/HIVE-0
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-0-10.patch, HIVE-0-11.patch,
> HIVE-0-12.patch, HIVE-0-branch-1.2.patch, HIVE-0.1.patch,
> HIVE-0.13.patch, HIVE-0.14.patch, HIVE-0.15.patch,
> HIVE-0.16.patch, HIVE-0.17.patch, HIVE-0.18.patch,
> HIVE-0.19.patch, HIVE-0.2.patch, HIVE-0.20.patch,
> HIVE-0.21.patch, HIVE-0.22.patch, HIVE-0.23.patch,
> HIVE-0.24.patch, HIVE-0.4.patch, HIVE-0.5.patch,
> HIVE-0.6.patch, HIVE-0.7.patch, HIVE-0.8.patch,
> HIVE-0.9.patch, HIVE-0.91.patch, HIVE-0.92.patch, HIVE-0.patch
>
>
> Query
> {code}
> select count(*)
> from store_sales
> ,store_returns
> ,date_dim d1
> ,date_dim d2
> where d1.d_quarter_name = '2000Q1'
>and d1.d_date_sk = ss_sold_date_sk
>and ss_customer_sk = sr_customer_sk
>and ss_item_sk = sr_item_sk
>and ss_ticket_number = sr_ticket_number
>and sr_returned_date_sk = d2.d_date_sk
>and d2.d_quarter_name in ('2000Q1','2000Q2','2000Q3’);
> {code}
> The store_sales table is partitioned on ss_sold_date_sk, which is also used
> in a join clause. The join clause should add a filter “filterExpr:
> ss_sold_date_sk is not null”, which should get pushed the MetaStore when
> fetching the stats. Currently this is not done in CBO planning, which results
> in the stats from __HIVE_DEFAULT_PARTITION__ to be fetched and considered in
> the optimization phase. In particular, this increases the NDV for the join
> columns and may result in wrong planning.
> Including HiveJoinAddNotNullRule in the optimization phase solves this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8396) Hive CliDriver command splitting can be broken when comments are present
[
https://issues.apache.org/jira/browse/HIVE-8396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006578#comment-15006578
]
Elliot West commented on HIVE-8396:
---
I expect we could introduce some full line comment filtering somewhere around
{{org.apache.hadoop.hive.cli.CliDriver.executeDriver(...):757}}:
{code}
...
while ((line = reader.readLine(curPrompt + "> ")) != null) {
if (!prefix.equals("")) {
prefix += '\n';
}
if (line.trim().endsWith(";") && !line.trim().endsWith("\\;")) {
line = prefix + line;
ret = cli.processLine(line, true);\
...
{code}
> Hive CliDriver command splitting can be broken when comments are present
>
>
> Key: HIVE-8396
> URL: https://issues.apache.org/jira/browse/HIVE-8396
> Project: Hive
> Issue Type: Bug
> Components: Parser, Query Processor
>Affects Versions: 0.14.0
>Reporter: Sergey Shelukhin
>
> {noformat}
> -- SORT_QUERY_RESULTS
> set hive.cbo.enable=true;
> ... commands ...
> {noformat}
> causes
> {noformat}
> 2014-10-07 18:55:57,193 ERROR ql.Driver (SessionState.java:printError(825)) -
> FAILED: ParseException line 2:4 missing KW_ROLE at 'hive' near 'hive'
> {noformat}
> If the comment is moved after the command it works.
> I noticed this earlier when I comment out parts of some random q file for
> debugging purposes, and it starts failing. This is annoying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12055) Create row-by-row shims for the write path
[ https://issues.apache.org/jira/browse/HIVE-12055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-12055: - Attachment: HIVE-12055.patch This depends on: * HIVE-11890 * HIVE-12054 > Create row-by-row shims for the write path > --- > > Key: HIVE-12055 > URL: https://issues.apache.org/jira/browse/HIVE-12055 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.0.0 > > Attachments: HIVE-12055.patch > > > As part of removing the row-by-row writer, we'll need to shim out the higher > level API (OrcSerde and OrcOutputFormat) so that we maintain backwards > compatibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12054) Create vectorized write method
[ https://issues.apache.org/jira/browse/HIVE-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007984#comment-15007984 ] Prasanth Jayachandran commented on HIVE-12054: -- +1 > Create vectorized write method > -- > > Key: HIVE-12054 > URL: https://issues.apache.org/jira/browse/HIVE-12054 > Project: Hive > Issue Type: Sub-task > Components: File Formats >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.0.0 > > Attachments: HIVE-12054.patch, HIVE-12054.patch, HIVE-12054.patch > > > We need to add writer methods that can write VectorizedRowBatch to an ORC > file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12424) Make use of Kryo's Object-to-Object deep copy
[ https://issues.apache.org/jira/browse/HIVE-12424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007988#comment-15007988 ] Prasanth Jayachandran commented on HIVE-12424: -- Update: the latest maven jars does have the changes for ignoring transient fields to be not copied. > Make use of Kryo's Object-to-Object deep copy > - > > Key: HIVE-12424 > URL: https://issues.apache.org/jira/browse/HIVE-12424 > Project: Hive > Issue Type: Sub-task > Components: Serializers/Deserializers >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > > Currently, plan serialization and operator tree serialization uses Object -> > bytes -> Object approach for deep copy. It also uses ByteArrayOutputStream as > intermediate buffer whose write method is synchronized. Similarly read from > ByteArrayInputStream is also synchronized. Also Utilities.clonePlan() creates > a new HiveConf object that scans through conf directories and adds site.xml > which is an expensive operation. All these can be avoided using Kryo's Object > -> Object deep copy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12184) DESCRIBE of fully qualified table fails when db and table name match and non-default database is in use
[
https://issues.apache.org/jira/browse/HIVE-12184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008021#comment-15008021
]
Hive QA commented on HIVE-12184:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12772440/HIVE-12184.7.patch
{color:green}SUCCESS:{color} +1 due to 23 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 9784 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ambiguitycheck
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lock2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_only_null
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_lockneg4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_lockneg5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_only_null
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6052/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6052/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6052/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12772440 - PreCommit-HIVE-TRUNK-Build
> DESCRIBE of fully qualified table fails when db and table name match and
> non-default database is in use
> ---
>
> Key: HIVE-12184
> URL: https://issues.apache.org/jira/browse/HIVE-12184
> Project: Hive
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.1
>Reporter: Lenni Kuff
>Assignee: Naveen Gangam
> Attachments: HIVE-12184.2.patch, HIVE-12184.3.patch,
> HIVE-12184.4.patch, HIVE-12184.5.patch, HIVE-12184.6.patch,
> HIVE-12184.7.patch, HIVE-12184.patch
>
>
> DESCRIBE of fully qualified table fails when db and table name match and
> non-default database is in use.
> Repro:
> {code}
> : jdbc:hive2://localhost:1/default> create database foo;
> No rows affected (0.116 seconds)
> 0: jdbc:hive2://localhost:1/default> create table foo.foo(i int);
> 0: jdbc:hive2://localhost:1/default> describe foo.foo;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | i | int| |
> +---++--+--+
> 1 row selected (0.049 seconds)
> 0: jdbc:hive2://localhost:1/default> use foo;
> 0: jdbc:hive2://localhost:1/default> describe foo.foo;
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Error in getting fields from
> serde.Invalid Field foo (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10438) Architecture for ResultSet Compression via external plugin
[ https://issues.apache.org/jira/browse/HIVE-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-10438: Issue Type: Sub-task (was: New Feature) Parent: HIVE-12427 > Architecture for ResultSet Compression via external plugin > --- > > Key: HIVE-10438 > URL: https://issues.apache.org/jira/browse/HIVE-10438 > Project: Hive > Issue Type: Sub-task > Components: Hive, Thrift API >Affects Versions: 1.2.0 >Reporter: Rohit Dholakia >Assignee: Rohit Dholakia > Labels: patch > Attachments: HIVE-10438-1.patch, HIVE-10438.patch, > Proposal-rscompressor.pdf, README.txt, > Results_Snappy_protobuf_TBinary_TCompact.pdf, hs2ResultSetCompressor.zip, > hs2driver-master.zip > > > This JIRA proposes an architecture for enabling ResultSet compression which > uses an external plugin. > The patch has three aspects to it: > 0. An architecture for enabling ResultSet compression with external plugins > 1. An example plugin to demonstrate end-to-end functionality > 2. A container to allow everyone to write and test ResultSet compressors with > a query submitter (https://github.com/xiaom/hs2driver) > Also attaching a design document explaining the changes, experimental results > document, and a pdf explaining how to setup the docker container to observe > end-to-end functionality of ResultSet compression. > https://reviews.apache.org/r/35792/ Review board link. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12427) Improve HiveServer2 *DBC ResultSet performance
[ https://issues.apache.org/jira/browse/HIVE-12427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-12427: Component/s: ODBC > Improve HiveServer2 *DBC ResultSet performance > -- > > Key: HIVE-12427 > URL: https://issues.apache.org/jira/browse/HIVE-12427 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, ODBC >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10438) Enable ResultSet compression for HS2/*DBC driver
[ https://issues.apache.org/jira/browse/HIVE-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-10438: Summary: Enable ResultSet compression for HS2/*DBC driver (was: Architecture for ResultSet Compression via external plugin) > Enable ResultSet compression for HS2/*DBC driver > > > Key: HIVE-10438 > URL: https://issues.apache.org/jira/browse/HIVE-10438 > Project: Hive > Issue Type: Sub-task > Components: Hive, Thrift API >Affects Versions: 1.2.0 >Reporter: Rohit Dholakia >Assignee: Rohit Dholakia > Labels: patch > Attachments: HIVE-10438-1.patch, HIVE-10438.patch, > Proposal-rscompressor.pdf, README.txt, > Results_Snappy_protobuf_TBinary_TCompact.pdf, hs2ResultSetCompressor.zip, > hs2driver-master.zip > > > This JIRA proposes an architecture for enabling ResultSet compression which > uses an external plugin. > The patch has three aspects to it: > 0. An architecture for enabling ResultSet compression with external plugins > 1. An example plugin to demonstrate end-to-end functionality > 2. A container to allow everyone to write and test ResultSet compressors with > a query submitter (https://github.com/xiaom/hs2driver) > Also attaching a design document explaining the changes, experimental results > document, and a pdf explaining how to setup the docker container to observe > end-to-end functionality of ResultSet compression. > https://reviews.apache.org/r/35792/ Review board link. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12428) Provide an option for HiveServer2 to stream serialized thrift results when they are available
[ https://issues.apache.org/jira/browse/HIVE-12428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-12428: Issue Type: Sub-task (was: Improvement) Parent: HIVE-12427 > Provide an option for HiveServer2 to stream serialized thrift results when > they are available > - > > Key: HIVE-12428 > URL: https://issues.apache.org/jira/browse/HIVE-12428 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12428) Provide an option for HiveServer2 to stream serialized thrift results when they are available
[ https://issues.apache.org/jira/browse/HIVE-12428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-12428: Description: Once HIVE-12049 is complete, we will have an option on HS2 side to avoid paying the deserialization penalty and simple stream batches of rows on the wire which the *DBC driver can appropriately interpret. This can be done by changing the FetchOperator such that when it has the thrift formatted row sets available, it uses the fastpath. For queries where tasks don't write thrift rowsets, the current implementation will suffice. > Provide an option for HiveServer2 to stream serialized thrift results when > they are available > - > > Key: HIVE-12428 > URL: https://issues.apache.org/jira/browse/HIVE-12428 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta > > Once HIVE-12049 is complete, we will have an option on HS2 side to avoid > paying the deserialization penalty and simple stream batches of rows on the > wire which the *DBC driver can appropriately interpret. This can be done by > changing the FetchOperator such that when it has the thrift formatted row > sets available, it uses the fastpath. For queries where tasks don't write > thrift rowsets, the current implementation will suffice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12399) Native Vector MapJoin can encounter "Null key not expected in MapJoin" and "Unexpected NULL in map join small table" exceptions
[ https://issues.apache.org/jira/browse/HIVE-12399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12399: Attachment: HIVE-12399.02.patch > Native Vector MapJoin can encounter "Null key not expected in MapJoin" and > "Unexpected NULL in map join small table" exceptions > > > Key: HIVE-12399 > URL: https://issues.apache.org/jira/browse/HIVE-12399 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12399.01.patch, HIVE-12399.02.patch > > > Instead of throw exception, just filter out NULLs in the Native Vector > MapJoin operators. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11055) HPL/SQL - Implementing Procedural SQL in Hive (PL/HQL Contribution)
[ https://issues.apache.org/jira/browse/HIVE-11055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007741#comment-15007741 ] Fei Wang commented on HIVE-11055: - is there any design doc for this feature? > HPL/SQL - Implementing Procedural SQL in Hive (PL/HQL Contribution) > --- > > Key: HIVE-11055 > URL: https://issues.apache.org/jira/browse/HIVE-11055 > Project: Hive > Issue Type: Improvement > Components: hpl/sql >Reporter: Dmitry Tolpeko >Assignee: Dmitry Tolpeko > Fix For: 2.0.0 > > Attachments: HIVE-11055.1.patch, HIVE-11055.2.patch, > HIVE-11055.3.patch, HIVE-11055.4.patch, hplsql-site.xml > > > There is PL/HQL tool (www.plhql.org) that implements procedural SQL for Hive > (actually any SQL-on-Hadoop implementation and any JDBC source). > Alan Gates offered to contribute it to Hive under HPL/SQL name > (org.apache.hive.hplsql package). This JIRA is to create a patch to > contribute the PL/HQL code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12417) Support for exclamation mark missing in regexp
[
https://issues.apache.org/jira/browse/HIVE-12417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007781#comment-15007781
]
Hive QA commented on HIVE-12417:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12772502/HIVE-12417.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 9783 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_not
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_shutdown
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_ptf
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6051/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6051/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6051/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12772502 - PreCommit-HIVE-TRUNK-Build
> Support for exclamation mark missing in regexp
> --
>
> Key: HIVE-12417
> URL: https://issues.apache.org/jira/browse/HIVE-12417
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Olaf Flebbe
>Assignee: Olaf Flebbe
> Attachments: HIVE-12417.1.patch
>
>
> with HIVE-6013 gets support for regular expressions. However, die ! character
> is valid, too. It is needed for expressions like
> {code}
> set hive.support.quoted.identifiers = none;
> select `^(?!donotuseme).*$` from table;
> {code}
> which is the idiom to select all but column {{donotuseme}} .
> See http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for
> a reference of supported chars in Java regexp.
> The patch simply fixes the lexer to support '!' as REGEX char. And does
> simply work.
> Please review.
> If you like to have an iTest for it, I beg you to help me. I tried several
> days on a different issue to figure out how it is supposed to work and failed
> miserably.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12430) Remove remaining reference to the hadoop-2 profile
[ https://issues.apache.org/jira/browse/HIVE-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007818#comment-15007818 ] Sergey Shelukhin commented on HIVE-12430: - +1 > Remove remaining reference to the hadoop-2 profile > -- > > Key: HIVE-12430 > URL: https://issues.apache.org/jira/browse/HIVE-12430 > Project: Hive > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-12430.1.patch > > > A reference to the hadoop-2 profile still exists in packaging/pom.xml. Leads > to llap-server not being built correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12417) Support for exclamation mark missing in regexp
[
https://issues.apache.org/jira/browse/HIVE-12417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007820#comment-15007820
]
Sergey Shelukhin commented on HIVE-12417:
-
Test failures appear related, some parsing stuff
> Support for exclamation mark missing in regexp
> --
>
> Key: HIVE-12417
> URL: https://issues.apache.org/jira/browse/HIVE-12417
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Olaf Flebbe
>Assignee: Olaf Flebbe
> Attachments: HIVE-12417.1.patch
>
>
> with HIVE-6013 gets support for regular expressions. However, die ! character
> is valid, too. It is needed for expressions like
> {code}
> set hive.support.quoted.identifiers = none;
> select `^(?!donotuseme).*$` from table;
> {code}
> which is the idiom to select all but column {{donotuseme}} .
> See http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for
> a reference of supported chars in Java regexp.
> The patch simply fixes the lexer to support '!' as REGEX char. And does
> simply work.
> Please review.
> If you like to have an iTest for it, I beg you to help me. I tried several
> days on a different issue to figure out how it is supposed to work and failed
> miserably.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
