[jira] [Updated] (HIVE-14849) Support google-compute-engine provider on Hive ptest framework
[ https://issues.apache.org/jira/browse/HIVE-14849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14849: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) thanks [~prasanth_j] for the quickly review. > Support google-compute-engine provider on Hive ptest framework > -- > > Key: HIVE-14849 > URL: https://issues.apache.org/jira/browse/HIVE-14849 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Fix For: 2.2.0 > > Attachments: HIVE-14849.1.patch > > > NO PRECOMMIT TESTS > Currently, Hive ptest only supports AWS EC2 to create a cluster. We should > add support for GCE as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[
https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531584#comment-15531584
]
Rui Li commented on HIVE-14735:
---
I think that means you have to install md5sum.
Maybe we should document md5sum is required for the build. [~spena], any ideas?
> Build Infra: Spark artifacts download takes a long time
> ---
>
> Key: HIVE-14735
> URL: https://issues.apache.org/jira/browse/HIVE-14735
> Project: Hive
> Issue Type: Bug
> Components: Build Infrastructure
>Reporter: Vaibhav Gumashta
>
> In particular this command:
> {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz
> http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13098) Add a strict check for when the decimal gets converted to null due to insufficient width
[ https://issues.apache.org/jira/browse/HIVE-13098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13098: Attachment: HIVE-13098.WIP2.patch This patch handles many more paths. I think it's close to being ready to review/commit.. The main question is how to handle OIs, cause OI static giant global methods are just too general. I almost wonder if I should give up and add a threadlocal just for that case in the static OI class that would read config once per thread... however, that will make the config global for that case. We can document that config is HS2-wide and cannot be changed per query and do a followup. As for write paths, these would need to be handled on case by case basis cause their initialization is obscure. I started on the ORC write path, but put the config in the wrong place for master (it's the right place for branch-1). I will finish that. [~jdere] [~ashutoshc] do you want to take a look? 95% of the changes are just propagating the stupid config. Main changes are in HiveDecimal and HiveDecimalOverflow classes. > Add a strict check for when the decimal gets converted to null due to > insufficient width > > > Key: HIVE-13098 > URL: https://issues.apache.org/jira/browse/HIVE-13098 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13098.WIP.patch, HIVE-13098.WIP2.patch > > > When e.g. 99 is selected as decimal(5,0), the result is null. This can be > problematic, esp. if the data is written to a table and lost without the user > realizing it. There should be an option to error out in such cases instead; > it should probably be on by default and the error message should instruct the > user on how to disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14856) create table with select from table limit is failing with NFE if limit exceed than allowed 32bit integer length
[ https://issues.apache.org/jira/browse/HIVE-14856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-14856: -- Attachment: HIVE-14856.patch > create table with select from table limit is failing with NFE if limit exceed > than allowed 32bit integer length > --- > > Key: HIVE-14856 > URL: https://issues.apache.org/jira/browse/HIVE-14856 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.1 > Environment: centos 6.6 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh > Fix For: 1.2.1 > > Attachments: HIVE-14856.patch > > > query with limit is failing with NumberFormatException if the limit exceeds > 32bit integer length. > create table sample1 as select * from sample limit 2248321440; > FAILED: NumberFormatException For input string: "2248321440" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14856) create table with select from table limit is failing with NFE if limit exceed than allowed 32bit integer length
[ https://issues.apache.org/jira/browse/HIVE-14856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-14856: -- Fix Version/s: 1.2.1 Status: Patch Available (was: Open) > create table with select from table limit is failing with NFE if limit exceed > than allowed 32bit integer length > --- > > Key: HIVE-14856 > URL: https://issues.apache.org/jira/browse/HIVE-14856 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.1 > Environment: centos 6.6 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh > Fix For: 1.2.1 > > > query with limit is failing with NumberFormatException if the limit exceeds > 32bit integer length. > create table sample1 as select * from sample limit 2248321440; > FAILED: NumberFormatException For input string: "2248321440" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14852) Change qtest logging to not redirect all logs to console
[
https://issues.apache.org/jira/browse/HIVE-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531487#comment-15531487
]
Hive QA commented on HIVE-14852:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12830763/HIVE-14852.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10645 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1335/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1335/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1335/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12830763 - PreCommit-HIVE-Build
> Change qtest logging to not redirect all logs to console
>
>
> Key: HIVE-14852
> URL: https://issues.apache.org/jira/browse/HIVE-14852
> Project: Hive
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14852.01.patch, HIVE-14852.02.patch
>
>
> A change was made recently to redirect all logs to console, to make IDE
> debugging of regular tests easier. That unfortunately makes qtest debugging
> tougher - since there's a lot of noise along with the diffs in the output
> file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9762) LLAP: hook up decider + dag utils
[ https://issues.apache.org/jira/browse/HIVE-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531448#comment-15531448 ] Lefty Leverenz commented on HIVE-9762: -- Commit error: The commit gave the wrong JIRA number (HIVE-9782 instead of HIVE-9762). http://svn.apache.org/r1661833 Since it was an SVN commit, it doesn't belong in errata.txt with the github errors. > LLAP: hook up decider + dag utils > - > > Key: HIVE-9762 > URL: https://issues.apache.org/jira/browse/HIVE-9762 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-9762.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14855) test patch
[ https://issues.apache.org/jira/browse/HIVE-14855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14855: -- Attachment: HIVE-14855.patch > test patch > -- > > Key: HIVE-14855 > URL: https://issues.apache.org/jira/browse/HIVE-14855 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14855.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14855) test patch
[ https://issues.apache.org/jira/browse/HIVE-14855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14855: -- Status: Patch Available (was: Open) > test patch > -- > > Key: HIVE-14855 > URL: https://issues.apache.org/jira/browse/HIVE-14855 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14849) Support google-compute-engine provider on Hive ptest framework
[ https://issues.apache.org/jira/browse/HIVE-14849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531406#comment-15531406 ] Prasanth Jayachandran commented on HIVE-14849: -- looks good to me, +1 > Support google-compute-engine provider on Hive ptest framework > -- > > Key: HIVE-14849 > URL: https://issues.apache.org/jira/browse/HIVE-14849 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-14849.1.patch > > > NO PRECOMMIT TESTS > Currently, Hive ptest only supports AWS EC2 to create a cluster. We should > add support for GCE as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14830) Move a majority of the MiniLlapCliDriver tests to use an inline AM
[ https://issues.apache.org/jira/browse/HIVE-14830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14830: -- Attachment: (was: HIVE-14830.02.patch) > Move a majority of the MiniLlapCliDriver tests to use an inline AM > -- > > Key: HIVE-14830 > URL: https://issues.apache.org/jira/browse/HIVE-14830 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14830.01.patch, HIVE-14830.01.patch, > HIVE-14830.02.patch, HIVE-14830.02_OnHive14854.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14830) Move a majority of the MiniLlapCliDriver tests to use an inline AM
[ https://issues.apache.org/jira/browse/HIVE-14830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14830: -- Attachment: HIVE-14830.02.patch > Move a majority of the MiniLlapCliDriver tests to use an inline AM > -- > > Key: HIVE-14830 > URL: https://issues.apache.org/jira/browse/HIVE-14830 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14830.01.patch, HIVE-14830.01.patch, > HIVE-14830.02.patch, HIVE-14830.02_OnHive14854.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14830) Move a majority of the MiniLlapCliDriver tests to use an inline AM
[ https://issues.apache.org/jira/browse/HIVE-14830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14830: -- Attachment: HIVE-14830.02.patch > Move a majority of the MiniLlapCliDriver tests to use an inline AM > -- > > Key: HIVE-14830 > URL: https://issues.apache.org/jira/browse/HIVE-14830 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14830.01.patch, HIVE-14830.01.patch, > HIVE-14830.02.patch, HIVE-14830.02_OnHive14854.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-14029: Labels: Incompatible TODOC2.2 (was: TODOC2.2) > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Labels: Incompatible, TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.4.patch, HIVE-14029.5.patch, > HIVE-14029.6.patch, HIVE-14029.7.patch, HIVE-14029.8.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. > To update Spark version to 2.0.0, the following changes are required: > * Spark API updates: > ** SparkShuffler#call return Iterator instead of Iterable > ** SparkListener -> JavaSparkListener > ** InputMetrics constructor doesn’t accept readMethod > ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics > return long type instead of integer > * Dependency upgrade: > ** Jackson: 2.4.2 -> 2.6.5 > ** Netty version: 4.0.23.Final -> 4.0.29.Final > ** Scala binary version: 2.10 -> 2.11 > ** Scala version: 2.10.4 -> 2.11.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14830) Move a majority of the MiniLlapCliDriver tests to use an inline AM
[ https://issues.apache.org/jira/browse/HIVE-14830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14830: -- Attachment: (was: HIVE-14854.02.patch) > Move a majority of the MiniLlapCliDriver tests to use an inline AM > -- > > Key: HIVE-14830 > URL: https://issues.apache.org/jira/browse/HIVE-14830 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14830.01.patch, HIVE-14830.01.patch, > HIVE-14830.02.patch, HIVE-14830.02_OnHive14854.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14830) Move a majority of the MiniLlapCliDriver tests to use an inline AM
[ https://issues.apache.org/jira/browse/HIVE-14830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14830: -- Attachment: HIVE-14854.02.patch Patch for jenkins (this + 14854 combined) > Move a majority of the MiniLlapCliDriver tests to use an inline AM > -- > > Key: HIVE-14830 > URL: https://issues.apache.org/jira/browse/HIVE-14830 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14830.01.patch, HIVE-14830.01.patch, > HIVE-14830.02.patch, HIVE-14830.02_OnHive14854.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531389#comment-15531389 ] Ferdinand Xu commented on HIVE-14029: - bq. I'm guessing this should go into the Hive 2.2.0 release since its an incompatible change Agree. bq. will Hive-on-Spark2 be able to run against a Spark1 cluster, or vice versa? AFAIK, it will not able to run against Spark1 cluster for the dependency conflicts. If we want to support different Spark cluster, we may need a shim loader for Spark in Spark client. Do we have a strong requirement for that? > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.4.patch, HIVE-14029.5.patch, > HIVE-14029.6.patch, HIVE-14029.7.patch, HIVE-14029.8.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. > To update Spark version to 2.0.0, the following changes are required: > * Spark API updates: > ** SparkShuffler#call return Iterator instead of Iterable > ** SparkListener -> JavaSparkListener > ** InputMetrics constructor doesn’t accept readMethod > ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics > return long type instead of integer > * Dependency upgrade: > ** Jackson: 2.4.2 -> 2.6.5 > ** Netty version: 4.0.23.Final -> 4.0.29.Final > ** Scala binary version: 2.10 -> 2.11 > ** Scala version: 2.10.4 -> 2.11.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14830) Move a majority of the MiniLlapCliDriver tests to use an inline AM
[ https://issues.apache.org/jira/browse/HIVE-14830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14830: -- Attachment: HIVE-14830.02_OnHive14854.txt Actual patch, on top of HIVE-14854. > Move a majority of the MiniLlapCliDriver tests to use an inline AM > -- > > Key: HIVE-14830 > URL: https://issues.apache.org/jira/browse/HIVE-14830 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14830.01.patch, HIVE-14830.01.patch, > HIVE-14830.02_OnHive14854.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14852) Change qtest logging to not redirect all logs to console
[ https://issues.apache.org/jira/browse/HIVE-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531380#comment-15531380 ] Prasanth Jayachandran commented on HIVE-14852: -- lgtm, +1 > Change qtest logging to not redirect all logs to console > > > Key: HIVE-14852 > URL: https://issues.apache.org/jira/browse/HIVE-14852 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14852.01.patch, HIVE-14852.02.patch > > > A change was made recently to redirect all logs to console, to make IDE > debugging of regular tests easier. That unfortunately makes qtest debugging > tougher - since there's a lot of noise along with the diffs in the output > file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14852) Change qtest logging to not redirect all logs to console
[ https://issues.apache.org/jira/browse/HIVE-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14852: -- Attachment: HIVE-14852.02.patch Updated patch to remove the extra file. > Change qtest logging to not redirect all logs to console > > > Key: HIVE-14852 > URL: https://issues.apache.org/jira/browse/HIVE-14852 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14852.01.patch, HIVE-14852.02.patch > > > A change was made recently to redirect all logs to console, to make IDE > debugging of regular tests easier. That unfortunately makes qtest debugging > tougher - since there's a lot of noise along with the diffs in the output > file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531350#comment-15531350 ] Gopal V commented on HIVE-14731: Thanks [~aplusplus], I'll take a look tomorrow. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14854) Add a core cluster type to QTestUtil
[ https://issues.apache.org/jira/browse/HIVE-14854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14854: -- Attachment: HIVE-14854.02.patch Updated patch to compile. > Add a core cluster type to QTestUtil > > > Key: HIVE-14854 > URL: https://issues.apache.org/jira/browse/HIVE-14854 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14854.01.patch, HIVE-14854.02.patch > > > Follow up to HIVE-14824. There's tez, tez_local, llap, llap_local - all of > which are of a single type, similaryl spark, sparkOnYarn, and none,mr. > Introducing a core cluster type to make a bunch of conditional checks simpler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14784) Operation logs are disabled automatically if the parent directory does not exist.
[
https://issues.apache.org/jira/browse/HIVE-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531287#comment-15531287
]
Hive QA commented on HIVE-14784:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12830753/HIVE-14784.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9882 tests executed
*Failed tests:*
{noformat}
TestNegativeCliDriver-nopart_insert.q-input41.q-having1.q-and-765-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.hcatalog.mapreduce.TestHCatMultiOutputFormat.org.apache.hive.hcatalog.mapreduce.TestHCatMultiOutputFormat
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1334/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1334/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1334/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12830753 - PreCommit-HIVE-Build
> Operation logs are disabled automatically if the parent directory does not
> exist.
> -
>
> Key: HIVE-14784
> URL: https://issues.apache.org/jira/browse/HIVE-14784
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14784.1.patch, HIVE-14784.patch
>
>
> Operation logging is disabled automatically for the query if for some reason
> the parent directory (named after the hive session id) that gets created when
> the session is established gets deleted (for any reason). For ex: if the
> operation logdir is /tmp which automatically can get purged at a configured
> interval by the OS.
> Running a query from that session leads to
> {code}
> 2016-09-15 15:09:16,723 WARN org.apache.hive.service.cli.operation.Operation:
> Unable to create operation log file:
> /tmp/hive/operation_logs/b8809985-6b38-47ec-a49b-6158a67cd9fc/d35414f7-2418-426c-8489-c6f643ca4599
> java.io.IOException: No such file or directory
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hive.service.cli.operation.Operation.createOperationLog(Operation.java:195)
> at
> org.apache.hive.service.cli.operation.Operation.beforeRun(Operation.java:237)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:255)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:398)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:385)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:271)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:490)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> This later leads to errors like (more prominent when using HUE as HUE does
> not close hive sessions and attempts to retrieve the operations logs days
> after they were created).
> {code}
>
[jira] [Commented] (HIVE-14082) Multi-Insert Query Fails with GROUP BY, DISTINCT, and WHERE clauses
[
https://issues.apache.org/jira/browse/HIVE-14082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531236#comment-15531236
]
Sahil Takiar commented on HIVE-14082:
-
Ping, anyone have any idea on this?
> Multi-Insert Query Fails with GROUP BY, DISTINCT, and WHERE clauses
> ---
>
> Key: HIVE-14082
> URL: https://issues.apache.org/jira/browse/HIVE-14082
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> The following MULTI-INSERT Query Fails in Hive. I've listed the query
> required to re-produce this failure, as well as a few similar queries that
> work properly.
> Setup Queries:
> {code}
> DROP SCHEMA IF EXISTS multi_table_insert_bug CASCADE;
> CREATE SCHEMA multi_table_insert_bug;
> USE multi_table_insert_bug;
> DROP TABLE IF EXISTS multi_table_insert_source;
> DROP TABLE IF EXISTS multi_table_insert_test;
> CREATE TABLE multi_table_insert_source (
> date_column DATE,
> column_1 STRING,
> column_2 STRING,
> column_3 STRING,
> column_4 STRING
> );
> CREATE TABLE multi_table_insert_test (
> column_1 STRING,
> column_2 STRING,
> line_count INT,
> distinct_count_by_1_column INT,
> distinct_count_by_2_columns INT
> )
> PARTITIONED BY (partition_column INT);
> INSERT OVERWRITE TABLE multi_table_insert_source VALUES
> ('2016-01-22', 'value_1_1', 'value_1_2', 'value_1_3', 'value_1_4'),
> ('2016-01-22', 'value_2_1', 'value_2_2', 'value_2_3', 'value_2_4'),
> ('2016-01-22', 'value_3_1', 'value_3_2', 'value_3_3', 'value_3_4'),
> ('2016-01-22', 'value_4_1', 'value_4_2', 'value_4_3', 'value_4_4'),
> ('2016-01-22', 'value_5_1', 'value_5_2', 'value_5_3', 'value_5_4');
> {code}
> The following queries run successfully:
> *Query 1:*
> {code}
> FROM multi_table_insert_source
> INSERT OVERWRITE TABLE multi_table_insert_test PARTITION (partition_column
> = 365)
> SELECT
> column_1,
> column_2,
> COUNT(*) AS line_count,
> COUNT(DISTINCT column_3) AS distinct_count_by_1_column,
> COUNT(DISTINCT date_column, column_3) AS distinct_count_by_2_columns
> WHERE date_column >= DATE_SUB(FROM_UNIXTIME(UNIX_TIMESTAMP()), 365)
> GROUP BY
> column_1,
> column_2;
> {code}
> *Query 2:*
> {code}
> FROM multi_table_insert_source
> INSERT OVERWRITE TABLE multi_table_insert_test PARTITION (partition_column
> = 365)
> SELECT
> column_1,
> column_2,
> COUNT(*) AS line_count,
> COUNT(DISTINCT column_3) AS distinct_count_by_1_column,
> COUNT(DISTINCT date_column, column_3) AS distinct_count_by_2_columns
> -- WHERE date_column >= DATE_SUB(FROM_UNIXTIME(UNIX_TIMESTAMP()), 365)
> GROUP BY
> column_1,
> column_2
> INSERT OVERWRITE TABLE multi_table_insert_test PARTITION (partition_column
> = 1096)
> SELECT
> column_1,
> column_2,
> COUNT(*) AS line_count,
> COUNT(DISTINCT column_3) AS distinct_count_by_1_column,
> COUNT(DISTINCT date_column, column_3) AS distinct_count_by_2_columns
> -- WHERE date_column >= DATE_SUB(FROM_UNIXTIME(UNIX_TIMESTAMP()), 1096)
> GROUP BY
> column_1,
> column_2;
> {code}
> The following query fails with a {{ClassCastException}}:
> *Query 3:*
> {code}
> FROM multi_table_insert_source
> INSERT OVERWRITE TABLE multi_table_insert_test PARTITION (partition_column
> = 365)
> SELECT
> column_1,
> column_2,
> COUNT(*) AS line_count,
> COUNT(DISTINCT column_3) AS distinct_count_by_1_column,
> COUNT(DISTINCT date_column, column_3) AS distinct_count_by_2_columns
> WHERE date_column >= DATE_SUB(FROM_UNIXTIME(UNIX_TIMESTAMP()), 365)
> GROUP BY
> column_1,
> column_2
> INSERT OVERWRITE TABLE multi_table_insert_test PARTITION (partition_column
> = 1096)
> SELECT
> column_1,
> column_2,
> COUNT(*) AS line_count,
> COUNT(DISTINCT column_3) AS distinct_count_by_1_column,
> COUNT(DISTINCT date_column, column_3) AS distinct_count_by_2_columns
> WHERE date_column >= DATE_SUB(FROM_UNIXTIME(UNIX_TIMESTAMP()), 1096)
> GROUP BY
> column_1,
> column_2;
> {code}
> Here is the full stack-trace of the exception:
> *Exception 1:*
> {code}
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=0)
> {"key":{"_col0":"value_1_1","_col1":"value_1_2","_col2":{0:{"_col0":"value_1_3"}}},"value":null}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:257)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:506)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:449)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531225#comment-15531225 ] Sahil Takiar commented on HIVE-7224: Hey [~leftylev] could you grant me permission to edit the wiki page? Username is "stakiar" > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Beeline, Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch, HIVE-7224.4.patch, HIVE-7224.5.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7926) long-lived daemons for query fragment execution, I/O and caching
[ https://issues.apache.org/jira/browse/HIVE-7926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531222#comment-15531222 ] Shannon Ladymon commented on HIVE-7926: --- [~asears], thanks for fixing the broken link! What did you want to do about the Web Services section? > long-lived daemons for query fragment execution, I/O and caching > > > Key: HIVE-7926 > URL: https://issues.apache.org/jira/browse/HIVE-7926 > Project: Hive > Issue Type: New Feature >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: LLAPdesigndocument.pdf > > > We are proposing a new execution model for Hive that is a combination of > existing process-based tasks and long-lived daemons running on worker nodes. > These nodes can take care of efficient I/O, caching and query fragment > execution, while heavy lifting like most joins, ordering, etc. can be handled > by tasks. > The proposed model is not a 2-system solution for small and large queries; > neither it is a separate execution engine like MR or Tez. It can be used by > any Hive execution engine, if support is added; in future even external > products (e.g. Pig) can use it. > The document with high-level design we are proposing will be attached shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7926) long-lived daemons for query fragment execution, I/O and caching
[ https://issues.apache.org/jira/browse/HIVE-7926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531210#comment-15531210 ] Shannon Ladymon commented on HIVE-7926: --- [~sseth], thank you so much for the clarifications! > long-lived daemons for query fragment execution, I/O and caching > > > Key: HIVE-7926 > URL: https://issues.apache.org/jira/browse/HIVE-7926 > Project: Hive > Issue Type: New Feature >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: LLAPdesigndocument.pdf > > > We are proposing a new execution model for Hive that is a combination of > existing process-based tasks and long-lived daemons running on worker nodes. > These nodes can take care of efficient I/O, caching and query fragment > execution, while heavy lifting like most joins, ordering, etc. can be handled > by tasks. > The proposed model is not a 2-system solution for small and large queries; > neither it is a separate execution engine like MR or Tez. It can be used by > any Hive execution engine, if support is added; in future even external > products (e.g. Pig) can use it. > The document with high-level design we are proposing will be attached shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14852) Change qtest logging to not redirect all logs to console
[ https://issues.apache.org/jira/browse/HIVE-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531183#comment-15531183 ] Siddharth Seth commented on HIVE-14852: --- bq. Say if I run TestOrcFile, I would like to see what test has failed and what assertion is causing the failure. But I don't want to see all debug level logs getting printed to the console. It used to the case. Will this behavior change? This is available today as well. The org.apcheTestOrc-output.txt normally contains the entire test log. org.apcheTestOrc.txt contains the list of tests run, failed with stack traces etc. bq. I am missing something here. If root logger is DRFA why do we even see something in the console? Is it because of the recent patch that logs to console as well. Yes > Change qtest logging to not redirect all logs to console > > > Key: HIVE-14852 > URL: https://issues.apache.org/jira/browse/HIVE-14852 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14852.01.patch > > > A change was made recently to redirect all logs to console, to make IDE > debugging of regular tests easier. That unfortunately makes qtest debugging > tougher - since there's a lot of noise along with the diffs in the output > file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14854) Add a core cluster type to QTestUtil
[ https://issues.apache.org/jira/browse/HIVE-14854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14854: -- Attachment: HIVE-14854.01.patch Simple patch adds a CoreClusterType similar to the defaultFsType introduced in HIVE-14824. cc [~prasanth_j] > Add a core cluster type to QTestUtil > > > Key: HIVE-14854 > URL: https://issues.apache.org/jira/browse/HIVE-14854 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14854.01.patch > > > Follow up to HIVE-14824. There's tez, tez_local, llap, llap_local - all of > which are of a single type, similaryl spark, sparkOnYarn, and none,mr. > Introducing a core cluster type to make a bunch of conditional checks simpler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14854) Add a core cluster type to QTestUtil
[ https://issues.apache.org/jira/browse/HIVE-14854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14854: -- Status: Patch Available (was: Open) > Add a core cluster type to QTestUtil > > > Key: HIVE-14854 > URL: https://issues.apache.org/jira/browse/HIVE-14854 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14854.01.patch > > > Follow up to HIVE-14824. There's tez, tez_local, llap, llap_local - all of > which are of a single type, similaryl spark, sparkOnYarn, and none,mr. > Introducing a core cluster type to make a bunch of conditional checks simpler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14830) Move a majority of the MiniLlapCliDriver tests to use an inline AM
[ https://issues.apache.org/jira/browse/HIVE-14830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531150#comment-15531150 ] Siddharth Seth commented on HIVE-14830: --- Splitting this into 2 patches. One with changes to QTestUtil, the other with the actual test changes. I suspect the one with test changes may require a few iterations. > Move a majority of the MiniLlapCliDriver tests to use an inline AM > -- > > Key: HIVE-14830 > URL: https://issues.apache.org/jira/browse/HIVE-14830 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14830.01.patch, HIVE-14830.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14761) Merge TestJdbcWithMiniMr with TestJdbcWithMiniHS2
[ https://issues.apache.org/jira/browse/HIVE-14761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-14761: Attachment: HIVE-14761.1.patch > Merge TestJdbcWithMiniMr with TestJdbcWithMiniHS2 > -- > > Key: HIVE-14761 > URL: https://issues.apache.org/jira/browse/HIVE-14761 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-14761.1.patch > > > Currently 2 min 30 sec -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14761) Merge TestJdbcWithMiniMr with TestJdbcWithMiniHS2
[ https://issues.apache.org/jira/browse/HIVE-14761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531137#comment-15531137 ] Vaibhav Gumashta commented on HIVE-14761: - [~sseth] Up for review whenever you get a chance. > Merge TestJdbcWithMiniMr with TestJdbcWithMiniHS2 > -- > > Key: HIVE-14761 > URL: https://issues.apache.org/jira/browse/HIVE-14761 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-14761.1.patch > > > Currently 2 min 30 sec -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14721) Fix TestJdbcWithMiniHS2 runtime
[ https://issues.apache.org/jira/browse/HIVE-14721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531126#comment-15531126 ] Vaibhav Gumashta commented on HIVE-14721: - Will also merge 2 tests from TestJdbcWithMiniMr in the next patch. > Fix TestJdbcWithMiniHS2 runtime > --- > > Key: HIVE-14721 > URL: https://issues.apache.org/jira/browse/HIVE-14721 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-14721.1.patch > > > Currently 450s -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14761) Merge TestJdbcWithMiniMr with TestJdbcWithMiniHS2
[ https://issues.apache.org/jira/browse/HIVE-14761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531124#comment-15531124 ] Vaibhav Gumashta commented on HIVE-14761: - I'll move those 2 tests as part of another patch for HIVE-14721. Just removing the whole class in this one. > Merge TestJdbcWithMiniMr with TestJdbcWithMiniHS2 > -- > > Key: HIVE-14761 > URL: https://issues.apache.org/jira/browse/HIVE-14761 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > > Currently 2 min 30 sec -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14761) Merge TestJdbcWithMiniMr with TestJdbcWithMiniHS2
[ https://issues.apache.org/jira/browse/HIVE-14761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-14761: Summary: Merge TestJdbcWithMiniMr with TestJdbcWithMiniHS2 (was: Improve TestJdbcWithMiniMr run time ) > Merge TestJdbcWithMiniMr with TestJdbcWithMiniHS2 > -- > > Key: HIVE-14761 > URL: https://issues.apache.org/jira/browse/HIVE-14761 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > > Currently 2 min 30 sec -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14761) Improve TestJdbcWithMiniMr run time
[ https://issues.apache.org/jira/browse/HIVE-14761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531119#comment-15531119 ] Vaibhav Gumashta commented on HIVE-14761: - Discussed a bit with [~jdere] on the tests in this class (for perm function and temp tables). I think we can do away with this class by moving the tests to TestJdbcWithMiniHS2. I'll move forward with that. > Improve TestJdbcWithMiniMr run time > > > Key: HIVE-14761 > URL: https://issues.apache.org/jira/browse/HIVE-14761 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > > Currently 2 min 30 sec -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13966) DbNotificationListener: can loose DDL operation notifications
[ https://issues.apache.org/jira/browse/HIVE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531063#comment-15531063 ] Sushanth Sowmyan commented on HIVE-13966: - Thanks for jumping in on this, [~alangates], this has been on my TODO pile for way too long and I've not done it justice. > DbNotificationListener: can loose DDL operation notifications > - > > Key: HIVE-13966 > URL: https://issues.apache.org/jira/browse/HIVE-13966 > Project: Hive > Issue Type: Bug > Components: HCatalog >Reporter: Nachiket Vaidya >Assignee: Rahul Sharma >Priority: Critical > Attachments: HIVE-13966.1.patch, HIVE-13966.2.patch, HIVE-13966.pdf > > > The code for each API in HiveMetaStore.java is like this: > 1. openTransaction() > 2. -- operation-- > 3. commit() or rollback() based on result of the operation. > 4. add entry to notification log (unconditionally) > If the operation is failed (in step 2), we still add entry to notification > log. Found this issue in testing. > It is still ok as this is the case of false positive. > If the operation is successful and adding to notification log failed, the > user will get an MetaException. It will not rollback the operation, as it is > already committed. We need to handle this case so that we will not have false > negatives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531017#comment-15531017 ] Sahil Takiar commented on HIVE-14029: - [~Ferd] can you also add an "Incompatible Change" Flag to this JIRA. I'm guessing this should go into the Hive 2.2.0 release since its an incompatible change, and I agree we should document this all on the wiki. I don't know much about Spark 2, but will Hive-on-Spark2 be able to run against a Spark1 cluster, or vice versa? > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.4.patch, HIVE-14029.5.patch, > HIVE-14029.6.patch, HIVE-14029.7.patch, HIVE-14029.8.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. > To update Spark version to 2.0.0, the following changes are required: > * Spark API updates: > ** SparkShuffler#call return Iterator instead of Iterable > ** SparkListener -> JavaSparkListener > ** InputMetrics constructor doesn’t accept readMethod > ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics > return long type instead of integer > * Dependency upgrade: > ** Jackson: 2.4.2 -> 2.6.5 > ** Netty version: 4.0.23.Final -> 4.0.29.Final > ** Scala binary version: 2.10 -> 2.11 > ** Scala version: 2.10.4 -> 2.11.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14137) Hive on Spark throws FileAlreadyExistsException for jobs with multiple empty tables
[
https://issues.apache.org/jira/browse/HIVE-14137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14137:
---
Fix Version/s: 2.1.1
> Hive on Spark throws FileAlreadyExistsException for jobs with multiple empty
> tables
> ---
>
> Key: HIVE-14137
> URL: https://issues.apache.org/jira/browse/HIVE-14137
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14137.1.patch, HIVE-14137.2.patch,
> HIVE-14137.3.patch, HIVE-14137.4.patch, HIVE-14137.5.patch,
> HIVE-14137.6-branch-2.1.patch, HIVE-14137.6.patch, HIVE-14137.patch
>
>
> The following queries:
> {code}
> -- Setup
> drop table if exists empty1;
> create table empty1 (col1 bigint) stored as parquet tblproperties
> ('parquet.compress'='snappy');
> drop table if exists empty2;
> create table empty2 (col1 bigint, col2 bigint) stored as parquet
> tblproperties ('parquet.compress'='snappy');
> drop table if exists empty3;
> create table empty3 (col1 bigint) stored as parquet tblproperties
> ('parquet.compress'='snappy');
> -- All empty HDFS directories.
> -- Fails with [08S01]: Error while processing statement: FAILED: Execution
> Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask.
> select empty1.col1
> from empty1
> inner join empty2
> on empty2.col1 = empty1.col1
> inner join empty3
> on empty3.col1 = empty2.col2;
> -- Two empty HDFS directories.
> -- Create an empty file in HDFS.
> insert into empty1 select * from empty1 where false;
> -- Same query fails with [08S01]: Error while processing statement: FAILED:
> Execution Error, return code 3 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.
> select empty1.col1
> from empty1
> inner join empty2
> on empty2.col1 = empty1.col1
> inner join empty3
> on empty3.col1 = empty2.col2;
> -- One empty HDFS directory.
> -- Create an empty file in HDFS.
> insert into empty2 select * from empty2 where false;
> -- Same query succeeds.
> select empty1.col1
> from empty1
> inner join empty2
> on empty2.col1 = empty1.col1
> inner join empty3
> on empty3.col1 = empty2.col2;
> {code}
> Will result in the following exception:
> {code}
> org.apache.hadoop.fs.FileAlreadyExistsException:
> /tmp/hive/hive/1f3837aa-9407-4780-92b1-42a66d205139/hive_2016-06-24_15-45-23_206_79177714958655528-2/-mr-10004/0/emptyFile
> for client 172.26.14.151 already exists
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2676)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2561)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:593)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.create(AuthorizationProviderProxyClientProtocol.java:111)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:393)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1902)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1738)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1663)
>
[jira] [Updated] (HIVE-13749) Memory leak in Hive Metastore
[ https://issues.apache.org/jira/browse/HIVE-13749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-13749: --- Fix Version/s: 2.1.1 > Memory leak in Hive Metastore > - > > Key: HIVE-13749 > URL: https://issues.apache.org/jira/browse/HIVE-13749 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-13749.1.patch, HIVE-13749.patch, Top_Consumers7.html > > > Looking a heap dump of 10GB, a large number of Configuration objects(> 66k > instances) are being retained. These objects along with its retained set is > occupying about 95% of the heap space. This leads to HMS crashes every few > days. > I will attach an exported snapshot from the eclipse MAT. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14519) Multi insert query bug
[
https://issues.apache.org/jira/browse/HIVE-14519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14519:
---
Fix Version/s: 2.1.1
> Multi insert query bug
> --
>
> Key: HIVE-14519
> URL: https://issues.apache.org/jira/browse/HIVE-14519
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14519.1.patch
>
>
> When running multi-insert queries, when one of the query is not returning
> results, the other query is not returning the right result.
> For example:
> After following query, there is no value in /tmp/emp/dir3/00_0
> {noformat}
> From (select * from src) a
> insert overwrite directory '/tmp/emp/dir1/'
> select key, value
> insert overwrite directory '/tmp/emp/dir2/'
> select 'header'
> where 1=2
> insert overwrite directory '/tmp/emp/dir3/'
> select key, value
> where key = 100;
> {noformat}
> where clause in the second insert should not affect the third insert.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14743) ArrayIndexOutOfBoundsException - HBASE-backed views' query with JOINs
[
https://issues.apache.org/jira/browse/HIVE-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14743:
---
Fix Version/s: 2.1.1
> ArrayIndexOutOfBoundsException - HBASE-backed views' query with JOINs
> -
>
> Key: HIVE-14743
> URL: https://issues.apache.org/jira/browse/HIVE-14743
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 1.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14743.1.patch, HIVE-14743.2.patch
>
>
> The stack:
> {noformat}
> 2016-09-13T09:38:49,972 ERROR [186b4545-65b5-4bfc-bc8e-3e14e251bb12 main]
> exec.Task: Job Submission failed with exception
> 'java.lang.ArrayIndexOutOfBoundsException(1)'
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.createFilterScan(HiveHBaseTableInputFormat.java:224)
> at
> org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplitsInternal(HiveHBaseTableInputFormat.java:492)
> at
> org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplits(HiveHBaseTableInputFormat.java:449)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:370)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:466)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getCombineSplits(CombineHiveInputFormat.java:356)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:546)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:329)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:320)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
> {noformat}
> Repro:
> {noformat}
> CREATE TABLE HBASE_TABLE_TEST_1(
> cvalue string ,
> pk string,
> ccount int )
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'='cf:val,:key,cf2:count',
> 'hbase.scan.cache'='500',
> 'hbase.scan.cacheblocks'='false',
> 'serialization.format'='1')
> TBLPROPERTIES (
> 'hbase.table.name'='hbase_table_test_1',
> 'serialization.null.format'='' );
> CREATE VIEW VIEW_HBASE_TABLE_TEST_1 AS SELECT
> hbase_table_test_1.cvalue,hbase_table_test_1.pk,hbase_table_test_1.ccount
> FROM hbase_table_test_1 WHERE hbase_table_test_1.ccount IS NOT NULL;
> CREATE TABLE HBASE_TABLE_TEST_2(
> cvalue string ,
> pk string ,
>ccount int )
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'='cf:val,:key,cf2:count',
> 'hbase.scan.cache'='500',
> 'hbase.scan.cacheblocks'='false',
> 'serialization.format'='1')
> TBLPROPERTIES (
> 'hbase.table.name'='hbase_table_test_2',
> 'serialization.null.format'='');
> CREATE VIEW VIEW_HBASE_TABLE_TEST_2 AS SELECT
> hbase_table_test_2.cvalue,hbase_table_test_2.pk,hbase_table_test_2.ccount
> FROM hbase_table_test_2 WHERE hbase_table_test_2.pk >='3-h-0' AND
> hbase_table_test_2.pk <= '3-h-g' AND hbase_table_test_2.ccount IS NOT
> NULL;
> set hive.auto.convert.join=false;
> SELECT p.cvalue cvalue
> FROM `VIEW_HBASE_TABLE_TEST_1` `p`
> LEFT OUTER JOIN `VIEW_HBASE_TABLE_TEST_2` `A1`
> ON `p`.cvalue = `A1`.cvalue
> LEFT OUTER JOIN `VIEW_HBASE_TABLE_TEST_1` `A2`
> ON `p`.cvalue = `A2`.cvalue;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14715) Hive throws NumberFormatException with query with Null value
[ https://issues.apache.org/jira/browse/HIVE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14715: --- Fix Version/s: 2,1 > Hive throws NumberFormatException with query with Null value > > > Key: HIVE-14715 > URL: https://issues.apache.org/jira/browse/HIVE-14715 > Project: Hive > Issue Type: Bug >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14715.1.patch, HIVE-14715.2.patch > > > The java.lang.NumberFormatException will throw with following reproduce: > set hive.cbo.enable=false; > CREATE TABLE `paqtest`( > `c1` int, > `s1` string, > `s2` string, > `bn1` bigint) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert into paqtest values (58, '', 'ABC', 0); > SELECT > 'PM' AS cy, > c1, > NULL AS iused, > NULL AS itp, > s2, > NULL AS cvg, > NULL AS acavg, > sum(bn1) AS cca > FROM paqtest > WHERE (s1 IS NULL OR length(s1) = 0) > GROUP BY 'Pricing mismatch', c1, NULL, NULL, s2, NULL, NULL; > The stack like following: > java.lang.NumberFormatException: ABC > GroupByOperator.process(Object, int) line: 773 > ExecReducer.reduce(Object, Iterator, OutputCollector, Reporter) line: 236 > ReduceTask.runOldReducer(JobConf, TaskUmbilicalProtocol, TaskReporter, > RawKeyValueIterator, RawComparator, Class, Class) > line: 444 > ReduceTask.run(JobConf, TaskUmbilicalProtocol) line: 392 > LocalJobRunner$Job$ReduceTaskRunnable.run() line: 319 > Executors$RunnableAdapter.call() line: 471 > It works fine when hive.cbo.enable = true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14715) Hive throws NumberFormatException with query with Null value
[ https://issues.apache.org/jira/browse/HIVE-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14715: --- Fix Version/s: (was: 2,1) 2.1.1 > Hive throws NumberFormatException with query with Null value > > > Key: HIVE-14715 > URL: https://issues.apache.org/jira/browse/HIVE-14715 > Project: Hive > Issue Type: Bug >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14715.1.patch, HIVE-14715.2.patch > > > The java.lang.NumberFormatException will throw with following reproduce: > set hive.cbo.enable=false; > CREATE TABLE `paqtest`( > `c1` int, > `s1` string, > `s2` string, > `bn1` bigint) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert into paqtest values (58, '', 'ABC', 0); > SELECT > 'PM' AS cy, > c1, > NULL AS iused, > NULL AS itp, > s2, > NULL AS cvg, > NULL AS acavg, > sum(bn1) AS cca > FROM paqtest > WHERE (s1 IS NULL OR length(s1) = 0) > GROUP BY 'Pricing mismatch', c1, NULL, NULL, s2, NULL, NULL; > The stack like following: > java.lang.NumberFormatException: ABC > GroupByOperator.process(Object, int) line: 773 > ExecReducer.reduce(Object, Iterator, OutputCollector, Reporter) line: 236 > ReduceTask.runOldReducer(JobConf, TaskUmbilicalProtocol, TaskReporter, > RawKeyValueIterator, RawComparator, Class, Class) > line: 444 > ReduceTask.run(JobConf, TaskUmbilicalProtocol) line: 392 > LocalJobRunner$Job$ReduceTaskRunnable.run() line: 319 > Executors$RunnableAdapter.call() line: 471 > It works fine when hive.cbo.enable = true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14090) JDOExceptions thrown by the Metastore have their full stack trace returned to clients
[
https://issues.apache.org/jira/browse/HIVE-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14090:
---
Fix Version/s: 2.1.1
> JDOExceptions thrown by the Metastore have their full stack trace returned to
> clients
> -
>
> Key: HIVE-14090
> URL: https://issues.apache.org/jira/browse/HIVE-14090
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14090.1.patch, HIVE-14090.patch
>
>
> When user try to create any database or table with a name longer than 128
> characters:
> {code}
> create database
> test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongNametableFAIL;
> {code}
> It dumps the full exception stack-trace in a non-user-friendly message. The
> lends to relatively negative user-experience for Beeline users who hit this
> exception, they are generally not interested in the full stack-trace.
> The formatted stack-trace is below:
> {code}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:javax.jdo.JDOFatalUserException: Attempt to store value
> "test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongnametablefail2"
> in column "`NAME`" that has maximum length of 128. Please correct your data!
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:528)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:569)
> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy10.createDatabase(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:923)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:962)
> at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy12.create_database(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:8863)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:8847)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:707)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:702)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:702)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745) NestedThrowablesStackTrace: Attempt
> to store value
> "test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongnametablefail2"
> in column "`NAME`" that has maximum length of 128. Please correct your data!
> org.datanucleus.exceptions.NucleusUserException: Attempt to store value
>
[jira] [Updated] (HIVE-14296) Session count is not decremented when HS2 clients do not shutdown cleanly.
[
https://issues.apache.org/jira/browse/HIVE-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14296:
---
Fix Version/s: 2.1.1
> Session count is not decremented when HS2 clients do not shutdown cleanly.
> --
>
> Key: HIVE-14296
> URL: https://issues.apache.org/jira/browse/HIVE-14296
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14296.2-branch-2.1.patch, HIVE-14296.2.patch,
> HIVE-14296.patch
>
>
> When a JDBC client like beeline abruptly disconnects from HS2, the session
> gets closed on the serverside but the session count reported in the logs is
> incorrect. It never gets decremented.
> For example, I created 6 connections from the same instance of beeline to HS2.
> {code}
> 2016-07-20T15:05:17,987 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e], current sessions: 1
> .
> 2016-07-20T15:05:24,239 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7], current sessions: 2
> .
> 2016-07-20T15:05:25,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54], current sessions: 3
> .
> 2016-07-20T15:05:26,795 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf], current sessions: 4
> 2016-07-20T15:05:28,160 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d], current sessions: 5
> .
> 2016-07-20T15:05:29,136 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [88b630c0-f272-427d-8263-febfef8d], current sessions: 6
> {code}
> When I CNTRL-C the beeline process, in the HS2 logs I see
> {code}
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [88b630c0-f272-427d-8263-febfef8d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54]
> {code}
> The next time I connect to HS2 via beeline, I see
> {code}
> 2016-07-20T15:14:33,679 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> 2016-07-20T15:14:33,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created HDFS directory:
> /tmp/hive/hive/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,725 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created local directory:
> /var/folders/_3/0w477k4j5bjd6h967rw4vflwgp/T/ngangam/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,735 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState:
[jira] [Updated] (HIVE-14215) Displaying inconsistent CPU usage data with MR execution engine
[
https://issues.apache.org/jira/browse/HIVE-14215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14215:
---
Fix Version/s: 2.1.1
> Displaying inconsistent CPU usage data with MR execution engine
> ---
>
> Key: HIVE-14215
> URL: https://issues.apache.org/jira/browse/HIVE-14215
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14215.patch
>
>
> If the MR task is finished after printing the cumulative CPU time then there
> is the possibility to print inconsistent CPU usage information.
> Correct one:
> {noformat}
> 2016-07-12 11:31:42,961 Stage-3 map = 0%, reduce = 0%
> 2016-07-12 11:31:48,237 Stage-3 map = 100%, reduce = 0%, Cumulative CPU 2.5
> sec
> MapReduce Total cumulative CPU time: 2 seconds 500 msec
> Ended Job = job_1468321038188_0003
> MapReduce Jobs Launched:
> Stage-Stage-3: Map: 1 Cumulative CPU: 2.5 sec HDFS Read: 5864 HDFS Write:
> 103 SUCCESS
> Total MapReduce CPU Time Spent: 2 seconds 500 msec
> {noformat}
> One type of inconsistent data (easily reproducible one):
> {noformat}
> 2016-07-12 11:39:00,540 Stage-3 map = 0%, reduce = 0%
> Ended Job = job_1468321038188_0004
> MapReduce Jobs Launched:
> Stage-Stage-3: Map: 1 Cumulative CPU: 2.51 sec HDFS Read: 5864 HDFS
> Write: 103 SUCCESS
> Total MapReduce CPU Time Spent: 2 seconds 510 msec
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14538) beeline throws exceptions with parsing hive config when using !sh statement
[
https://issues.apache.org/jira/browse/HIVE-14538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14538:
---
Fix Version/s: 2.1.1
> beeline throws exceptions with parsing hive config when using !sh statement
> ---
>
> Key: HIVE-14538
> URL: https://issues.apache.org/jira/browse/HIVE-14538
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14538.1.patch, HIVE-14538.2-branch-2.1.patch,
> HIVE-14538.2.patch
>
>
> When beeline has a connection to a server, in some env it has following
> problem:
> {noformat}
> 0: jdbc:hive2://localhost> !verbose
> verbose: on
> 0: jdbc:hive2://localhost> !sh id
> java.lang.ArrayIndexOutOfBoundsException: 1
> at org.apache.hive.beeline.Commands.addConf(Commands.java:758)
> at org.apache.hive.beeline.Commands.getHiveConf(Commands.java:704)
> at org.apache.hive.beeline.Commands.sh(Commands.java:1002)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1081)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:845)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:482)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> 0: jdbc:hive2://localhost> !sh echo hello
> java.lang.ArrayIndexOutOfBoundsException: 1
> at org.apache.hive.beeline.Commands.addConf(Commands.java:758)
> at org.apache.hive.beeline.Commands.getHiveConf(Commands.java:704)
> at org.apache.hive.beeline.Commands.sh(Commands.java:1002)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1081)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:845)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:482)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> 0: jdbc:hive2://localhost>
> {noformat}
> Also it breaks if there is no connection established:
> {noformat}
> beeline> !sh id
> java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.createStatement(BeeLine.java:1897)
> at org.apache.hive.beeline.Commands.getConfInternal(Commands.java:724)
> at org.apache.hive.beeline.Commands.getHiveConf(Commands.java:702)
> at org.apache.hive.beeline.Commands.sh(Commands.java:1002)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1081)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:845)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:482)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
[jira] [Updated] (HIVE-14187) JDOPersistenceManager objects remain cached if MetaStoreClient#close is not called
[ https://issues.apache.org/jira/browse/HIVE-14187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14187: --- Fix Version/s: 2.1.1 > JDOPersistenceManager objects remain cached if MetaStoreClient#close is not > called > -- > > Key: HIVE-14187 > URL: https://issues.apache.org/jira/browse/HIVE-14187 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14187.1.patch, HIVE-14187.2.patch, > HIVE-14187.patch, HIVE-14187.patch > > > JDOPersistenceManager objects are cached in JDOPersistenceManagerFactory by > DataNuclues. > A new JDOPersistenceManager object gets created for every HMS thread since > ObjectStore is a thread local. > In non-embedded metastore mode, JDOPersistenceManager associated with a > thread only gets cleaned up if IMetaStoreClient#close is called by the client > (which calls ObjectStore#shutdown which calls JDOPersistenceManager#close > which in turn removes the object from cache in > JDOPersistenceManagerFactory#releasePersistenceManager > https://github.com/datanucleus/datanucleus-api-jdo/blob/master/src/main/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L1271), > i.e. the object will remain cached if client does not call close. > For example: If one interrupts out of hive CLI shell (instead of using > 'exit;' command), SessionState#close does not get called, and hence > IMetaStoreClient#close does not get called. > Instead of relying the client to call close, it's cleaner to automatically > perform RawStore related cleanup at the server end via deleteContext() which > gets called when the server detects a lost/closed connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13836) DbNotifications giving an error = Invalid state. Transaction has already started
[
https://issues.apache.org/jira/browse/HIVE-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-13836:
---
Fix Version/s: 2.1.1
> DbNotifications giving an error = Invalid state. Transaction has already
> started
>
>
> Key: HIVE-13836
> URL: https://issues.apache.org/jira/browse/HIVE-13836
> Project: Hive
> Issue Type: Bug
>Reporter: Nachiket Vaidya
>Assignee: Nachiket Vaidya
>Priority: Critical
> Labels: patch-available
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-13836.2-branch-2.1.patch, HIVE-13836.2.patch,
> HIVE-13836.patch
>
>
> I used pyhs2 python client to create tables/partitions in hive. I was working
> fine until I moved to multithreaded scripts which created 8 connections and
> ran DDL queries concurrently.
> I got the error as
> {noformat}
> 2016-05-04 17:49:26,226 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-4-thread-194]:
> HMSHandler Fatal error: Invalid state. Transaction has already started
> org.datanucleus.transaction.NucleusTransactionException: Invalid state.
> Transaction has already started
> at
> org.datanucleus.transaction.TransactionManager.begin(TransactionManager.java:47)
> at org.datanucleus.TransactionImpl.begin(TransactionImpl.java:131)
> at
> org.datanucleus.api.jdo.JDOTransaction.internalBegin(JDOTransaction.java:88)
> at
> org.datanucleus.api.jdo.JDOTransaction.begin(JDOTransaction.java:80)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.openTransaction(ObjectStore.java:463)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:7522)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy10.addNotificationEvent(Unknown Source)
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener.enqueue(DbNotificationListener.java:261)
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:123)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1483)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1502)
> at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at
> com.sun.proxy.$Proxy14.create_table_with_environment_context(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:9267)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14772) NPE when MSCK REPAIR
[
https://issues.apache.org/jira/browse/HIVE-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530892#comment-15530892
]
Anbu Cheeralan commented on HIVE-14772:
---
https://github.com/apache/hive/commit/ab605910e45cfa65b9895399f09194801f2cc091
Is this the solution that fixed the issue in master branch?
> NPE when MSCK REPAIR
>
>
> Key: HIVE-14772
> URL: https://issues.apache.org/jira/browse/HIVE-14772
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.1.0
> Environment: HiveRunner on OSX Yosemite
>Reporter: Per Ullberg
>
> HiveMetaStoreChecker throws NullPointerException when doing a MSCK REPAIR
> TABLE.
> The bug is here:
> {code}
> ...
> 18 package org.apache.hadoop.hive.ql.metadata;
> ...
> 58 public class HiveMetaStoreChecker {
> ...
> 408if (!directoryFound) {
> 409 allDirs.put(path, null);
> 410}
> ...
> {code}
> allDirs is a ConcurrentHashMap and those does not allow either key or value
> to be null.
> I found the bug while trying to port https://github.com/klarna/HiveRunner to
> Hive 2.1.0
> Implemented explicit test case that exposes the bug here:
> https://github.com/klarna/HiveRunner/blob/hive-2.1.0-NPE-at-msck-repair/src/test/java/com/klarna/hiverunner/MSCKRepairNPE.java
> Reproduce by cloning branch
> https://github.com/klarna/HiveRunner/tree/hive-2.1.0-NPE-at-msck-repair
> and run
> {code} mvn -Dtest=MSCKRepairNPE clean test{code}
> ( Does not work on windows :( )
> Looks like this email thread talks about the same issue:
> http://user.hive.apache.narkive.com/ETOpbKk5/msck-repair-table-and-hive-v2-1-0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14822) Add support for credential provider for jobs launched from Hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-14822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530873#comment-15530873
]
Hive QA commented on HIVE-14822:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12830598/HIVE-14822.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 10651 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_stats_orc]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[archive_excludeHadoop20]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_8]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fetch_aggregation]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nonmr_fetch]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_analyze]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample10]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample_islocalmode_hook]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample_islocalmode_hook_use_metadata]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_parquet]
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[dboutput]
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testCliDriver[case_with_row_sequence]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapPlan1
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapPlan2
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan1
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan2
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan4
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan5
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan6
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1332/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1332/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1332/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12830598 - PreCommit-HIVE-Build
> Add support for credential provider for jobs launched from Hiveserver2
> --
>
> Key: HIVE-14822
> URL: https://issues.apache.org/jira/browse/HIVE-14822
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-14822.01.patch
>
>
> When using encrypted passwords via the Hadoop Credential Provider,
> HiveServer2 currently does not correctly forward enough information to the
> job configuration for jobs to read those secrets. If your job needs to access
> any secrets, like S3 credentials, then there's no convenient and secure way
> to configure this today.
> You could specify the decryption key in files like mapred-site.xml that
> HiveServer2 uses, but this would place the encryption password on local disk
> in plaintext, which can be a security concern.
> To solve this problem, HiveServer2 should modify job configuration to include
> the environment variable settings needed to decrypt the passwords.
> Specifically, it will need to modify:
> * For MR2 jobs:
> ** yarn.app.mapreduce.am.admin.user.env
> ** mapreduce.admin.user.env
> * For Spark jobs:
> ** spark.yarn.appMasterEnv.HADOOP_CREDSTORE_PASSWORD
> ** spark.executorEnv.HADOOP_CREDSTORE_PASSWORD
> HiveServer2 can get the decryption password from its own environment, the
> same way it does for its own credential provider store today.
> Additionally, it can be desirable for HiveServer2 to have a separate
> encrypted password file than what is used by the job. HiveServer2 may have
> secrets that the job should not have, such as the metastore database password
> or the password to decrypt its private SSL certificate. It is also best
[jira] [Commented] (HIVE-14852) Change qtest logging to not redirect all logs to console
[ https://issues.apache.org/jira/browse/HIVE-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530836#comment-15530836 ] Prasanth Jayachandran commented on HIVE-14852: -- bq. Most systems don't log test output to a separate file. Say if I run TestOrcFile, I would like to see what test has failed and what assertion is causing the failure. But I don't want to see all debug level logs getting printed to the console. It used to the case. Will this behavior change? bq. To avoid an extra file, we could try setting up log4j properties which suppresses output to the console, instead of creating a new file. I am missing something here. If root logger is DRFA why do we even see something in the console? Is it because of the recent patch that logs to console as well. > Change qtest logging to not redirect all logs to console > > > Key: HIVE-14852 > URL: https://issues.apache.org/jira/browse/HIVE-14852 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14852.01.patch > > > A change was made recently to redirect all logs to console, to make IDE > debugging of regular tests easier. That unfortunately makes qtest debugging > tougher - since there's a lot of noise along with the diffs in the output > file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14852) Change qtest logging to not redirect all logs to console
[ https://issues.apache.org/jira/browse/HIVE-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530819#comment-15530819 ] Siddharth Seth commented on HIVE-14852: --- Most systems don't log test output to a separate file. For qtests, I can see some use for this, however for the other tests - I think output should be to console. A separate profile for IDEs is very avoidable. To avoid an extra file, we could try setting up log4j properties which suppresses output to the console, instead of creating a new file. > Change qtest logging to not redirect all logs to console > > > Key: HIVE-14852 > URL: https://issues.apache.org/jira/browse/HIVE-14852 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14852.01.patch > > > A change was made recently to redirect all logs to console, to make IDE > debugging of regular tests easier. That unfortunately makes qtest debugging > tougher - since there's a lot of noise along with the diffs in the output > file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14824) Separate fstype from cluster type in QTestUtil
[ https://issues.apache.org/jira/browse/HIVE-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14824: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the review. > Separate fstype from cluster type in QTestUtil > -- > > Key: HIVE-14824 > URL: https://issues.apache.org/jira/browse/HIVE-14824 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.2.0 > > Attachments: HIVE-14824.01.patch, HIVE-14824.02.patch, > HIVE-14824.02.patch > > > QTestUtil cluster type encodes the file system. e.g. > MiniClusterType.encrypted means mr + encrypted hdfs, spark means file://, mr > means hdfs etc. > These can be separated out. e.g. To add tests for tez against encrypted, and > llap against encrypted - I'd need to introduce 2 new cluster types. > Instead it's better to separate the storage into it's own types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14852) Change qtest logging to not redirect all logs to console
[ https://issues.apache.org/jira/browse/HIVE-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530799#comment-15530799 ] Prasanth Jayachandran commented on HIVE-14852: -- is set twice? Is this patch trying to use different properties file for qtest and different properties file for all other unit tests? I think we should avoid adding another properties file as this it will get harder to find which file was used during initialization. We also bundle some properties files in the jars. Is it possible to reuse data/conf/hive-log4j2.properties file and write to file like we used to do before? For IDE, may be we can add a maven profile that gets activated only for IDEs that will print logs to console. > Change qtest logging to not redirect all logs to console > > > Key: HIVE-14852 > URL: https://issues.apache.org/jira/browse/HIVE-14852 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14852.01.patch > > > A change was made recently to redirect all logs to console, to make IDE > debugging of regular tests easier. That unfortunately makes qtest debugging > tougher - since there's a lot of noise along with the diffs in the output > file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14296) Session count is not decremented when HS2 clients do not shutdown cleanly.
[
https://issues.apache.org/jira/browse/HIVE-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-14296:
--
Attachment: HIVE-14296.2-branch-2.1.patch
Added a patch for branch 2.1
> Session count is not decremented when HS2 clients do not shutdown cleanly.
> --
>
> Key: HIVE-14296
> URL: https://issues.apache.org/jira/browse/HIVE-14296
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Peter Vary
> Fix For: 2.2.0
>
> Attachments: HIVE-14296.2-branch-2.1.patch, HIVE-14296.2.patch,
> HIVE-14296.patch
>
>
> When a JDBC client like beeline abruptly disconnects from HS2, the session
> gets closed on the serverside but the session count reported in the logs is
> incorrect. It never gets decremented.
> For example, I created 6 connections from the same instance of beeline to HS2.
> {code}
> 2016-07-20T15:05:17,987 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e], current sessions: 1
> .
> 2016-07-20T15:05:24,239 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7], current sessions: 2
> .
> 2016-07-20T15:05:25,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54], current sessions: 3
> .
> 2016-07-20T15:05:26,795 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf], current sessions: 4
> 2016-07-20T15:05:28,160 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d], current sessions: 5
> .
> 2016-07-20T15:05:29,136 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [88b630c0-f272-427d-8263-febfef8d], current sessions: 6
> {code}
> When I CNTRL-C the beeline process, in the HS2 logs I see
> {code}
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [88b630c0-f272-427d-8263-febfef8d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54]
> {code}
> The next time I connect to HS2 via beeline, I see
> {code}
> 2016-07-20T15:14:33,679 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> 2016-07-20T15:14:33,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created HDFS directory:
> /tmp/hive/hive/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,725 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created local directory:
> /var/folders/_3/0w477k4j5bjd6h967rw4vflwgp/T/ngangam/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,735 INFO [HiveServer2-Handler-Pool:
[jira] [Updated] (HIVE-14296) Session count is not decremented when HS2 clients do not shutdown cleanly.
[
https://issues.apache.org/jira/browse/HIVE-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-14296:
--
Assignee: Naveen Gangam (was: Peter Vary)
> Session count is not decremented when HS2 clients do not shutdown cleanly.
> --
>
> Key: HIVE-14296
> URL: https://issues.apache.org/jira/browse/HIVE-14296
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Fix For: 2.2.0
>
> Attachments: HIVE-14296.2-branch-2.1.patch, HIVE-14296.2.patch,
> HIVE-14296.patch
>
>
> When a JDBC client like beeline abruptly disconnects from HS2, the session
> gets closed on the serverside but the session count reported in the logs is
> incorrect. It never gets decremented.
> For example, I created 6 connections from the same instance of beeline to HS2.
> {code}
> 2016-07-20T15:05:17,987 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e], current sessions: 1
> .
> 2016-07-20T15:05:24,239 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7], current sessions: 2
> .
> 2016-07-20T15:05:25,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54], current sessions: 3
> .
> 2016-07-20T15:05:26,795 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf], current sessions: 4
> 2016-07-20T15:05:28,160 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d], current sessions: 5
> .
> 2016-07-20T15:05:29,136 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [88b630c0-f272-427d-8263-febfef8d], current sessions: 6
> {code}
> When I CNTRL-C the beeline process, in the HS2 logs I see
> {code}
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [88b630c0-f272-427d-8263-febfef8d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54]
> {code}
> The next time I connect to HS2 via beeline, I see
> {code}
> 2016-07-20T15:14:33,679 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> 2016-07-20T15:14:33,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created HDFS directory:
> /tmp/hive/hive/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,725 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created local directory:
> /var/folders/_3/0w477k4j5bjd6h967rw4vflwgp/T/ngangam/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,735 INFO [HiveServer2-Handler-Pool: Thread-50]
>
[jira] [Assigned] (HIVE-14296) Session count is not decremented when HS2 clients do not shutdown cleanly.
[
https://issues.apache.org/jira/browse/HIVE-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary reassigned HIVE-14296:
-
Assignee: Peter Vary (was: Naveen Gangam)
> Session count is not decremented when HS2 clients do not shutdown cleanly.
> --
>
> Key: HIVE-14296
> URL: https://issues.apache.org/jira/browse/HIVE-14296
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Peter Vary
> Fix For: 2.2.0
>
> Attachments: HIVE-14296.2-branch-2.1.patch, HIVE-14296.2.patch,
> HIVE-14296.patch
>
>
> When a JDBC client like beeline abruptly disconnects from HS2, the session
> gets closed on the serverside but the session count reported in the logs is
> incorrect. It never gets decremented.
> For example, I created 6 connections from the same instance of beeline to HS2.
> {code}
> 2016-07-20T15:05:17,987 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e], current sessions: 1
> .
> 2016-07-20T15:05:24,239 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7], current sessions: 2
> .
> 2016-07-20T15:05:25,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54], current sessions: 3
> .
> 2016-07-20T15:05:26,795 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf], current sessions: 4
> 2016-07-20T15:05:28,160 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d], current sessions: 5
> .
> 2016-07-20T15:05:29,136 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [88b630c0-f272-427d-8263-febfef8d], current sessions: 6
> {code}
> When I CNTRL-C the beeline process, in the HS2 logs I see
> {code}
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [88b630c0-f272-427d-8263-febfef8d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54]
> {code}
> The next time I connect to HS2 via beeline, I see
> {code}
> 2016-07-20T15:14:33,679 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> 2016-07-20T15:14:33,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created HDFS directory:
> /tmp/hive/hive/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,725 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created local directory:
> /var/folders/_3/0w477k4j5bjd6h967rw4vflwgp/T/ngangam/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,735 INFO [HiveServer2-Handler-Pool: Thread-50]
>
[jira] [Updated] (HIVE-14538) beeline throws exceptions with parsing hive config when using !sh statement
[
https://issues.apache.org/jira/browse/HIVE-14538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-14538:
--
Attachment: HIVE-14538.2-branch-2.1.patch
Added a patch for branch 2.1
> beeline throws exceptions with parsing hive config when using !sh statement
> ---
>
> Key: HIVE-14538
> URL: https://issues.apache.org/jira/browse/HIVE-14538
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Peter Vary
> Fix For: 2.2.0
>
> Attachments: HIVE-14538.1.patch, HIVE-14538.2-branch-2.1.patch,
> HIVE-14538.2.patch
>
>
> When beeline has a connection to a server, in some env it has following
> problem:
> {noformat}
> 0: jdbc:hive2://localhost> !verbose
> verbose: on
> 0: jdbc:hive2://localhost> !sh id
> java.lang.ArrayIndexOutOfBoundsException: 1
> at org.apache.hive.beeline.Commands.addConf(Commands.java:758)
> at org.apache.hive.beeline.Commands.getHiveConf(Commands.java:704)
> at org.apache.hive.beeline.Commands.sh(Commands.java:1002)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1081)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:845)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:482)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> 0: jdbc:hive2://localhost> !sh echo hello
> java.lang.ArrayIndexOutOfBoundsException: 1
> at org.apache.hive.beeline.Commands.addConf(Commands.java:758)
> at org.apache.hive.beeline.Commands.getHiveConf(Commands.java:704)
> at org.apache.hive.beeline.Commands.sh(Commands.java:1002)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1081)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:845)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:482)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> 0: jdbc:hive2://localhost>
> {noformat}
> Also it breaks if there is no connection established:
> {noformat}
> beeline> !sh id
> java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.createStatement(BeeLine.java:1897)
> at org.apache.hive.beeline.Commands.getConfInternal(Commands.java:724)
> at org.apache.hive.beeline.Commands.getHiveConf(Commands.java:702)
> at org.apache.hive.beeline.Commands.sh(Commands.java:1002)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1081)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:845)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:482)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)
> at
[jira] [Assigned] (HIVE-14538) beeline throws exceptions with parsing hive config when using !sh statement
[
https://issues.apache.org/jira/browse/HIVE-14538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary reassigned HIVE-14538:
-
Assignee: Peter Vary (was: Yongzhi Chen)
> beeline throws exceptions with parsing hive config when using !sh statement
> ---
>
> Key: HIVE-14538
> URL: https://issues.apache.org/jira/browse/HIVE-14538
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Peter Vary
> Fix For: 2.2.0
>
> Attachments: HIVE-14538.1.patch, HIVE-14538.2-branch-2.1.patch,
> HIVE-14538.2.patch
>
>
> When beeline has a connection to a server, in some env it has following
> problem:
> {noformat}
> 0: jdbc:hive2://localhost> !verbose
> verbose: on
> 0: jdbc:hive2://localhost> !sh id
> java.lang.ArrayIndexOutOfBoundsException: 1
> at org.apache.hive.beeline.Commands.addConf(Commands.java:758)
> at org.apache.hive.beeline.Commands.getHiveConf(Commands.java:704)
> at org.apache.hive.beeline.Commands.sh(Commands.java:1002)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1081)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:845)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:482)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> 0: jdbc:hive2://localhost> !sh echo hello
> java.lang.ArrayIndexOutOfBoundsException: 1
> at org.apache.hive.beeline.Commands.addConf(Commands.java:758)
> at org.apache.hive.beeline.Commands.getHiveConf(Commands.java:704)
> at org.apache.hive.beeline.Commands.sh(Commands.java:1002)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1081)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:845)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:482)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> 0: jdbc:hive2://localhost>
> {noformat}
> Also it breaks if there is no connection established:
> {noformat}
> beeline> !sh id
> java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.createStatement(BeeLine.java:1897)
> at org.apache.hive.beeline.Commands.getConfInternal(Commands.java:724)
> at org.apache.hive.beeline.Commands.getHiveConf(Commands.java:702)
> at org.apache.hive.beeline.Commands.sh(Commands.java:1002)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1081)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:845)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:482)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)
> at
[jira] [Updated] (HIVE-14538) beeline throws exceptions with parsing hive config when using !sh statement
[
https://issues.apache.org/jira/browse/HIVE-14538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-14538:
--
Assignee: Yongzhi Chen (was: Peter Vary)
> beeline throws exceptions with parsing hive config when using !sh statement
> ---
>
> Key: HIVE-14538
> URL: https://issues.apache.org/jira/browse/HIVE-14538
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Fix For: 2.2.0
>
> Attachments: HIVE-14538.1.patch, HIVE-14538.2-branch-2.1.patch,
> HIVE-14538.2.patch
>
>
> When beeline has a connection to a server, in some env it has following
> problem:
> {noformat}
> 0: jdbc:hive2://localhost> !verbose
> verbose: on
> 0: jdbc:hive2://localhost> !sh id
> java.lang.ArrayIndexOutOfBoundsException: 1
> at org.apache.hive.beeline.Commands.addConf(Commands.java:758)
> at org.apache.hive.beeline.Commands.getHiveConf(Commands.java:704)
> at org.apache.hive.beeline.Commands.sh(Commands.java:1002)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1081)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:845)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:482)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> 0: jdbc:hive2://localhost> !sh echo hello
> java.lang.ArrayIndexOutOfBoundsException: 1
> at org.apache.hive.beeline.Commands.addConf(Commands.java:758)
> at org.apache.hive.beeline.Commands.getHiveConf(Commands.java:704)
> at org.apache.hive.beeline.Commands.sh(Commands.java:1002)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1081)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:845)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:482)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> 0: jdbc:hive2://localhost>
> {noformat}
> Also it breaks if there is no connection established:
> {noformat}
> beeline> !sh id
> java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.createStatement(BeeLine.java:1897)
> at org.apache.hive.beeline.Commands.getConfInternal(Commands.java:724)
> at org.apache.hive.beeline.Commands.getHiveConf(Commands.java:702)
> at org.apache.hive.beeline.Commands.sh(Commands.java:1002)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1081)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:917)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:845)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:482)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:465)
> at
[jira] [Updated] (HIVE-14137) Hive on Spark throws FileAlreadyExistsException for jobs with multiple empty tables
[
https://issues.apache.org/jira/browse/HIVE-14137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-14137:
--
Assignee: Sahil Takiar (was: Peter Vary)
> Hive on Spark throws FileAlreadyExistsException for jobs with multiple empty
> tables
> ---
>
> Key: HIVE-14137
> URL: https://issues.apache.org/jira/browse/HIVE-14137
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Fix For: 2.2.0
>
> Attachments: HIVE-14137.1.patch, HIVE-14137.2.patch,
> HIVE-14137.3.patch, HIVE-14137.4.patch, HIVE-14137.5.patch,
> HIVE-14137.6-branch-2.1.patch, HIVE-14137.6.patch, HIVE-14137.patch
>
>
> The following queries:
> {code}
> -- Setup
> drop table if exists empty1;
> create table empty1 (col1 bigint) stored as parquet tblproperties
> ('parquet.compress'='snappy');
> drop table if exists empty2;
> create table empty2 (col1 bigint, col2 bigint) stored as parquet
> tblproperties ('parquet.compress'='snappy');
> drop table if exists empty3;
> create table empty3 (col1 bigint) stored as parquet tblproperties
> ('parquet.compress'='snappy');
> -- All empty HDFS directories.
> -- Fails with [08S01]: Error while processing statement: FAILED: Execution
> Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask.
> select empty1.col1
> from empty1
> inner join empty2
> on empty2.col1 = empty1.col1
> inner join empty3
> on empty3.col1 = empty2.col2;
> -- Two empty HDFS directories.
> -- Create an empty file in HDFS.
> insert into empty1 select * from empty1 where false;
> -- Same query fails with [08S01]: Error while processing statement: FAILED:
> Execution Error, return code 3 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.
> select empty1.col1
> from empty1
> inner join empty2
> on empty2.col1 = empty1.col1
> inner join empty3
> on empty3.col1 = empty2.col2;
> -- One empty HDFS directory.
> -- Create an empty file in HDFS.
> insert into empty2 select * from empty2 where false;
> -- Same query succeeds.
> select empty1.col1
> from empty1
> inner join empty2
> on empty2.col1 = empty1.col1
> inner join empty3
> on empty3.col1 = empty2.col2;
> {code}
> Will result in the following exception:
> {code}
> org.apache.hadoop.fs.FileAlreadyExistsException:
> /tmp/hive/hive/1f3837aa-9407-4780-92b1-42a66d205139/hive_2016-06-24_15-45-23_206_79177714958655528-2/-mr-10004/0/emptyFile
> for client 172.26.14.151 already exists
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2676)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2561)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:593)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.create(AuthorizationProviderProxyClientProtocol.java:111)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:393)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1902)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1738)
> at
[jira] [Assigned] (HIVE-14137) Hive on Spark throws FileAlreadyExistsException for jobs with multiple empty tables
[
https://issues.apache.org/jira/browse/HIVE-14137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary reassigned HIVE-14137:
-
Assignee: Peter Vary (was: Sahil Takiar)
> Hive on Spark throws FileAlreadyExistsException for jobs with multiple empty
> tables
> ---
>
> Key: HIVE-14137
> URL: https://issues.apache.org/jira/browse/HIVE-14137
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Peter Vary
> Fix For: 2.2.0
>
> Attachments: HIVE-14137.1.patch, HIVE-14137.2.patch,
> HIVE-14137.3.patch, HIVE-14137.4.patch, HIVE-14137.5.patch,
> HIVE-14137.6-branch-2.1.patch, HIVE-14137.6.patch, HIVE-14137.patch
>
>
> The following queries:
> {code}
> -- Setup
> drop table if exists empty1;
> create table empty1 (col1 bigint) stored as parquet tblproperties
> ('parquet.compress'='snappy');
> drop table if exists empty2;
> create table empty2 (col1 bigint, col2 bigint) stored as parquet
> tblproperties ('parquet.compress'='snappy');
> drop table if exists empty3;
> create table empty3 (col1 bigint) stored as parquet tblproperties
> ('parquet.compress'='snappy');
> -- All empty HDFS directories.
> -- Fails with [08S01]: Error while processing statement: FAILED: Execution
> Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask.
> select empty1.col1
> from empty1
> inner join empty2
> on empty2.col1 = empty1.col1
> inner join empty3
> on empty3.col1 = empty2.col2;
> -- Two empty HDFS directories.
> -- Create an empty file in HDFS.
> insert into empty1 select * from empty1 where false;
> -- Same query fails with [08S01]: Error while processing statement: FAILED:
> Execution Error, return code 3 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.
> select empty1.col1
> from empty1
> inner join empty2
> on empty2.col1 = empty1.col1
> inner join empty3
> on empty3.col1 = empty2.col2;
> -- One empty HDFS directory.
> -- Create an empty file in HDFS.
> insert into empty2 select * from empty2 where false;
> -- Same query succeeds.
> select empty1.col1
> from empty1
> inner join empty2
> on empty2.col1 = empty1.col1
> inner join empty3
> on empty3.col1 = empty2.col2;
> {code}
> Will result in the following exception:
> {code}
> org.apache.hadoop.fs.FileAlreadyExistsException:
> /tmp/hive/hive/1f3837aa-9407-4780-92b1-42a66d205139/hive_2016-06-24_15-45-23_206_79177714958655528-2/-mr-10004/0/emptyFile
> for client 172.26.14.151 already exists
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2676)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2561)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:593)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.create(AuthorizationProviderProxyClientProtocol.java:111)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:393)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1902)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1738)
> at
[jira] [Updated] (HIVE-14137) Hive on Spark throws FileAlreadyExistsException for jobs with multiple empty tables
[
https://issues.apache.org/jira/browse/HIVE-14137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-14137:
--
Attachment: HIVE-14137.6-branch-2.1.patch
Added a patch for branch 2.1
> Hive on Spark throws FileAlreadyExistsException for jobs with multiple empty
> tables
> ---
>
> Key: HIVE-14137
> URL: https://issues.apache.org/jira/browse/HIVE-14137
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Peter Vary
> Fix For: 2.2.0
>
> Attachments: HIVE-14137.1.patch, HIVE-14137.2.patch,
> HIVE-14137.3.patch, HIVE-14137.4.patch, HIVE-14137.5.patch,
> HIVE-14137.6-branch-2.1.patch, HIVE-14137.6.patch, HIVE-14137.patch
>
>
> The following queries:
> {code}
> -- Setup
> drop table if exists empty1;
> create table empty1 (col1 bigint) stored as parquet tblproperties
> ('parquet.compress'='snappy');
> drop table if exists empty2;
> create table empty2 (col1 bigint, col2 bigint) stored as parquet
> tblproperties ('parquet.compress'='snappy');
> drop table if exists empty3;
> create table empty3 (col1 bigint) stored as parquet tblproperties
> ('parquet.compress'='snappy');
> -- All empty HDFS directories.
> -- Fails with [08S01]: Error while processing statement: FAILED: Execution
> Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask.
> select empty1.col1
> from empty1
> inner join empty2
> on empty2.col1 = empty1.col1
> inner join empty3
> on empty3.col1 = empty2.col2;
> -- Two empty HDFS directories.
> -- Create an empty file in HDFS.
> insert into empty1 select * from empty1 where false;
> -- Same query fails with [08S01]: Error while processing statement: FAILED:
> Execution Error, return code 3 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.
> select empty1.col1
> from empty1
> inner join empty2
> on empty2.col1 = empty1.col1
> inner join empty3
> on empty3.col1 = empty2.col2;
> -- One empty HDFS directory.
> -- Create an empty file in HDFS.
> insert into empty2 select * from empty2 where false;
> -- Same query succeeds.
> select empty1.col1
> from empty1
> inner join empty2
> on empty2.col1 = empty1.col1
> inner join empty3
> on empty3.col1 = empty2.col2;
> {code}
> Will result in the following exception:
> {code}
> org.apache.hadoop.fs.FileAlreadyExistsException:
> /tmp/hive/hive/1f3837aa-9407-4780-92b1-42a66d205139/hive_2016-06-24_15-45-23_206_79177714958655528-2/-mr-10004/0/emptyFile
> for client 172.26.14.151 already exists
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2676)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2561)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:593)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.create(AuthorizationProviderProxyClientProtocol.java:111)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:393)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1902)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1738)
> at
[jira] [Commented] (HIVE-14830) Move a majority of the MiniLlapCliDriver tests to use an inline AM
[ https://issues.apache.org/jira/browse/HIVE-14830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530698#comment-15530698 ] Prasanth Jayachandran commented on HIVE-14830: -- cool! +1 for this patch > Move a majority of the MiniLlapCliDriver tests to use an inline AM > -- > > Key: HIVE-14830 > URL: https://issues.apache.org/jira/browse/HIVE-14830 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14830.01.patch, HIVE-14830.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14824) Separate fstype from cluster type in QTestUtil
[ https://issues.apache.org/jira/browse/HIVE-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530679#comment-15530679 ] Prasanth Jayachandran commented on HIVE-14824: -- lgtm, +1 > Separate fstype from cluster type in QTestUtil > -- > > Key: HIVE-14824 > URL: https://issues.apache.org/jira/browse/HIVE-14824 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14824.01.patch, HIVE-14824.02.patch, > HIVE-14824.02.patch > > > QTestUtil cluster type encodes the file system. e.g. > MiniClusterType.encrypted means mr + encrypted hdfs, spark means file://, mr > means hdfs etc. > These can be separated out. e.g. To add tests for tez against encrypted, and > llap against encrypted - I'd need to introduce 2 new cluster types. > Instead it's better to separate the storage into it's own types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11812) datediff sometimes returns incorrect results when called with dates
[
https://issues.apache.org/jira/browse/HIVE-11812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530549#comment-15530549
]
Jason Dere commented on HIVE-11812:
---
The added test in TestGenericUDFDateDiff.testDateDiffEvaluate() does not
actually fail for me on Hive master .. what time zone are you running these
tests?
Also there are a number of existing .q file tests failing here. For date_udf
looks like the expected output is incorrect and needs to be regenerated, but
for timestamp_udf and vectorized_date_funcs I think the expected results are
correct, and the patch seems to have broken them.
> datediff sometimes returns incorrect results when called with dates
> ---
>
> Key: HIVE-11812
> URL: https://issues.apache.org/jira/browse/HIVE-11812
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 2.0.0
>Reporter: Nicholas Brenwald
>Assignee: Chetna Chaudhari
>Priority: Minor
> Attachments: HIVE-11812.1.patch
>
>
> DATEDIFF returns an incorrect result when one of the arguments is a date
> type.
> The Hive Language Manual provides the following signature for datediff:
> {code}
> int datediff(string enddate, string startdate)
> {code}
> I think datediff should either throw an error (if date types are not
> supported), or return the correct result.
> To reproduce, create a table:
> {code}
> create table t (c1 string, c2 date);
> {code}
> Assuming you have a table x containing some data, populate table t with 1 row:
> {code}
> insert into t select '2015-09-15', '2015-09-15' from x limit 1;
> {code}
> Then run the following 12 test queries:
> {code}
> select datediff(c1, '2015-09-14') from t;
> select datediff(c1, '2015-09-15') from t;
> select datediff(c1, '2015-09-16') from t;
> select datediff('2015-09-14', c1) from t;
> select datediff('2015-09-15', c1) from t;
> select datediff('2015-09-16', c1) from t;
> select datediff(c2, '2015-09-14') from t;
> select datediff(c2, '2015-09-15') from t;
> select datediff(c2, '2015-09-16') from t;
> select datediff('2015-09-14', c2) from t;
> select datediff('2015-09-15', c2) from t;
> select datediff('2015-09-16', c2) from t;
> {code}
> The below table summarises the result. All results for column c1 (which is a
> string) are correct, but when using c2 (which is a date), two of the results
> are incorrect.
> || Test || Expected Result || Actual Result || Passed / Failed ||
> |datediff(c1, '2015-09-14')| 1 | 1| Passed |
> |datediff(c1, '2015-09-15')| 0 | 0| Passed |
> |datediff(c1, '2015-09-16') | -1 | -1| Passed |
> |datediff('2015-09-14', c1) | -1 | -1| Passed |
> |datediff('2015-09-15', c1)| 0 | 0| Passed |
> |datediff('2015-09-16', c1)| 1 | 1| Passed |
> |datediff(c2, '2015-09-14')| 1 | 0| {color:red}Failed{color} |
> |datediff(c2, '2015-09-15')| 0 | 0| Passed |
> |datediff(c2, '2015-09-16') | -1 | -1| Passed |
> |datediff('2015-09-14', c2) | -1 | 0 | {color:red}Failed{color} |
> |datediff('2015-09-15', c2)| 0 | 0| Passed |
> |datediff('2015-09-16', c2)| 1 | 1| Passed |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-1539) Concurrent metastore threading problem
[ https://issues.apache.org/jira/browse/HIVE-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530543#comment-15530543 ] Sharath MK commented on HIVE-1539: -- I'm facing the same problem. I'm not able to run concurrent query using JDBC java program. If any one solved, please direct me to some example java program. Thanks > Concurrent metastore threading problem > --- > > Key: HIVE-1539 > URL: https://issues.apache.org/jira/browse/HIVE-1539 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.7.0 >Reporter: Bennie Schut >Assignee: Bennie Schut > Attachments: ClassLoaderResolver.patch, HIVE-1539-1.patch, > HIVE-1539.patch, thread_dump_hanging.txt > > > When running hive as a service and running a high number of queries > concurrently I end up with multiple threads running at 100% cpu without any > progress. > Looking at these threads I notice this thread(484e): > at > org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:598) > But on a different thread(63a2): > at > org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceField(MStorageDescriptor.java) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14851) Trying to run two query concurrently, but getting error.
[ https://issues.apache.org/jira/browse/HIVE-14851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharath MK updated HIVE-14851: -- Attachment: hiveserver2.log Hiveserver logs > Trying to run two query concurrently, but getting error. > - > > Key: HIVE-14851 > URL: https://issues.apache.org/jira/browse/HIVE-14851 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.2.1 > Environment: Hortonworks >Reporter: Sharath MK >Priority: Blocker > Attachments: hive_error.PNG, hiveserver2.log > > > Hi, > I'm trying to use same hive connection object and running two thread by > passing connection object. > In Runnable class constructor i'm creating statement object and in run > function i'm trying to query and displaying result. > Please help me how to run concurrent query using JDBC java program. > But when i try running i'm getting below error, > java.sql.SQLException: org.apache.thrift.transport.TTransportException > at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:262) > at > org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:392) > at com.infy.Runn.executeQuery(Runn.java:51) > at com.infy.Runn.run(Runn.java:30) > at java.lang.Thread.run(Unknown Source) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14849) Support google-compute-engine provider on Hive ptest framework
[ https://issues.apache.org/jira/browse/HIVE-14849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14849: --- Attachment: HIVE-14849.1.patch > Support google-compute-engine provider on Hive ptest framework > -- > > Key: HIVE-14849 > URL: https://issues.apache.org/jira/browse/HIVE-14849 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-14849.1.patch > > > NO PRECOMMIT TESTS > Currently, Hive ptest only supports AWS EC2 to create a cluster. We should > add support for GCE as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14852) Change qtest logging to not redirect all logs to console
[ https://issues.apache.org/jira/browse/HIVE-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14852: -- Status: Patch Available (was: Open) > Change qtest logging to not redirect all logs to console > > > Key: HIVE-14852 > URL: https://issues.apache.org/jira/browse/HIVE-14852 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14852.01.patch > > > A change was made recently to redirect all logs to console, to make IDE > debugging of regular tests easier. That unfortunately makes qtest debugging > tougher - since there's a lot of noise along with the diffs in the output > file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14852) Change qtest logging to not redirect all logs to console
[ https://issues.apache.org/jira/browse/HIVE-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14852: -- Attachment: HIVE-14852.01.patch Patch to log all tests other than qtests to console. qtest will only write stdout etc to console. cc [~prasanth_j] for review. > Change qtest logging to not redirect all logs to console > > > Key: HIVE-14852 > URL: https://issues.apache.org/jira/browse/HIVE-14852 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14852.01.patch > > > A change was made recently to redirect all logs to console, to make IDE > debugging of regular tests easier. That unfortunately makes qtest debugging > tougher - since there's a lot of noise along with the diffs in the output > file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14852) Change qtest logging to not redirect all logs to console
[ https://issues.apache.org/jira/browse/HIVE-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth reassigned HIVE-14852: - Assignee: Siddharth Seth > Change qtest logging to not redirect all logs to console > > > Key: HIVE-14852 > URL: https://issues.apache.org/jira/browse/HIVE-14852 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14852.01.patch > > > A change was made recently to redirect all logs to console, to make IDE > debugging of regular tests easier. That unfortunately makes qtest debugging > tougher - since there's a lot of noise along with the diffs in the output > file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14851) Trying to run two query concurrently, but getting error.
[ https://issues.apache.org/jira/browse/HIVE-14851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharath MK updated HIVE-14851: -- Attachment: hive_error.PNG Error screenshot > Trying to run two query concurrently, but getting error. > - > > Key: HIVE-14851 > URL: https://issues.apache.org/jira/browse/HIVE-14851 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.2.1 > Environment: Hortonworks >Reporter: Sharath MK >Priority: Blocker > Attachments: hive_error.PNG > > > Hi, > I'm trying to use same hive connection object and running two thread by > passing connection object. > In Runnable class constructor i'm creating statement object and in run > function i'm trying to query and displaying result. > Please help me how to run concurrent query using JDBC java program. > But when i try running i'm getting below error, > java.sql.SQLException: org.apache.thrift.transport.TTransportException > at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:262) > at > org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:392) > at com.infy.Runn.executeQuery(Runn.java:51) > at com.infy.Runn.run(Runn.java:30) > at java.lang.Thread.run(Unknown Source) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14849) Support google-compute-engine provider on Hive ptest framework
[ https://issues.apache.org/jira/browse/HIVE-14849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14849: --- Attachment: (was: HIVE-14849.1.patch) > Support google-compute-engine provider on Hive ptest framework > -- > > Key: HIVE-14849 > URL: https://issues.apache.org/jira/browse/HIVE-14849 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > > NO PRECOMMIT TESTS > Currently, Hive ptest only supports AWS EC2 to create a cluster. We should > add support for GCE as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14830) Move a majority of the MiniLlapCliDriver tests to use an inline AM
[ https://issues.apache.org/jira/browse/HIVE-14830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530489#comment-15530489 ] Siddharth Seth commented on HIVE-14830: --- I want to get rid of setUseLocalMode - that only applies to MiniTez and MiniLlap. HIVE-14824 adds the FileSystem in a better way - that will get rid of useLocalFs. > Move a majority of the MiniLlapCliDriver tests to use an inline AM > -- > > Key: HIVE-14830 > URL: https://issues.apache.org/jira/browse/HIVE-14830 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14830.01.patch, HIVE-14830.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14100) current_user() returns invalid information
[
https://issues.apache.org/jira/browse/HIVE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530478#comment-15530478
]
Mohit Sabharwal commented on HIVE-14100:
Thanks [~pvary]], sounds good!
Your patch is getting user name from {{SessionState.get().getUserName()}}
HIVE-9143 is getting user name from {{SessionState.getUserFromAuthenticator()}}
which calls {{SessionStateUserAuthenticator.getUserName()}} which calls the
{{sessionState.getUserName()}} (same as your patch).
Just to confirm, you're saying that *if* SessionStateUserAuthenticator is the
not the default (which it is, in {{hiveServer2.cmd}} file ), then
other HiveAuthenticationProviders (like say HadoopDefaultAuthenticator), may
return user that runs the hiveserver2 process ?
> current_user() returns invalid information
> --
>
> Key: HIVE-14100
> URL: https://issues.apache.org/jira/browse/HIVE-14100
> Project: Hive
> Issue Type: Bug
> Components: Authentication, Beeline
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Attachments: HIVE-14100.2.patch, HIVE-14100.2.patch,
> HIVE-14100.2.patch, HIVE-14100.patch
>
>
> Using HadoopDeaultAuthenticator the current_user() returns the username of
> the unix user running hiveservice2.
> Using SessionStateUserAuthenticator the current_user returns the username
> which is provided when the connection started.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14819) FunctionInfo for permanent functions shows TEMPORARY FunctionType
[ https://issues.apache.org/jira/browse/HIVE-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530437#comment-15530437 ] Sergey Shelukhin commented on HIVE-14819: - +1 > FunctionInfo for permanent functions shows TEMPORARY FunctionType > - > > Key: HIVE-14819 > URL: https://issues.apache.org/jira/browse/HIVE-14819 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 2.1.0 >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14819.1.patch, HIVE-14819.2.patch > > > The FunctionInfo has a FunctionType field which describes if the function is > a builtin/persistent/temporary function. But for permanent functions, the > FunctionInfo being returned by the FunctionRegistry is showing the type to be > TEMPORARY. > This affects things which may be depending on function type, for example > LlapDecider, which will allow builtin/persistent UDFs to be used in LLAP but > not temporary functions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14775) Investigate IOException usage in Metrics APIs
[ https://issues.apache.org/jira/browse/HIVE-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530411#comment-15530411 ] Mohit Sabharwal commented on HIVE-14775: [~zsombor.klara] please attach & submit your patch to trigger tests. Thanks! > Investigate IOException usage in Metrics APIs > - > > Key: HIVE-14775 > URL: https://issues.apache.org/jira/browse/HIVE-14775 > Project: Hive > Issue Type: Sub-task > Components: Hive, HiveServer2, Metastore >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > > A large number of metrics APIs seem to declare to throw IOExceptions > needlessly. (incrementCounter, decrementCounter etc.) > This is not only misleading but it fills up the code with unnecessary catch > blocks never to be reached. > We should investigate if these exceptions are thrown at all, and remove them > if it is truly unused. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14784) Operation logs are disabled automatically if the parent directory does not exist.
[
https://issues.apache.org/jira/browse/HIVE-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-14784:
-
Attachment: HIVE-14784.1.patch
New patch with feedback
> Operation logs are disabled automatically if the parent directory does not
> exist.
> -
>
> Key: HIVE-14784
> URL: https://issues.apache.org/jira/browse/HIVE-14784
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14784.1.patch, HIVE-14784.patch
>
>
> Operation logging is disabled automatically for the query if for some reason
> the parent directory (named after the hive session id) that gets created when
> the session is established gets deleted (for any reason). For ex: if the
> operation logdir is /tmp which automatically can get purged at a configured
> interval by the OS.
> Running a query from that session leads to
> {code}
> 2016-09-15 15:09:16,723 WARN org.apache.hive.service.cli.operation.Operation:
> Unable to create operation log file:
> /tmp/hive/operation_logs/b8809985-6b38-47ec-a49b-6158a67cd9fc/d35414f7-2418-426c-8489-c6f643ca4599
> java.io.IOException: No such file or directory
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hive.service.cli.operation.Operation.createOperationLog(Operation.java:195)
> at
> org.apache.hive.service.cli.operation.Operation.beforeRun(Operation.java:237)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:255)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:398)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:385)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:271)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:490)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> This later leads to errors like (more prominent when using HUE as HUE does
> not close hive sessions and attempts to retrieve the operations logs days
> after they were created).
> {code}
> WARN org.apache.hive.service.cli.thrift.ThriftCLIService: Error fetching
> results:
> org.apache.hive.service.cli.HiveSQLException: Couldn't find log associated
> with operation handle: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=d35414f7-2418-426c-8489-c6f643ca4599]
> at
> org.apache.hive.service.cli.operation.OperationManager.getOperationLogRowSet(OperationManager.java:259)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:701)
> at
> org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:451)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:676)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by
[jira] [Updated] (HIVE-14784) Operation logs are disabled automatically if the parent directory does not exist.
[
https://issues.apache.org/jira/browse/HIVE-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-14784:
-
Status: Patch Available (was: Open)
> Operation logs are disabled automatically if the parent directory does not
> exist.
> -
>
> Key: HIVE-14784
> URL: https://issues.apache.org/jira/browse/HIVE-14784
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14784.1.patch, HIVE-14784.patch
>
>
> Operation logging is disabled automatically for the query if for some reason
> the parent directory (named after the hive session id) that gets created when
> the session is established gets deleted (for any reason). For ex: if the
> operation logdir is /tmp which automatically can get purged at a configured
> interval by the OS.
> Running a query from that session leads to
> {code}
> 2016-09-15 15:09:16,723 WARN org.apache.hive.service.cli.operation.Operation:
> Unable to create operation log file:
> /tmp/hive/operation_logs/b8809985-6b38-47ec-a49b-6158a67cd9fc/d35414f7-2418-426c-8489-c6f643ca4599
> java.io.IOException: No such file or directory
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hive.service.cli.operation.Operation.createOperationLog(Operation.java:195)
> at
> org.apache.hive.service.cli.operation.Operation.beforeRun(Operation.java:237)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:255)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:398)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:385)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:271)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:490)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> This later leads to errors like (more prominent when using HUE as HUE does
> not close hive sessions and attempts to retrieve the operations logs days
> after they were created).
> {code}
> WARN org.apache.hive.service.cli.thrift.ThriftCLIService: Error fetching
> results:
> org.apache.hive.service.cli.HiveSQLException: Couldn't find log associated
> with operation handle: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=d35414f7-2418-426c-8489-c6f643ca4599]
> at
> org.apache.hive.service.cli.operation.OperationManager.getOperationLogRowSet(OperationManager.java:259)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:701)
> at
> org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:451)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:676)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
[jira] [Updated] (HIVE-14784) Operation logs are disabled automatically if the parent directory does not exist.
[
https://issues.apache.org/jira/browse/HIVE-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-14784:
-
Status: Open (was: Patch Available)
Will upload a patch with slightly modified fix.
> Operation logs are disabled automatically if the parent directory does not
> exist.
> -
>
> Key: HIVE-14784
> URL: https://issues.apache.org/jira/browse/HIVE-14784
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14784.patch
>
>
> Operation logging is disabled automatically for the query if for some reason
> the parent directory (named after the hive session id) that gets created when
> the session is established gets deleted (for any reason). For ex: if the
> operation logdir is /tmp which automatically can get purged at a configured
> interval by the OS.
> Running a query from that session leads to
> {code}
> 2016-09-15 15:09:16,723 WARN org.apache.hive.service.cli.operation.Operation:
> Unable to create operation log file:
> /tmp/hive/operation_logs/b8809985-6b38-47ec-a49b-6158a67cd9fc/d35414f7-2418-426c-8489-c6f643ca4599
> java.io.IOException: No such file or directory
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hive.service.cli.operation.Operation.createOperationLog(Operation.java:195)
> at
> org.apache.hive.service.cli.operation.Operation.beforeRun(Operation.java:237)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:255)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:398)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:385)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:271)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:490)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> This later leads to errors like (more prominent when using HUE as HUE does
> not close hive sessions and attempts to retrieve the operations logs days
> after they were created).
> {code}
> WARN org.apache.hive.service.cli.thrift.ThriftCLIService: Error fetching
> results:
> org.apache.hive.service.cli.HiveSQLException: Couldn't find log associated
> with operation handle: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=d35414f7-2418-426c-8489-c6f643ca4599]
> at
> org.apache.hive.service.cli.operation.OperationManager.getOperationLogRowSet(OperationManager.java:259)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:701)
> at
> org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:451)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:676)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent
[jira] [Commented] (HIVE-14775) Investigate IOException usage in Metrics APIs
[ https://issues.apache.org/jira/browse/HIVE-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530366#comment-15530366 ] Szehon Ho commented on HIVE-14775: -- Makes sense, +1 on latest patch to me. > Investigate IOException usage in Metrics APIs > - > > Key: HIVE-14775 > URL: https://issues.apache.org/jira/browse/HIVE-14775 > Project: Hive > Issue Type: Sub-task > Components: Hive, HiveServer2, Metastore >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > > A large number of metrics APIs seem to declare to throw IOExceptions > needlessly. (incrementCounter, decrementCounter etc.) > This is not only misleading but it fills up the code with unnecessary catch > blocks never to be reached. > We should investigate if these exceptions are thrown at all, and remove them > if it is truly unused. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14849) Support google-compute-engine provider on Hive ptest framework
[ https://issues.apache.org/jira/browse/HIVE-14849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530340#comment-15530340 ] Prasanth Jayachandran commented on HIVE-14849: -- [~spena] can you please upload patch to RB or give a PR? > Support google-compute-engine provider on Hive ptest framework > -- > > Key: HIVE-14849 > URL: https://issues.apache.org/jira/browse/HIVE-14849 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-14849.1.patch > > > NO PRECOMMIT TESTS > Currently, Hive ptest only supports AWS EC2 to create a cluster. We should > add support for GCE as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14849) Support google-compute-engine provider on Hive ptest framework
[ https://issues.apache.org/jira/browse/HIVE-14849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14849: --- Status: Patch Available (was: Open) [~sseth] [~prasanth_j] I added suport for GCE, could you help me review this? I'm gonna switch our current Hive ptest slaves to GCE instead of AWS. > Support google-compute-engine provider on Hive ptest framework > -- > > Key: HIVE-14849 > URL: https://issues.apache.org/jira/browse/HIVE-14849 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-14849.1.patch > > > NO PRECOMMIT TESTS > Currently, Hive ptest only supports AWS EC2 to create a cluster. We should > add support for GCE as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14063) beeline to auto connect to the HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-14063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-14063: --- Status: Open (was: Patch Available) Need to sync up the latest code changes and make some changes accordingly. Will publish another patch soon > beeline to auto connect to the HiveServer2 > -- > > Key: HIVE-14063 > URL: https://issues.apache.org/jira/browse/HIVE-14063 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Minor > Attachments: HIVE-14063.01.patch, beeline.conf.template > > > Currently one has to give an jdbc:hive2 url in order for Beeline to connect a > hiveserver2 instance. It would be great if Beeline can get the info somehow > (from a properties file at a well-known location?) and connect automatically > if user doesn't specify such a url. If the properties file is not present, > then beeline would expect user to provide the url and credentials using > !connect or ./beeline -u .. commands > While Beeline is flexible (being a mere JDBC client), most environments would > have just a single HS2. Having users to manually connect into this via either > "beeline ~/.propsfile" or -u or !connect statements is lowering the > experience part. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14849) Support google-compute-engine provider on Hive ptest framework
[ https://issues.apache.org/jira/browse/HIVE-14849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14849: --- Description: NO PRECOMMIT TESTS Currently, Hive ptest only supports AWS EC2 to create a cluster. We should add support for GCE as well. was:Currently, Hive ptest only supports AWS EC2 to create a cluster. We should add support for GCE as well. > Support google-compute-engine provider on Hive ptest framework > -- > > Key: HIVE-14849 > URL: https://issues.apache.org/jira/browse/HIVE-14849 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-14849.1.patch > > > NO PRECOMMIT TESTS > Currently, Hive ptest only supports AWS EC2 to create a cluster. We should > add support for GCE as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14822) Add support for credential provider for jobs launched from Hiveserver2
[ https://issues.apache.org/jira/browse/HIVE-14822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-14822: --- Status: Patch Available (was: Open) > Add support for credential provider for jobs launched from Hiveserver2 > -- > > Key: HIVE-14822 > URL: https://issues.apache.org/jira/browse/HIVE-14822 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-14822.01.patch > > > When using encrypted passwords via the Hadoop Credential Provider, > HiveServer2 currently does not correctly forward enough information to the > job configuration for jobs to read those secrets. If your job needs to access > any secrets, like S3 credentials, then there's no convenient and secure way > to configure this today. > You could specify the decryption key in files like mapred-site.xml that > HiveServer2 uses, but this would place the encryption password on local disk > in plaintext, which can be a security concern. > To solve this problem, HiveServer2 should modify job configuration to include > the environment variable settings needed to decrypt the passwords. > Specifically, it will need to modify: > * For MR2 jobs: > ** yarn.app.mapreduce.am.admin.user.env > ** mapreduce.admin.user.env > * For Spark jobs: > ** spark.yarn.appMasterEnv.HADOOP_CREDSTORE_PASSWORD > ** spark.executorEnv.HADOOP_CREDSTORE_PASSWORD > HiveServer2 can get the decryption password from its own environment, the > same way it does for its own credential provider store today. > Additionally, it can be desirable for HiveServer2 to have a separate > encrypted password file than what is used by the job. HiveServer2 may have > secrets that the job should not have, such as the metastore database password > or the password to decrypt its private SSL certificate. It is also best > practices to have separate passwords on separate files. To facilitate this, > Hive will also accept: > * A configuration for a path to a credential store to use for jobs. This > should already be uploaded in HDFS. (hive.server2.job.keystore.location or a > better name) If this is not specified, then HS2 will simply use the value of > hadoop.security.credential.provider.path. > * An environment variable for the password to decrypt the credential store > (HIVE_JOB_KEYSTORE_PASSWORD or better). If this is not specified, then HS2 > will simply use the standard environment variable for decrypting the Hadoop > Credential Provider. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11812) datediff sometimes returns incorrect results when called with dates
[
https://issues.apache.org/jira/browse/HIVE-11812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530156#comment-15530156
]
Hive QA commented on HIVE-11812:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12830654/HIVE-11812.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10645 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[date_udf]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamp_udf]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vectorized_date_funcs]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[date_udf]
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[timestamp_udf]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1331/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1331/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1331/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12830654 - PreCommit-HIVE-Build
> datediff sometimes returns incorrect results when called with dates
> ---
>
> Key: HIVE-11812
> URL: https://issues.apache.org/jira/browse/HIVE-11812
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 2.0.0
>Reporter: Nicholas Brenwald
>Assignee: Chetna Chaudhari
>Priority: Minor
> Attachments: HIVE-11812.1.patch
>
>
> DATEDIFF returns an incorrect result when one of the arguments is a date
> type.
> The Hive Language Manual provides the following signature for datediff:
> {code}
> int datediff(string enddate, string startdate)
> {code}
> I think datediff should either throw an error (if date types are not
> supported), or return the correct result.
> To reproduce, create a table:
> {code}
> create table t (c1 string, c2 date);
> {code}
> Assuming you have a table x containing some data, populate table t with 1 row:
> {code}
> insert into t select '2015-09-15', '2015-09-15' from x limit 1;
> {code}
> Then run the following 12 test queries:
> {code}
> select datediff(c1, '2015-09-14') from t;
> select datediff(c1, '2015-09-15') from t;
> select datediff(c1, '2015-09-16') from t;
> select datediff('2015-09-14', c1) from t;
> select datediff('2015-09-15', c1) from t;
> select datediff('2015-09-16', c1) from t;
> select datediff(c2, '2015-09-14') from t;
> select datediff(c2, '2015-09-15') from t;
> select datediff(c2, '2015-09-16') from t;
> select datediff('2015-09-14', c2) from t;
> select datediff('2015-09-15', c2) from t;
> select datediff('2015-09-16', c2) from t;
> {code}
> The below table summarises the result. All results for column c1 (which is a
> string) are correct, but when using c2 (which is a date), two of the results
> are incorrect.
> || Test || Expected Result || Actual Result || Passed / Failed ||
> |datediff(c1, '2015-09-14')| 1 | 1| Passed |
> |datediff(c1, '2015-09-15')| 0 | 0| Passed |
> |datediff(c1, '2015-09-16') | -1 | -1| Passed |
> |datediff('2015-09-14', c1) | -1 | -1| Passed |
> |datediff('2015-09-15', c1)| 0 | 0| Passed |
> |datediff('2015-09-16', c1)| 1 | 1| Passed |
> |datediff(c2, '2015-09-14')| 1 | 0| {color:red}Failed{color} |
> |datediff(c2, '2015-09-15')| 0 | 0| Passed |
> |datediff(c2, '2015-09-16') | -1 | -1| Passed |
> |datediff('2015-09-14', c2) | -1 | 0 | {color:red}Failed{color} |
> |datediff('2015-09-15', c2)| 0 | 0| Passed |
> |datediff('2015-09-16', c2)| 1 | 1| Passed |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14847) HiveServer2: Implement some admission control mechanism for graceful degradation when resources are exhausted
[
https://issues.apache.org/jira/browse/HIVE-14847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530125#comment-15530125
]
Xuefu Zhang commented on HIVE-14847:
With HIVE-9423, I'm not sure what else we need to do here.
> HiveServer2: Implement some admission control mechanism for graceful
> degradation when resources are exhausted
> -
>
> Key: HIVE-14847
> URL: https://issues.apache.org/jira/browse/HIVE-14847
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14721) Fix TestJdbcWithMiniHS2 runtime
[ https://issues.apache.org/jira/browse/HIVE-14721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-14721: Attachment: HIVE-14721.1.patch > Fix TestJdbcWithMiniHS2 runtime > --- > > Key: HIVE-14721 > URL: https://issues.apache.org/jira/browse/HIVE-14721 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-14721.1.patch > > > Currently 450s -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14721) Fix TestJdbcWithMiniHS2 runtime
[ https://issues.apache.org/jira/browse/HIVE-14721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530110#comment-15530110 ] Vaibhav Gumashta commented on HIVE-14721: - [~sseth] This is good for review. Test time down to ~215s. > Fix TestJdbcWithMiniHS2 runtime > --- > > Key: HIVE-14721 > URL: https://issues.apache.org/jira/browse/HIVE-14721 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-14721.1.patch > > > Currently 450s -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12222) Define port range in property for RPCServer
[
https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-1:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks Xuefu for reviewing.
> Define port range in property for RPCServer
> ---
>
> Key: HIVE-1
> URL: https://issues.apache.org/jira/browse/HIVE-1
> Project: Hive
> Issue Type: Improvement
> Components: CLI, Spark
>Affects Versions: 1.2.1
> Environment: Apache Hadoop 2.7.0
> Apache Hive 1.2.1
> Apache Spark 1.5.1
>Reporter: Andrew Lee
>Assignee: Aihua Xu
> Fix For: 2.2.0
>
> Attachments: HIVE-1.1.patch, HIVE-1.2.patch,
> HIVE-1.3.patch
>
>
> Creating this JIRA after discussin with Xuefu on the dev mailing list. Would
> need some help to review and update the fields in this JIRA ticket, thanks.
> I notice that in
> ./spark-client/src/main/java/org/apache/hive/spark/client/rpc/RpcServer.java
> The port number is assigned with 0 which means it will be a random port every
> time when the RPC Server is created to talk to Spark in the same session.
> Because of this, this is causing problems to configure firewall between the
> HiveCLI RPC Server and Spark due to unpredictable port numbers here. In other
> word, users need to open all hive ports range
> from Data Node => HiveCLI (edge node).
> {code}
> this.channel = new ServerBootstrap()
> .group(group)
> .channel(NioServerSocketChannel.class)
> .childHandler(new ChannelInitializer() {
> @Override
> public void initChannel(SocketChannel ch) throws Exception {
> SaslServerHandler saslHandler = new SaslServerHandler(config);
> final Rpc newRpc = Rpc.createServer(saslHandler, config, ch,
> group);
> saslHandler.rpc = newRpc;
> Runnable cancelTask = new Runnable() {
> @Override
> public void run() {
> LOG.warn("Timed out waiting for hello from client.");
> newRpc.close();
> }
> };
> saslHandler.cancelTask = group.schedule(cancelTask,
> RpcServer.this.config.getServerConnectTimeoutMs(),
> TimeUnit.MILLISECONDS);
> }
> })
> {code}
> 2 Main reasons.
> - Most users (what I see and encounter) use HiveCLI as a command line tool,
> and in order to use that, they need to login to the edge node (via SSH). Now,
> here comes the interesting part.
> Could be true or not, but this is what I observe and encounter from time to
> time. Most users will abuse the resource on that edge node (increasing
> HADOOP_HEAPSIZE, dumping output to local disk, running huge python workflow,
> etc), this may cause the HS2 process to run into OOME, choke and die, etc.
> various resource issues including others like login, etc.
> - Analyst connects to Hive via HS2 + ODBC. So HS2 needs to be highly
> available. This makes sense to run it on the gateway node or a service node
> and separated from the HiveCLI.
> The logs are located in different location, monitoring and auditing is easier
> to run HS2 with a daemon user account, etc. so we don't want users to run
> HiveCLI where HS2 is running.
> It's better to isolate the resource this way to avoid any memory, file
> handlers, disk space, issues.
> From a security standpoint,
> - Since users can login to edge node (via SSH), the security on the edge node
> needs to be fortified and enhanced. Therefore, all the FW comes in and
> auditing.
> - Regulation/compliance for auditing is another requirement to monitor all
> traffic, specifying ports and locking down the ports makes it easier since we
> can focus
> on a range to monitor and audit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13836) DbNotifications giving an error = Invalid state. Transaction has already started
[
https://issues.apache.org/jira/browse/HIVE-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barna Zsombor Klara reassigned HIVE-13836:
--
Assignee: Nachiket Vaidya (was: Barna Zsombor Klara)
> DbNotifications giving an error = Invalid state. Transaction has already
> started
>
>
> Key: HIVE-13836
> URL: https://issues.apache.org/jira/browse/HIVE-13836
> Project: Hive
> Issue Type: Bug
>Reporter: Nachiket Vaidya
>Assignee: Nachiket Vaidya
>Priority: Critical
> Labels: patch-available
> Fix For: 2.2.0
>
> Attachments: HIVE-13836.2-branch-2.1.patch, HIVE-13836.2.patch,
> HIVE-13836.patch
>
>
> I used pyhs2 python client to create tables/partitions in hive. I was working
> fine until I moved to multithreaded scripts which created 8 connections and
> ran DDL queries concurrently.
> I got the error as
> {noformat}
> 2016-05-04 17:49:26,226 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-4-thread-194]:
> HMSHandler Fatal error: Invalid state. Transaction has already started
> org.datanucleus.transaction.NucleusTransactionException: Invalid state.
> Transaction has already started
> at
> org.datanucleus.transaction.TransactionManager.begin(TransactionManager.java:47)
> at org.datanucleus.TransactionImpl.begin(TransactionImpl.java:131)
> at
> org.datanucleus.api.jdo.JDOTransaction.internalBegin(JDOTransaction.java:88)
> at
> org.datanucleus.api.jdo.JDOTransaction.begin(JDOTransaction.java:80)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.openTransaction(ObjectStore.java:463)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:7522)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy10.addNotificationEvent(Unknown Source)
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener.enqueue(DbNotificationListener.java:261)
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:123)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1483)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1502)
> at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at
> com.sun.proxy.$Proxy14.create_table_with_environment_context(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:9267)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13836) DbNotifications giving an error = Invalid state. Transaction has already started
[
https://issues.apache.org/jira/browse/HIVE-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barna Zsombor Klara updated HIVE-13836:
---
Attachment: HIVE-13836.2-branch-2.1.patch
Added patch for branch-2.1.
> DbNotifications giving an error = Invalid state. Transaction has already
> started
>
>
> Key: HIVE-13836
> URL: https://issues.apache.org/jira/browse/HIVE-13836
> Project: Hive
> Issue Type: Bug
>Reporter: Nachiket Vaidya
>Assignee: Barna Zsombor Klara
>Priority: Critical
> Labels: patch-available
> Fix For: 2.2.0
>
> Attachments: HIVE-13836.2-branch-2.1.patch, HIVE-13836.2.patch,
> HIVE-13836.patch
>
>
> I used pyhs2 python client to create tables/partitions in hive. I was working
> fine until I moved to multithreaded scripts which created 8 connections and
> ran DDL queries concurrently.
> I got the error as
> {noformat}
> 2016-05-04 17:49:26,226 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-4-thread-194]:
> HMSHandler Fatal error: Invalid state. Transaction has already started
> org.datanucleus.transaction.NucleusTransactionException: Invalid state.
> Transaction has already started
> at
> org.datanucleus.transaction.TransactionManager.begin(TransactionManager.java:47)
> at org.datanucleus.TransactionImpl.begin(TransactionImpl.java:131)
> at
> org.datanucleus.api.jdo.JDOTransaction.internalBegin(JDOTransaction.java:88)
> at
> org.datanucleus.api.jdo.JDOTransaction.begin(JDOTransaction.java:80)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.openTransaction(ObjectStore.java:463)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:7522)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy10.addNotificationEvent(Unknown Source)
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener.enqueue(DbNotificationListener.java:261)
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:123)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1483)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1502)
> at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at
> com.sun.proxy.$Proxy14.create_table_with_environment_context(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:9267)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13836) DbNotifications giving an error = Invalid state. Transaction has already started
[
https://issues.apache.org/jira/browse/HIVE-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barna Zsombor Klara reassigned HIVE-13836:
--
Assignee: Barna Zsombor Klara (was: Nachiket Vaidya)
> DbNotifications giving an error = Invalid state. Transaction has already
> started
>
>
> Key: HIVE-13836
> URL: https://issues.apache.org/jira/browse/HIVE-13836
> Project: Hive
> Issue Type: Bug
>Reporter: Nachiket Vaidya
>Assignee: Barna Zsombor Klara
>Priority: Critical
> Labels: patch-available
> Fix For: 2.2.0
>
> Attachments: HIVE-13836.2.patch, HIVE-13836.patch
>
>
> I used pyhs2 python client to create tables/partitions in hive. I was working
> fine until I moved to multithreaded scripts which created 8 connections and
> ran DDL queries concurrently.
> I got the error as
> {noformat}
> 2016-05-04 17:49:26,226 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-4-thread-194]:
> HMSHandler Fatal error: Invalid state. Transaction has already started
> org.datanucleus.transaction.NucleusTransactionException: Invalid state.
> Transaction has already started
> at
> org.datanucleus.transaction.TransactionManager.begin(TransactionManager.java:47)
> at org.datanucleus.TransactionImpl.begin(TransactionImpl.java:131)
> at
> org.datanucleus.api.jdo.JDOTransaction.internalBegin(JDOTransaction.java:88)
> at
> org.datanucleus.api.jdo.JDOTransaction.begin(JDOTransaction.java:80)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.openTransaction(ObjectStore.java:463)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:7522)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy10.addNotificationEvent(Unknown Source)
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener.enqueue(DbNotificationListener.java:261)
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:123)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1483)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1502)
> at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at
> com.sun.proxy.$Proxy14.create_table_with_environment_context(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:9267)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14830) Move a majority of the MiniLlapCliDriver tests to use an inline AM
[
https://issues.apache.org/jira/browse/HIVE-14830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529932#comment-15529932
]
Hive QA commented on HIVE-14830:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12830632/HIVE-14830.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10646 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cte_2]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testThreadPoolMetrics
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1330/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1330/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1330/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12830632 - PreCommit-HIVE-Build
> Move a majority of the MiniLlapCliDriver tests to use an inline AM
> --
>
> Key: HIVE-14830
> URL: https://issues.apache.org/jira/browse/HIVE-14830
> Project: Hive
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14830.01.patch, HIVE-14830.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14558) Add support for listing views similar to "show tables"
[ https://issues.apache.org/jira/browse/HIVE-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529929#comment-15529929 ] Naveen Gangam commented on HIVE-14558: -- I have looked at the test failures. They do not appear to be related to my change. The same tests fail with the same errors when I run them locally too. The same set of tests are failing in all prior builds too. So +1 for me. > Add support for listing views similar to "show tables" > -- > > Key: HIVE-14558 > URL: https://issues.apache.org/jira/browse/HIVE-14558 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14558.1.patch, HIVE-14558.2.patch, > HIVE-14558.4.patch, HIVE-14558.4.patch > > > Users have been asking for such feature where they can get a lists of views > separately. > So perhaps a syntax similar to "show tables" command? > show views [in/from ] [] > Does it make sense to add such command? or is it not worth the effort? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
