[jira] [Created] (HIVE-19125) TestStreaming fails with java.lang.NoClassDefFoundError

[Pravin Dsilva (JIRA)]

Pravin Dsilva created HIVE-19125:


 Summary: TestStreaming fails with java.lang.NoClassDefFoundError
 Key: HIVE-19125
 URL: https://issues.apache.org/jira/browse/HIVE-19125
 Project: Hive
  Issue Type: Bug
  Components: HCatalog, Test
Affects Versions: 3.0.0
 Environment: $ uname -a
Linux 90ab83b583eb 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:42:36 
UTC 2015 ppc64le ppc64le ppc64le GNU/Linux

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
Reporter: Pravin Dsilva


The failures appear only when running the entire test suite but pass when the 
tests are run individually.

command used: mvn test -fn -pl ql,hcatalog,hcatalog/streaming 
-Dtest=TestStreaming

 
{code:java}
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[ERROR]   
TestStreaming.setup:235->createDbAndTable:2126->addPartition:2136->getPartitionPath:2140->queryTable:2221
 ? NoClassDefFound
[INFO] 
[ERROR] Tests run: 26, Failures: 0, Errors: 26, Skipped: 0
[INFO] 
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Hive ... SUCCESS [  1.536 s]
[INFO] Hive Query Language  SUCCESS [ 24.866 s]
[INFO] Hive HCatalog .. SUCCESS [  1.414 s]
[INFO] Hive HCatalog Streaming  FAILURE [ 49.459 s]
[INFO] 

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/#review200626
---




ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 334 (patched)


That's a REPL event & the trouble with IOW is that it also destroys commits 
in progress with the new base_n files, where n > all previous open txns.


- Gopal V


On April 6, 2018, 1:54 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66485/
> ---
> 
> (Updated April 6, 2018, 1:54 a.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  5966740f88 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b1c2288d01 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
> 
> 
> Diff: https://reviews.apache.org/r/66485/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: reverting test-breaking changes

[Prasanth Jayachandran]

It may be because of legitimate memory issue/leak. Alternative is to decrease 
the batch size of negative cli driver on ptest cluster but again we wouldn't 
know if there is any actual memory issue.

Thanks
Prasanth



On Thu, Apr 5, 2018 at 1:36 PM -0700, "Vineet Garg" 
> wrote:


TestNegativeCli driver tests are still failing with java.lang.OutOfMemoryError: 
GC overhead limit exceeded error.
Can we increase the amount of memory for tests?

Vineet

On Mar 5, 2018, at 11:35 AM, Sergey Shelukhin > wrote:

On a semi-related note, I noticed recently that negative tests seem to OOM
in setup from time to time.
Can we increase the amount of memory for the tests a little bit, and/or
maybe add the dump on OOM to them, saved to test logs directory, so we
could investigate?

On 18/3/5, 11:07, "Vineet Garg" > wrote:

+1 for nightly build. We could generate reports to identify both frequent
and sporadic test failures plus other interesting bits like average build
time, yetus failures etc. It'll also help narrow down the culprit
commit(s) range to one day.
If you guys decide to go ahead with this I would like to help.

Vineet

On Mar 5, 2018, at 8:50 AM, Sahil Takiar > wrote:

Wow that HBase UI looks super useful. +1 to having something like that.

If not, +1 to having a proper nightly build, it would help devs identify
which commits break which tests. I find using git-bisect can take a long
time to run, and can be difficult to use (e.g. finding a known good
commit
isn't always easy).

On Mon, Mar 5, 2018 at 9:03 AM, Peter Vary > wrote:

Without a nightly build and with this many flaky tests it is very hard
to
identify the braking commits. We can use something like bisect and
multiple
test runs.

There is a more elegant way to do this with nightly test runs:
https://issues.apache.org/jira/browse/HBASE-15917 <
https://issues.apache.org/jira/browse/HBASE-15917>
https://builds.apache.org/job/HBASE-Find-Flaky-Tests/
lastSuccessfulBuild/artifact/dashboard.html

This also helps to identify the flaky tests, and creates a continuos,
updated list of them.

On Feb 23, 2018, at 6:55 PM, Sahil Takiar
wrote:

+1

Does anyone have suggestions about how to efficiently identify which
commit
is breaking a test? Is it just git-bisect or is there an easier way?
Hive
QA isn't always that helpful, it will say a test is failing for the
past
"x" builds, but that doesn't help much since Hive QA isn't a nightly
build.

On Thu, Feb 22, 2018 at 10:31 AM, Vihang Karajgaonkar <
vih...@cloudera.com>
wrote:

+1
Commenting on JIRA and giving a 24hr heads-up (excluding weekends)
would be
good.

On Thu, Feb 22, 2018 at 10:19 AM, Alan Gates
wrote:

+1.

Alan.

On Thu, Feb 22, 2018 at 8:25 AM, Thejas Nair
wrote:

+1
I agree, this makes sense. The number of failures keeps increasing.
A 24 hour heads up in either case before revert would be good.


On Thu, Feb 22, 2018 at 2:45 AM, Peter Vary
wrote:

I agree with Zoltan. The continuously braking tests make it very
hard
to
spot real issues.
Any thoughts on doing it automatically?

On Feb 22, 2018, at 10:47 AM, Zoltan Haindrich
wrote:

*

Hello,

*
*

**

In the last couple weeks the number of broken tests have started
to
go
up...and even tho I run bisect/etc from time to time ; sometimes
people
don't react to my comments/tickets/etc.

Because keeping this many failing tests makes it easier for a new
one
to
slip in...I think reverting the patch introducing the test
failures
would
also help in some case.

I think it would help a lot to prevent further test breaks to
revert
the
patch if any of the following conditions is met:

*
*

C1) if the notification/comment about the fact that the patch
indeed
broken a test somehow have been unanswered for at least 24 hours.

C2) if the patch is in for 7 days; but the test failure is still
not
addressed (note that in this case there might be a conversation
about
fixing it...but in this case ; to enable other people to work in a
cleaner
environment is more important than a single patch - and if it
can't
be
fixed in 7 days...well it might not get fixed in a month).

*
*

I would like to also note that I've seen a few tickets which have
been
picked up by people who were not involved in creating the original
change -
and although the intention was good, they might miss the context
of
the
original patch and may "fix" the tests in the wrong way: accept a
q.out
which is inappropriate or ignore the test...

*
*

would it be ok to implement this from now on? because it makes my
efforts practically useless if people are not reacting...

*
*

note: just to be on the same page - this is only about running a
single
test which falls on its own - I feel that flaky tests are an
entirely
different topic.

*
*

cheers,

Zoltan

**
*








--
Sahil Takiar
Software Engineer
takiar.sa...@gmail.com | (510) 673-0309




--
Sahil Takiar
Software Engineer
takiar.sa...@gmail.com | (510) 673-0309

Re: Need edit permission to Hive Confluence page

[Bharathkrishna Guruvayoor Murali]

Hi Lefty,

Thanks for pointing me to the wiki.
My username is: bharos92


On Thu, Apr 5, 2018 at 8:55 PM, Lefty Leverenz 
wrote:

> Bharath, first you need to get a Confluence username as described here:
>
> About This Wiki -- How to get permission to edit
> 
>
>
> (It's okay to use the dev@hive list instead of user@hive, just tell us
> your username in this thread.)
>
> -- Lefty
>
>
> On Thu, Apr 5, 2018 at 2:57 PM Bharathkrishna Guruvayoor Murali <
> bhar...@cloudera.com> wrote:
>
>> Hi,
>>
>> I need edit permission to Confluence page.
>> Currently I need to make a doc update related to HIVE-16944.
>>
>> I would like to update SchemaTool wiki page
>>  to
>> include the parameters driver and URL which are currently missing in the
>> documentation.
>>
>> Someone who has the rights please add me.
>>
>> Thanks,
>> Bharath
>>
>

Re: Need edit permission to Hive Confluence page

[Lefty Leverenz]

Bharath, first you need to get a Confluence username as described here:

About This Wiki -- How to get permission to edit



(It's okay to use the dev@hive list instead of user@hive, just tell us your
username in this thread.)

-- Lefty


On Thu, Apr 5, 2018 at 2:57 PM Bharathkrishna Guruvayoor Murali <
bhar...@cloudera.com> wrote:

> Hi,
>
> I need edit permission to Confluence page.
> Currently I need to make a doc update related to HIVE-16944.
>
> I would like to update SchemaTool wiki page
>  to
> include the parameters driver and URL which are currently missing in the
> documentation.
>
> Someone who has the rights please add me.
>
> Thanks,
> Bharath
>

Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/
---

Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 5966740f88 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
b1c2288d01 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 


Diff: https://reviews.apache.org/r/66485/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-19124) implement a basic major compactor for MM tables

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-19124:
---

 Summary: implement a basic major compactor for MM tables
 Key: HIVE-19124
 URL: https://issues.apache.org/jira/browse/HIVE-19124
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


For now, it will run a query directly and only major compactions will be 
supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66381: HIVE-18905: HS2: SASL auth loads HiveConf for every JDBC call

[Igor Kryvenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66381/
---

(Updated Апрель 5, 2018, 9:32 п.п.)


Review request for hive, Ashutosh Chauhan and Gopal V.


Changes
---

Fixing checkstyle


Bugs: HIVE-18905
https://issues.apache.org/jira/browse/HIVE-18905


Repository: hive-git


Description
---

SASL authentication filter does a new HiveConf() for no good reason.


Diffs (updated)
-

  
service/src/java/org/apache/hive/service/auth/AuthenticationProviderFactory.java
 c684318986b028a92c255a087d8375b08557faf1 
  
service/src/java/org/apache/hive/service/auth/CustomAuthenticationProviderImpl.java
 3d7ccd96feaacfc36d262d3396e97a7982921fb1 
  
service/src/java/org/apache/hive/service/auth/LdapAuthenticationProviderImpl.java
 73bbb6bdf8af451d78ac72a58da91f2285d81a79 
  
service/src/java/org/apache/hive/service/auth/PamAuthenticationProviderImpl.java
 fff378a7e9e7183d6e27ca229535168eda349d3f 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java 
70ffa3c6a3ad5d447f1d2dc2ebb9be20e22be10d 


Diff: https://reviews.apache.org/r/66381/diff/2/

Changes: https://reviews.apache.org/r/66381/diff/1-2/


Testing
---


Thanks,

Igor Kryvenko

Re: Reload of aux jars on HiveServer2 failing

[Adam Szita]

Hi Shashank,

First of all make sure your jar is not on a path configured for the
non-reloadable jars as well i.e. hive.aux.jars.path; it should only be on
the reloadable one.

As far as I know RELOAD statement creates new classloader over the old one
to make sure new jar content is loaded, however, you could try the
following:
After removing the old jar from your path execute a RELOAD. Then put your
new jar on the path and hit RELOAD again, and run your query.

Hope that helps.

Adam

On 8 March 2018 at 00:37, Shashank Pedamallu 
wrote:

> Hi,
>
> I have a jar that is configured under hive.reloadable.aux.jars.path. Upon
> updating the jar and successful execution of *RELOAD;* command via Beeline,
> the latest jar is not getting picked up. These are the steps I followed:
> 0. Started hs2 process with --hiveconf hive.reloadable.aux.jars.path=
> 
> 1. Removed the *desired_jar*
> 2. Added an updated jar (that has some log statements) with same name in
> the same location as *desired_jar_absolute_path*
> 3. Connected to HS2 process via Beeline and issued a RELOAD; command
> 4. Ran a query that should have emitted a log statement as per updated jar.
> But it was not.
>
> Please note that if I did a restart of hs2 process, the latest jar is being
> picked and the log statements are getting printed.
>
> Could someone please help me if I'm not using the RELOAD command right or
> if there is a known issue surrounding it.
>
> Thanks in advance!
> -Shashank
>

Re: Review Request 66370: HIVE-18725: Improve error handling for subqueries if there is wrong column reference

[Igor Kryvenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66370/
---

(Updated Апрель 5, 2018, 8:59 п.п.)


Review request for hive, Ashutosh Chauhan and Vineet Garg.


Changes
---

Deleted redundant converting from SemanticException to CalciteSemanticException


Bugs: HIVE-18725
https://issues.apache.org/jira/browse/HIVE-18725


Repository: hive-git


Description
---

If there is a column reference within subquery which doesn't exist Hive throws 
misleading error message.


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSubquerySemanticException.java
 4321a5c7894cec0713ef6e89092ec923f8273b4c 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteViewSemanticException.java
 c2a4e94a032b46134932d4bec5f70b197788654b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
41de17fd4679009ef6a4fb5a6d976cbc794ce791 
  ql/src/test/queries/clientnegative/subquery_non_exisiting_column.q 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_non_exisiting_column.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/66370/diff/3/

Changes: https://reviews.apache.org/r/66370/diff/2-3/


Testing
---


Thanks,

Igor Kryvenko

Re: Review Request 66370: HIVE-18725: Improve error handling for subqueries if there is wrong column reference

[Igor Kryvenko]

> On Апрель 3, 2018, 9:58 п.п., Vineet Garg wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
> > Line 1314 (original), 1309 (patched)
> > 
> >
> > I didn't understand this. If it is CalciteSemanticException we convert 
> > it to CalciteSemancticException but if it is SemanticException we convert 
> > it to CalciteSemanticException? Why do we do that?

If it is CalciteSemanticException we convert it to SemanticException for saving 
original cause(CalciteSubquerySemanticException or 
CalciteViewSemanticException). 
If it is Semantic exceptin we just wrapped original message into 
CalciteSemanticException. If 
we will do 
throw new CalciteSemanticException(first) we will have redundant exception 
message such as 
SemanticException : CalciteSemanticException : CalciteSubquerySemanticException.


- Igor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66370/#review200401
---


On Апрель 1, 2018, 8:36 п.п., Igor Kryvenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66370/
> ---
> 
> (Updated Апрель 1, 2018, 8:36 п.п.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Vineet Garg.
> 
> 
> Bugs: HIVE-18725
> https://issues.apache.org/jira/browse/HIVE-18725
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> If there is a column reference within subquery which doesn't exist Hive 
> throws misleading error message.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSubquerySemanticException.java
>  4321a5c7894cec0713ef6e89092ec923f8273b4c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteViewSemanticException.java
>  c2a4e94a032b46134932d4bec5f70b197788654b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 41de17fd4679009ef6a4fb5a6d976cbc794ce791 
>   ql/src/test/queries/clientnegative/subquery_non_exisiting_column.q 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/subquery_non_exisiting_column.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66370/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Igor Kryvenko
> 
>

Re: Review Request 66416: HIVE-17647 DDLTask.generateAddMmTasks(Table tbl) and other random code should not start transactions

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66416/
---

(Updated April 5, 2018, 8:51 p.m.)


Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties d2e077b509 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 79db006c74 
  ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java f53afaff2b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java fb1efe01dc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ImportCommitTask.java b3c62ad1a8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ImportCommitWork.java a119250464 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 10a2ed2663 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java eed37a1937 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
3e8e1b3981 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
9e66422904 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
8b639f7922 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java 
79b2e48ee2 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableDesc.java d7b224772d 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DDLDesc.java 65f4cf233b 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
8406caa761 
  ql/src/test/results/clientpositive/mm_conversions.q.out 4754710291 


Diff: https://reviews.apache.org/r/66416/diff/2/

Changes: https://reviews.apache.org/r/66416/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-19123) TestNegativeCliDriver nopart_insert failing

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-19123:
--

 Summary: TestNegativeCliDriver nopart_insert failing
 Key: HIVE-19123
 URL: https://issues.apache.org/jira/browse/HIVE-19123
 Project: Hive
  Issue Type: Test
Reporter: Vineet Garg
Assignee: Vineet Garg
 Fix For: 3.0.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66370: HIVE-18725: Improve error handling for subqueries if there is wrong column reference

[Vineet Garg]

> On April 5, 2018, 8:34 p.m., Igor Kryvenko wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
> > Line 1314 (original), 1309 (patched)
> > 
> >
> > If it is CalciteSemanticException we convert it to SemanticException 
> > for saving original cause(CalciteSubquerySemanticException or 
> > CalciteViewSemanticException). 
> > If it is Semantic exceptin we just wrapped original message into 
> > CalciteSemanticException. If 
> > we will do 
> > throw new CalciteSemanticException(first) we will have redundant 
> > exception message such as 
> > SemanticException : CalciteSemanticException : 
> > CalciteSubquerySemanticException.

Why create a new exception object if it is SemanticException. Previously if 
first is instance of SemanticException it was rethrown i.e. throw 
(SemanticException) first. But now we create a new CalciteSemanticException 
object. I think we should keep this case like it was i.e. just throw 
SemanticException as it is.


- Vineet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66370/#review200565
---


On April 1, 2018, 8:36 p.m., Igor Kryvenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66370/
> ---
> 
> (Updated April 1, 2018, 8:36 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Vineet Garg.
> 
> 
> Bugs: HIVE-18725
> https://issues.apache.org/jira/browse/HIVE-18725
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> If there is a column reference within subquery which doesn't exist Hive 
> throws misleading error message.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSubquerySemanticException.java
>  4321a5c7894cec0713ef6e89092ec923f8273b4c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteViewSemanticException.java
>  c2a4e94a032b46134932d4bec5f70b197788654b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 41de17fd4679009ef6a4fb5a6d976cbc794ce791 
>   ql/src/test/queries/clientnegative/subquery_non_exisiting_column.q 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/subquery_non_exisiting_column.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66370/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Igor Kryvenko
> 
>

Re: reverting test-breaking changes

[Vineet Garg]

TestNegativeCli driver tests are still failing with java.lang.OutOfMemoryError: 
GC overhead limit exceeded error.
Can we increase the amount of memory for tests?

Vineet

On Mar 5, 2018, at 11:35 AM, Sergey Shelukhin 
> wrote:

On a semi-related note, I noticed recently that negative tests seem to OOM
in setup from time to time.
Can we increase the amount of memory for the tests a little bit, and/or
maybe add the dump on OOM to them, saved to test logs directory, so we
could investigate?

On 18/3/5, 11:07, "Vineet Garg" 
> wrote:

+1 for nightly build. We could generate reports to identify both frequent
and sporadic test failures plus other interesting bits like average build
time, yetus failures etc. It’ll also help narrow down the culprit
commit(s) range to one day.
If you guys decide to go ahead with this I would like to help.

Vineet

On Mar 5, 2018, at 8:50 AM, Sahil Takiar 
> wrote:

Wow that HBase UI looks super useful. +1 to having something like that.

If not, +1 to having a proper nightly build, it would help devs identify
which commits break which tests. I find using git-bisect can take a long
time to run, and can be difficult to use (e.g. finding a known good
commit
isn't always easy).

On Mon, Mar 5, 2018 at 9:03 AM, Peter Vary 
> wrote:

Without a nightly build and with this many flaky tests it is very hard
to
identify the braking commits. We can use something like bisect and
multiple
test runs.

There is a more elegant way to do this with nightly test runs:
https://issues.apache.org/jira/browse/HBASE-15917 <
https://issues.apache.org/jira/browse/HBASE-15917>
https://builds.apache.org/job/HBASE-Find-Flaky-Tests/
lastSuccessfulBuild/artifact/dashboard.html 

This also helps to identify the flaky tests, and creates a continuos,
updated list of them.

On Feb 23, 2018, at 6:55 PM, Sahil Takiar 
wrote:

+1

Does anyone have suggestions about how to efficiently identify which
commit
is breaking a test? Is it just git-bisect or is there an easier way?
Hive
QA isn't always that helpful, it will say a test is failing for the
past
"x" builds, but that doesn't help much since Hive QA isn't a nightly
build.

On Thu, Feb 22, 2018 at 10:31 AM, Vihang Karajgaonkar <
vih...@cloudera.com>
wrote:

+1
Commenting on JIRA and giving a 24hr heads-up (excluding weekends)
would be
good.

On Thu, Feb 22, 2018 at 10:19 AM, Alan Gates 
wrote:

+1.

Alan.

On Thu, Feb 22, 2018 at 8:25 AM, Thejas Nair 
wrote:

+1
I agree, this makes sense. The number of failures keeps increasing.
A 24 hour heads up in either case before revert would be good.


On Thu, Feb 22, 2018 at 2:45 AM, Peter Vary 
wrote:

I agree with Zoltan. The continuously braking tests make it very
hard
to
spot real issues.
Any thoughts on doing it automatically?

On Feb 22, 2018, at 10:47 AM, Zoltan Haindrich 
wrote:

*

Hello,

*
*

**

In the last couple weeks the number of broken tests have started
to
go
up...and even tho I run bisect/etc from time to time ; sometimes
people
don’t react to my comments/tickets/etc.

Because keeping this many failing tests makes it easier for a new
one
to
slip in...I think reverting the patch introducing the test
failures
would
also help in some case.

I think it would help a lot to prevent further test breaks to
revert
the
patch if any of the following conditions is met:

*
*

C1) if the notification/comment about the fact that the patch
indeed
broken a test somehow have been unanswered for at least 24 hours.

C2) if the patch is in for 7 days; but the test failure is still
not
addressed (note that in this case there might be a conversation
about
fixing it...but in this case ; to enable other people to work in a
cleaner
environment is more important than a single patch - and if it
can't
be
fixed in 7 days...well it might not get fixed in a month).

*
*

I would like to also note that I've seen a few tickets which have
been
picked up by people who were not involved in creating the original
change -
and although the intention was good, they might miss the context
of
the
original patch and may "fix" the tests in the wrong way: accept a
q.out
which is inappropriate or ignore the test...

*
*

would it be ok to implement this from now on? because it makes my
efforts practically useless if people are not reacting…

*
*

note: just to be on the same page - this is only about running a
single
test which falls on its own - I feel that flaky tests are an
entirely
different topic.

*
*

cheers,

Zoltan

**
*








--
Sahil Takiar
Software Engineer
takiar.sa...@gmail.com | (510) 673-0309




--
Sahil Takiar
Software Engineer

Re: Review Request 66370: HIVE-18725: Improve error handling for subqueries if there is wrong column reference

[Igor Kryvenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66370/#review200565
---




ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Line 1314 (original), 1309 (patched)


If it is CalciteSemanticException we convert it to SemanticException for 
saving original cause(CalciteSubquerySemanticException or 
CalciteViewSemanticException). 
If it is Semantic exceptin we just wrapped original message into 
CalciteSemanticException. If 
we will do 
throw new CalciteSemanticException(first) we will have redundant exception 
message such as 
SemanticException : CalciteSemanticException : 
CalciteSubquerySemanticException.


- Igor Kryvenko


On Апрель 1, 2018, 8:36 п.п., Igor Kryvenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66370/
> ---
> 
> (Updated Апрель 1, 2018, 8:36 п.п.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Vineet Garg.
> 
> 
> Bugs: HIVE-18725
> https://issues.apache.org/jira/browse/HIVE-18725
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> If there is a column reference within subquery which doesn't exist Hive 
> throws misleading error message.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSubquerySemanticException.java
>  4321a5c7894cec0713ef6e89092ec923f8273b4c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteViewSemanticException.java
>  c2a4e94a032b46134932d4bec5f70b197788654b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 41de17fd4679009ef6a4fb5a6d976cbc794ce791 
>   ql/src/test/queries/clientnegative/subquery_non_exisiting_column.q 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/subquery_non_exisiting_column.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66370/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Igor Kryvenko
> 
>

Re: Review Request 66414: HIVE-17970 MM LOAD DATA with OVERWRITE doesn't use base_n directory concept

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66414/
---

(Updated April 5, 2018, 7:24 p.m.)


Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 75c07b41b2 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
 0168472bdc 
  itests/src/test/resources/testconfiguration.properties d2e077b509 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7eba5e88d8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5fbe045df5 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java eed37a1937 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
e49089b91e 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java b0dfc48165 
  ql/src/test/results/clientpositive/mm_loaddata.q.out  


Diff: https://reviews.apache.org/r/66414/diff/2/

Changes: https://reviews.apache.org/r/66414/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

Need edit permission to Hive Confluence page

[Bharathkrishna Guruvayoor Murali]

Hi,

I need edit permission to Confluence page.
Currently I need to make a doc update related to HIVE-16944.

I would like to update SchemaTool wiki page
 to
include the parameters driver and URL which are currently missing in the
documentation.

Someone who has the rights please add me.

Thanks,
Bharath

Re: Apache Hive 3.0.0 release preparation

[Vineet Garg]

Hello,

I plan to cut off branch for Hive 3.0.0 on Monday (9 April) since bunch of 
folks have big patches pending.

Regards,
Vineet G

> On Apr 2, 2018, at 3:14 PM, Vineet Garg  wrote:
> 
> Hello,
> 
> We have enough votes to prepare a release candidate for Hive 3.0.0. I am 
> going to cutoff a branch in a day or two. I’ll send an email as soon as I 
> have the branch ready.
> Meanwhile there are approximately 69 JIRAs which are currently opened with 
> fix version 3.0.0. I’ll appreciate if their respective owners would update 
> the JIRA if it is a blocker. Otherwise I’ll update them to defer the fix 
> version to next release.
> 
> Regards,
> Vineet G
> 

[jira] [Created] (HIVE-19122) Transactional tables can't handle 'rename' table

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-19122:
-

 Summary: Transactional tables can't handle 'rename' table
 Key: HIVE-19122
 URL: https://issues.apache.org/jira/browse/HIVE-19122
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Reporter: Eugene Koifman


All the metastore metadata tables related to Acid use dbname/table name.  There 
is no logic to update these tables when table is renamed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19121) Fix HiveSchemaTool validation for databases that don't support schema

[Miklos Gergely (JIRA)]

Miklos Gergely created HIVE-19121:
-

 Summary: Fix HiveSchemaTool validation for databases that don't 
support schema
 Key: HIVE-19121
 URL: https://issues.apache.org/jira/browse/HIVE-19121
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Miklos Gergely
Assignee: Miklos Gergely
 Fix For: 3.0.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19120) catalog not properly set for some tables in SQL upgrade scripts

[Alan Gates (JIRA)]

Alan Gates created HIVE-19120:
-

 Summary: catalog not properly set for some tables in SQL upgrade 
scripts
 Key: HIVE-19120
 URL: https://issues.apache.org/jira/browse/HIVE-19120
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Affects Versions: 3.0.0
Reporter: Alan Gates
Assignee: Alan Gates
 Fix For: 3.0.0


A catalog column is added to the PARTITION_EVENTS and NOTIFICATION_LOG but the 
upgrade scripts do not include an UPDATE statement to set this to the default 
value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 66477: HIVE-19119: Fix the TestAppendPartitions tests which are failing in the pre-commit runs

[Marta Kuczora via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66477/
---

Review request for hive, Peter Vary and Adam Szita.


Bugs: HIVE-19119
https://issues.apache.org/jira/browse/HIVE-19119


Repository: hive-git


Description
---

The test got fixed in HIVE-19060, but the fix got overwritten by an other 
commit. Fixing the testAppendPartitionNullPartValues and 
testAppendPartitionEmptyPartValues test cases again.


Diffs
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 8539fea 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAppendPartitions.java
 75b26f2 


Diff: https://reviews.apache.org/r/66477/diff/1/


Testing
---


Thanks,

Marta Kuczora

[jira] [Created] (HIVE-19119) Fix the TestAppendPartitions tests which are failing in the pre-commit runs

[Marta Kuczora (JIRA)]

Marta Kuczora created HIVE-19119:


 Summary: Fix the TestAppendPartitions tests which are failing in 
the pre-commit runs
 Key: HIVE-19119
 URL: https://issues.apache.org/jira/browse/HIVE-19119
 Project: Hive
  Issue Type: Bug
  Components: Test
Reporter: Marta Kuczora
Assignee: Marta Kuczora






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results

[Matt McCline (JIRA)]

Matt McCline created HIVE-19118:
---

 Summary: Vectorization: Turning on vectorization in escape_crlf 
produces wrong results
 Key: HIVE-19118
 URL: https://issues.apache.org/jira/browse/HIVE-19118
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Matt McCline
Assignee: Matt McCline


Found in vectorization enable by default experiment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65716: HIVE-18696: The partition folders might not get cleaned up properly in the HiveMetaStore.add_partitions_core method if an exception occurs

[Marta Kuczora via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65716/
---

(Updated April 5, 2018, 3:40 p.m.)


Review request for hive, Alexander Kolbasov, Peter Vary, and Adam Szita.


Changes
---

Rebasing the patch


Bugs: HIVE-18696
https://issues.apache.org/jira/browse/HIVE-18696


Repository: hive-git


Description
---

The idea behind the patch is

1) Separate the partition validation from starting the tasks which create the 
partition folders. 
Instead of doing the checks on the partitions and submit the tasks in one loop, 
separated the validation into a different loop. So first iterate through the 
partitions, validate the table/db names, and check for duplicates. Then if all 
partitions were correct, in the second loop submit the tasks to create the 
partition folders. This way if one of the partitions is incorrect, the 
exception will be thrown in the first loop, before the tasks are submitted. So 
we can be sure that no partition folder will be created if the list contains an 
invalid partition.

2) Handle the exceptions which occur during the execution of the tasks 
differently.
Previously if an exception occured in one task, the remaining tasks were 
canceled, and the newly created partition folders were cleaned up in the 
finally part. The problem was that it could happen that some tasks were still 
not finished with the folder creation when cleaning up the others, so there 
could have been leftover folders. After doing some testing it turned out that 
this use case cannot be avoided completely when canceling the tasks.
The idea of this patch is to set a flag if an exception is thrown in one of the 
tasks. This flag is visible in the tasks and if its value is true, the 
partition folders won't be created. Then iterate through the remaining tasks 
and wait for them to finish. The tasks which are started before the flag got 
set will then finish creating the partition folders. The tasks which are 
started after the flag got set, won't create the partition folders, to avoid 
unnecessary work. This way it is sure that all tasks are finished, when 
entering the finally part where the partition folders are cleaned up.


Diffs (updated)
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 8539fea 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
 8555eee 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitionsFromPartSpec.java
 b32954f 


Diff: https://reviews.apache.org/r/65716/diff/6/

Changes: https://reviews.apache.org/r/65716/diff/5-6/


Testing
---

Added some new tests cases to the TestAddPartitions and 
TestAddPartitionsFromPartSpec tests.


Thanks,

Marta Kuczora

Re: Review Request 66359: HIVE-19076: Fix NPE and TApplicationException in function related HiveMetastore methods

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66359/#review200549
---


Ship it!




LGTM +1


standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Line 6912 (original), 6957 (patched)


Note for later: We should think about this startFunction/endFunction usage. 
In several cases we can have Exceptions which can cause uneven start/end calls. 
Also if the checks take more time, then we miss some runtime


- Peter Vary


On April 5, 2018, 1:43 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66359/
> ---
> 
> (Updated April 5, 2018, 1:43 p.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-19076
> https://issues.apache.org/jira/browse/HIVE-19076
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The TestFunctions tests revealed that NPE is thrown in some cases. These NPEs 
> could be prevented with a simple null check and a MetaException with a proper 
> error message should be thrown instead.
> 
> Also there are some alter function tests where InvalidObjectException is 
> thrown with Embedded MetaStore, but TApplicationException it thrown with 
> Remote MetaStore. The reason is that the InvalidObjectException is not 
> defined for the alter_function method in the thrift interface, so we got the 
> TApplicationException when the InvalidObjectException was thrown. In these 
> cases the InvalidObjectException could be handled on the server side and 
> re-throw it as a MetaException
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  8539fea 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  ebbf465 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestFunctions.java
>  9857c4e 
> 
> 
> Diff: https://reviews.apache.org/r/66359/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Re: Review Request 66359: HIVE-19076: Fix NPE and TApplicationException in function related HiveMetastore methods

[Marta Kuczora via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66359/
---

(Updated April 5, 2018, 1:43 p.m.)


Review request for hive, Peter Vary and Adam Szita.


Changes
---

Rebased the patch and fixed the issues which came up.


Bugs: HIVE-19076
https://issues.apache.org/jira/browse/HIVE-19076


Repository: hive-git


Description
---

The TestFunctions tests revealed that NPE is thrown in some cases. These NPEs 
could be prevented with a simple null check and a MetaException with a proper 
error message should be thrown instead.

Also there are some alter function tests where InvalidObjectException is thrown 
with Embedded MetaStore, but TApplicationException it thrown with Remote 
MetaStore. The reason is that the InvalidObjectException is not defined for the 
alter_function method in the thrift interface, so we got the 
TApplicationException when the InvalidObjectException was thrown. In these 
cases the InvalidObjectException could be handled on the server side and 
re-throw it as a MetaException


Diffs (updated)
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 8539fea 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 ebbf465 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestFunctions.java
 9857c4e 


Diff: https://reviews.apache.org/r/66359/diff/2/

Changes: https://reviews.apache.org/r/66359/diff/1-2/


Testing
---


Thanks,

Marta Kuczora

[jira] [Created] (HIVE-19117) hiveserver2 org.apache.thrift.transport.TTransportException error when running 2nd query after minute of inactivity

[t oo (JIRA)]

t oo created HIVE-19117:
---

 Summary: hiveserver2 
org.apache.thrift.transport.TTransportException error when running 2nd query 
after minute of inactivity
 Key: HIVE-19117
 URL: https://issues.apache.org/jira/browse/HIVE-19117
 Project: Hive
  Issue Type: Bug
  Components: Hive, HiveServer2, Metastore, Thrift API
Affects Versions: 2.1.1
 Environment: * Hive 2.1.1 with hive.server2.transport.mode set to 
binary (sample JDBC string is jdbc:hive2://remotehost:1/default)
 * Hadoop 2.8.3
 * Metastore using MySQL
 * Java 8
Reporter: t oo


I make a JDBC connection from my SQL tool (ie Squirrel SQL, Oracle SQL 
Developer) to HiveServer2 (running on remote server) with port 1.

I am able to run some queries successfully. I then do something else (not in 
the SQL tool) for 1-2minutes and then return to my SQL tool and attempt to run 
a query but I get this error: {{}}
{code:java}
org.apache.thrift.transport.TTransportException: java.net.SocketException: 
Software caused connection abort: socket write error{code}
If I now disconnect and reconnect in my SQL tool I can run queries again. But 
does anyone know what HiveServer2 settings I should change to prevent the 
error? I assume something in hive-site.xml

>From the hiveserver2 logs below, can see an exact 1 minute gap from 30th min 
>to 31stmin where the disconnect happens.
{code:java}
2018-04-05T03:30:41,706 INFO [HiveServer2-Handler-Pool: Thread-36] 
session.SessionState: Resetting thread name to HiveServer2-Handler-Pool: 
Thread-36
 2018-04-05T03:30:41,712 INFO [HiveServer2-Handler-Pool: Thread-36] 
session.SessionState: Updating thread name to 
c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
 2018-04-05T03:30:41,712 INFO [HiveServer2-Handler-Pool: Thread-36] 
session.SessionState: Resetting thread name to HiveServer2-Handler-Pool: 
Thread-36
 2018-04-05T03:30:41,718 INFO [HiveServer2-Handler-Pool: Thread-36] 
session.SessionState: Updating thread name to 
c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
 2018-04-05T03:30:41,719 INFO [HiveServer2-Handler-Pool: Thread-36] 
session.SessionState: Resetting thread name to HiveServer2-Handler-Pool: 
Thread-36
 2018-04-05T03:31:41,232 INFO [HiveServer2-Handler-Pool: Thread-36] 
thrift.ThriftCLIService: Session disconnected without closing properly.
 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36] 
thrift.ThriftCLIService: Closing the session: SessionHandle 
[c81ec0f9-7a9d-46b6-9708-e7d78520a48a]
 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36] 
service.CompositeService: Session closed, SessionHandle 
[c81ec0f9-7a9d-46b6-9708-e7d78520a48a], current sessions:0
 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36] 
session.SessionState: Updating thread name to 
c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36] 
session.SessionState: Resetting thread name to HiveServer2-Handler-Pool: 
Thread-36
 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36] 
session.SessionState: Updating thread name to 
c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36] 
session.HiveSessionImpl: Operation log session directory is deleted: 
/var/hive/hs2log/tmp/c81ec0f9-7a9d-46b6-9708-e7d78520a48a
 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36] 
session.SessionState: Resetting thread name to HiveServer2-Handler-Pool: 
Thread-36
 2018-04-05T03:31:41,236 INFO [HiveServer2-Handler-Pool: Thread-36] 
session.SessionState: Deleted directory: 
/var/hive/scratch/tmp/anonymous/c81ec0f9-7a9d-46b6-9708-e7d78520a48a on fs with 
scheme file
 2018-04-05T03:31:41,236 INFO [HiveServer2-Handler-Pool: Thread-36] 
session.SessionState: Deleted directory: 
/var/hive/ec2-user/c81ec0f9-7a9d-46b6-9708-e7d78520a48a on fs with scheme file
 2018-04-05T03:31:41,236 INFO [HiveServer2-Handler-Pool: Thread-36] 
hive.metastore: Closed a connection to metastore, current connections: 1{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19116) Vectorization: Vector Map data type doesn't keep the order of the key/values pairs as read

[Matt McCline (JIRA)]

Matt McCline created HIVE-19116:
---

 Summary: Vectorization: Vector Map data type doesn't keep the 
order of the key/values pairs as read
 Key: HIVE-19116
 URL: https://issues.apache.org/jira/browse/HIVE-19116
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Matt McCline
Assignee: Matt McCline


The VectorExtractRow class does not preserve the order of the key/value pairs 
when going from MapColumnVector to a Map object.  This causes Q file 
differences in tests with the MAP data type making it seem like we are getting 
Wrong Results (well, actually we are).

When LazyMap class (for example) adds key/value pairs to its "map" it uses a 
LinkedHashSet to preserve insert order.

FYI: [~teddy.choi]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)