[GitHub] hive pull request #347: HIVE-17989: Adding support for Schema Registry objec...

[harshach]

Github user harshach closed the pull request at:

https://github.com/apache/hive/pull/347


---

[jira] [Created] (HIVE-20131) SQL Script changes for creating txn write notification in 3.2.0 files

[mahesh kumar behera (JIRA)]

mahesh kumar behera created HIVE-20131:
--

 Summary: SQL Script changes for creating  txn write notification 
in 3.2.0 files 
 Key: HIVE-20131
 URL: https://issues.apache.org/jira/browse/HIVE-20131
 Project: Hive
  Issue Type: Sub-task
  Components: repl, Transactions
Affects Versions: 3.0.0
Reporter: mahesh kumar behera
Assignee: mahesh kumar behera
 Fix For: 4.0.0


 
h1. Replicate ACID write Events
 * Create new EVENT_WRITE event with related message format to log the write 
operations with in a txn along with data associated.

 * Log this event when perform any writes (insert into, insert overwrite, load 
table, delete, update, merge, truncate) on table/partition.

 * If a single MERGE/UPDATE/INSERT/DELETE statement operates on multiple 
partitions, then need to log one event per partition.

 * DbNotificationListener should log this type of event to special metastore 
table named "MTxnWriteNotificationLog".

 * This table should maintain a map of txn ID against list of tables/partitions 
written by given txn.

 * The entry for a given txn should be removed by the cleaner thread that 
removes the expired events from EventNotificationTable.

h1. Replicate Commit Txn operation (with writes)

Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions 
modified within the txn.

*Source warehouse:*
 * This event should read the EVENT_WRITEs from "MTxnWriteNotificationLog" 
metastore table to consolidate the list of tables/partitions modified within 
this txn scope.

 * Based on the list of tables/partitions modified and table Write ID, need to 
compute the list of delta files added by this txn.

 * Repl dump should read this message and dump the metadata and delta files 
list.

*Target warehouse:*
 * Ensure snapshot isolation at target for on-going read txns which shouldn't 
view the data replicated from committed txn. (Ensured with open and allocate 
write ID events).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20130) Better logging for information schema synchronizer

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-20130:
-

 Summary: Better logging for information schema synchronizer
 Key: HIVE-20130
 URL: https://issues.apache.org/jira/browse/HIVE-20130
 Project: Hive
  Issue Type: Improvement
Reporter: Daniel Dai
Assignee: Daniel Dai
 Attachments: HIVE-20130.1.patch

The logging of information schema synchronizer should be more useful.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20129) Revert to position based schema evolution for orc tables

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-20129:


 Summary: Revert to position based schema evolution for orc tables
 Key: HIVE-20129
 URL: https://issues.apache.org/jira/browse/HIVE-20129
 Project: Hive
  Issue Type: Bug
Affects Versions: 4.0.0, 3.2.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran
 Attachments: HIVE-20129.1.patch

Hive has been doing positional based schema evolution. ORC-54 changed it to 
column name based schema evolution causing unexpected results. Queries returned 
results earlier are now returning no results. Change the default in hive to 
positional schema evolution. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20128) add partial read support to LLAP Parquet cache

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20128:
---

 Summary: add partial read support to LLAP Parquet cache
 Key: HIVE-20128
 URL: https://issues.apache.org/jira/browse/HIVE-20128
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


Follow-up from HIVE-20127 to allow the reader to accumulate reads for several 
buffers until the aggregate data read covers the pre-determined cache 
boundaries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20127) fix some issues with LLAP Parquet cache

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-20127:
---

 Summary: fix some issues with LLAP Parquet cache
 Key: HIVE-20127
 URL: https://issues.apache.org/jira/browse/HIVE-20127
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 67858: HIVE-20126: OrcInputFormat does not pass conf to orc reader options

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67858/
---

Review request for hive and Sergey Shelukhin.


Bugs: HIVE-20126
https://issues.apache.org/jira/browse/HIVE-20126


Repository: hive-git


Description
---

HIVE-20126: OrcInputFormat does not pass conf to orc reader options


Diffs
-

  data/files/studenttab10k PRE-CREATION 
  itests/src/test/resources/testconfiguration.properties 
2795e3deca407e45ec22b43767ff48c906cc3823 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 
3c118474af67f2cc680a07747f142af9433db04a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
892fcc0843b1137317ac00343a5ba55419baadc5 
  ql/src/test/queries/clientpositive/orc_schema_evol_quoted.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/orc_schema_evol_quoted.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/orc_schema_evol_quoted.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/67858/diff/1/


Testing
---


Thanks,

Prasanth_J

[jira] [Created] (HIVE-20126) VectorizedOrcInputFormat does not pass conf to orc reader options

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-20126:


 Summary: VectorizedOrcInputFormat does not pass conf to orc reader 
options
 Key: HIVE-20126
 URL: https://issues.apache.org/jira/browse/HIVE-20126
 Project: Hive
  Issue Type: Bug
Affects Versions: 4.0.0, 3.2.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


VectorizedOrcInputFormat creates Orc reader options without passing in the 
configuration object. Without it setting orc configurations will not have any 
impact. 

Example: 

set orc.force.positional.evolution=true;

does not work for positional schema evolution (will attach test case).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20125) Typo in MetricsCollection for OutputMetrics

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-20125:
---

 Summary: Typo in MetricsCollection for OutputMetrics
 Key: HIVE-20125
 URL: https://issues.apache.org/jira/browse/HIVE-20125
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Reporter: Sahil Takiar


When creating {{OutputMetrics}} in the {{aggregate}} method we check for 
{{hasInputMetrics}} instead of {{hasOutputMetrics}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20124) Re-add HIVE-19787: Log message when spark-submit has completed

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-20124:
---

 Summary: Re-add HIVE-19787: Log message when spark-submit has 
completed
 Key: HIVE-20124
 URL: https://issues.apache.org/jira/browse/HIVE-20124
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Reporter: Sahil Takiar
Assignee: Sahil Takiar


Think this accidentally go reverted when re-basing HIVE-18916



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20123) Fix masking tests after HIVE-19617

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-20123:
--

 Summary: Fix masking tests after HIVE-19617
 Key: HIVE-20123
 URL: https://issues.apache.org/jira/browse/HIVE-20123
 Project: Hive
  Issue Type: Test
Affects Versions: 3.0.0, 3.1.0, 4.0.0, 3.2.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Masking tests results were changed inadvertently when HIVE-19617 went in, since 
table names were changed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20122) Deploy and test standalone metastore for Hive 3.1

[Vihang Karajgaonkar (JIRA)]

Vihang Karajgaonkar created HIVE-20122:
--

 Summary: Deploy and test standalone metastore for Hive 3.1
 Key: HIVE-20122
 URL: https://issues.apache.org/jira/browse/HIVE-20122
 Project: Hive
  Issue Type: Task
Affects Versions: 3.1.0
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


Creating a blocker JIRA for 3.1 so that this does not slip under radar. This 
jira tracks testing effort for standalone metastore for 3.1 release. I will 
create sub-tasks if I find any issues.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Should we release storage-api 2.7.0 rc0?

[Deepak Jaiswal]

Thanks Alan.

On 7/9/18, 10:17 AM, "Alan Gates"  wrote:

+1.  Did a build with a clean maven repo, checked the signature and sha
hash, ran RAT.

Alan.

On Fri, Jul 6, 2018 at 2:21 PM Deepak Jaiswal 
wrote:

> Hi,
>
> I would like to make a new release of the storage-api. It contains changes
> required for Hive 3.1 release.
>
> Artifcats:
> Tag :
> https://github.com/apache/hive/releases/tag/storage-release-2.7.0-rc0
> Tar Ball : http://home.apache.org/~djaiswal/hive-storage-2.7.0/
>
> Regards,
> Deepak
>

Re: [VOTE] Should we release storage-api 2.7.0 rc0?

[Alan Gates]

+1.  Did a build with a clean maven repo, checked the signature and sha
hash, ran RAT.

Alan.

On Fri, Jul 6, 2018 at 2:21 PM Deepak Jaiswal 
wrote:

> Hi,
>
> I would like to make a new release of the storage-api. It contains changes
> required for Hive 3.1 release.
>
> Artifcats:
> Tag :
> https://github.com/apache/hive/releases/tag/storage-release-2.7.0-rc0
> Tar Ball : http://home.apache.org/~djaiswal/hive-storage-2.7.0/
>
> Regards,
> Deepak
>

Register now for ApacheCon and save $250

[Rich Bowen]

Greetings, Apache software enthusiasts!

(You’re getting this because you’re on one or more dev@ or users@ lists 
for some Apache Software Foundation project.)


ApacheCon North America, in Montreal, is now just 80 days away, and 
early bird prices end in just two weeks - on July 21. Prices will be 
going up from $550 to $800 so register NOW to save $250, at 
http://apachecon.com/acna18


And don’t forget to reserve your hotel room. We have negotiated a 
special rate and the room block closes August 24. 
http://www.apachecon.com/acna18/venue.html


Our schedule includes over 100 talks and we’ll be featuring talks from 
dozens of ASF projects.,  We have inspiring keynotes from some of the 
brilliant members of our community and the wider tech space, including:


 * Myrle Krantz, PMC chair for Apache Fineract, and leader in the open 
source financing space
 * Cliff Schmidt, founder of Literacy Bridge (now Amplio) and creator 
of the Talking Book project

 * Bridget Kromhout, principal cloud developer advocate at Microsoft
 * Euan McLeod, Comcast engineer, and pioneer in streaming video

We’ll also be featuring tracks for Geospatial science, Tomcat, 
Cloudstack, and Big Data, as well as numerous other fields where Apache 
software is leading the way. See the full schedule at 
http://apachecon.com/acna18/schedule.html


As usual we’ll be running our Apache BarCamp, the traditional ApacheCon 
Hackathon, and the Wednesday evening Lighting Talks, too, so you’ll want 
to be there.


Register today at http://apachecon.com/acna18 and we’ll see you in Montreal!

--
Rich Bowen
VP, Conferences, The Apache Software Foundation
h...@apachecon.com
@ApacheCon

Re: Hive QA batches timing out

[Mahesh Kumar Behera]

The " Unable to shutdown metastore client' error is coming from shutting down 
syncMetaStoreClient, as both syncMetaStoreClient and metaStoreClient shares the 
same client.  I think in Hive:: close method, we should not call 
syncMetaStoreClient.close().

On 7/9/18, 12:49 PM, "Zoltan Haindrich"  wrote:

I've a feeling that sometimes the same issue happens in other tests - but I 
agree; disabling it will make our life easier - until the real cause is 
uncovered and fixed.

cheers,
Zoltan


On 07/09/2018 09:05 AM, Deepak Jaiswal wrote:
> Thanks Zoltan for the analysis. Perhaps we should disable the test in the 
meantime as it is blocking several people from committing.
> 
> I can go ahead and create a patch for it.
> 
> Regards,
> Deepak
> 
> On 7/8/18, 11:33 PM, "Zoltan Haindrich"  wrote:
> 
>  Hello
>  
>  Thank you Deepak for taking a closer look! from what you've 
found I've noticed that the runtime of TestReplicationScenariosAcidTables have 
jumped up to ~2000sec in the
>  runs which have failedit seems like this problem is there for a 
long time now; I've found jira tickets in which this test was "timed out" and 
the HiveQA comment was
>  date at April 03so it's not entirely new...
>  
>  The problem which prohibits this test from completing successfully 
seems like that it has difficulties closing down the metastore client - which 
goes on for a while ...
>  I don't know if this is an acid/replication/metastore/? issue...but 
it seems intermittent - I've a hunch that somehow it might happen more reliably 
with this test...I've
>  opened HIVE-20121 to investigate this...
>  
>  2018-07-08T22:07:33,461 DEBUG [main] metastore.HiveMetaStoreClient: 
Unable to shutdown metastore client. Will try closing transport directly.
>  org.apache.thrift.transport.TTransportException: Cannot write to 
null outputStream
>  
>  some links to more or less recent logs:
>  
http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/maven-test.txt
>  the hive.log is ~200M:
>  
http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/logs/hive.log
>  
>  
>  cheers,
>  Zoltan
>  
>  On 07/08/2018 06:49 PM, Deepak Jaiswal wrote:
>  > I am seeing tests timing out in my latest ptest run,
>  >
>  > https://builds.apache.org/job/PreCommit-HIVE-Build/12468/testReport
>  > https://builds.apache.org/job/PreCommit-HIVE-Build/12468/console
>  >
>  > TestAlterTableMetadata - did not produce a TEST-*.xml file (likely 
timed out) (batchId=240)
>  > TestAutoPurgeTables - did not produce a TEST-*.xml file (likely 
timed out) (batchId=240)
>  > TestLocationQueries - did not produce a TEST-*.xml file (likely 
timed out) (batchId=240)
>  > TestReplicationScenariosAcidTables - did not produce a TEST-*.xml 
file (likely timed out) (batchId=240)
>  > TestSemanticAnalyzerHookLoading - did not produce a TEST-*.xml 
file (likely timed out) (batchId=240)
>  > TestSparkStatistics - did not produce a TEST-*.xml file (likely 
timed out) (batchId=240)
>  >
>  >
>  >  From the Hive QA homepage, the last stable build was 12444 
whereas the current run is 12473. I looked at some of the runs in between and 
it looks like most of the runs are failing due to the above batch of unit tests.
>  >
>  > Regards,
>  > Deepak
>  >
>  
>  
> 

Re: Review Request 67801: HIVE-19337 Partition whitelist regex doesn't work (and never did)

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67801/#review205844
---



Looks good. Might be good idea to have a test for this.


metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Line 307 (original), 307 (patched)


Thanks, for catching this!

Do we have tests to confirm that we use this correctly now, and it will not 
regress?


- Peter Vary


On júl. 3, 2018, 12:28 de, Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67801/
> ---
> 
> (Updated júl. 3, 2018, 12:28 de)
> 
> 
> Review request for hive, Andrew Sherman, Janaki Lahorani, Peter Vary, Sahil 
> Takiar, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19337
> https://issues.apache.org/jira/browse/HIVE-19337
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-19337 Partition whitelist regex doesn't work (and never did)
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> ba46e63839e4b02d325007b16756ccd8f3240f63 
> 
> 
> Diff: https://reviews.apache.org/r/67801/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: Hive QA batches timing out

[Zoltan Haindrich]

I've a feeling that sometimes the same issue happens in other tests - but I 
agree; disabling it will make our life easier - until the real cause is 
uncovered and fixed.

cheers,
Zoltan


On 07/09/2018 09:05 AM, Deepak Jaiswal wrote:

Thanks Zoltan for the analysis. Perhaps we should disable the test in the 
meantime as it is blocking several people from committing.

I can go ahead and create a patch for it.

Regards,
Deepak

On 7/8/18, 11:33 PM, "Zoltan Haindrich"  wrote:

 Hello
 
 Thank you Deepak for taking a closer look! from what you've found I've noticed that the runtime of TestReplicationScenariosAcidTables have jumped up to ~2000sec in the

 runs which have failedit seems like this problem is there for a long time now; 
I've found jira tickets in which this test was "timed out" and the HiveQA 
comment was
 date at April 03so it's not entirely new...
 
 The problem which prohibits this test from completing successfully seems like that it has difficulties closing down the metastore client - which goes on for a while ...

 I don't know if this is an acid/replication/metastore/? issue...but it 
seems intermittent - I've a hunch that somehow it might happen more reliably 
with this test...I've
 opened HIVE-20121 to investigate this...
 
 2018-07-08T22:07:33,461 DEBUG [main] metastore.HiveMetaStoreClient: Unable to shutdown metastore client. Will try closing transport directly.

 org.apache.thrift.transport.TTransportException: Cannot write to null 
outputStream
 
 some links to more or less recent logs:

 
http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/maven-test.txt
 the hive.log is ~200M:
 
http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/logs/hive.log
 
 
 cheers,

 Zoltan
 
 On 07/08/2018 06:49 PM, Deepak Jaiswal wrote:

 > I am seeing tests timing out in my latest ptest run,
 >
 > https://builds.apache.org/job/PreCommit-HIVE-Build/12468/testReport
 > https://builds.apache.org/job/PreCommit-HIVE-Build/12468/console
 >
 > TestAlterTableMetadata - did not produce a TEST-*.xml file (likely timed 
out) (batchId=240)
 > TestAutoPurgeTables - did not produce a TEST-*.xml file (likely timed 
out) (batchId=240)
 > TestLocationQueries - did not produce a TEST-*.xml file (likely timed 
out) (batchId=240)
 > TestReplicationScenariosAcidTables - did not produce a TEST-*.xml file 
(likely timed out) (batchId=240)
 > TestSemanticAnalyzerHookLoading - did not produce a TEST-*.xml file 
(likely timed out) (batchId=240)
 > TestSparkStatistics - did not produce a TEST-*.xml file (likely timed 
out) (batchId=240)
 >
 >
 >  From the Hive QA homepage, the last stable build was 12444 whereas the 
current run is 12473. I looked at some of the runs in between and it looks like 
most of the runs are failing due to the above batch of unit tests.
 >
 > Regards,
 > Deepak
 >
 
 

Re: Hive QA batches timing out

[Deepak Jaiswal]

Thanks Zoltan for the analysis. Perhaps we should disable the test in the 
meantime as it is blocking several people from committing.

I can go ahead and create a patch for it.

Regards,
Deepak

On 7/8/18, 11:33 PM, "Zoltan Haindrich"  wrote:

Hello

Thank you Deepak for taking a closer look! from what you've found I've 
noticed that the runtime of TestReplicationScenariosAcidTables have jumped up 
to ~2000sec in the 
runs which have failedit seems like this problem is there for a long 
time now; I've found jira tickets in which this test was "timed out" and the 
HiveQA comment was 
date at April 03so it's not entirely new...

The problem which prohibits this test from completing successfully seems 
like that it has difficulties closing down the metastore client - which goes on 
for a while ...
I don't know if this is an acid/replication/metastore/? issue...but it 
seems intermittent - I've a hunch that somehow it might happen more reliably 
with this test...I've 
opened HIVE-20121 to investigate this...

2018-07-08T22:07:33,461 DEBUG [main] metastore.HiveMetaStoreClient: Unable 
to shutdown metastore client. Will try closing transport directly.
org.apache.thrift.transport.TTransportException: Cannot write to null 
outputStream

some links to more or less recent logs:

http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/maven-test.txt
the hive.log is ~200M:

http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/logs/hive.log


cheers,
Zoltan

On 07/08/2018 06:49 PM, Deepak Jaiswal wrote:
> I am seeing tests timing out in my latest ptest run,
> 
> https://builds.apache.org/job/PreCommit-HIVE-Build/12468/testReport
> https://builds.apache.org/job/PreCommit-HIVE-Build/12468/console
> 
> TestAlterTableMetadata - did not produce a TEST-*.xml file (likely timed 
out) (batchId=240)
> TestAutoPurgeTables - did not produce a TEST-*.xml file (likely timed 
out) (batchId=240)
> TestLocationQueries - did not produce a TEST-*.xml file (likely timed 
out) (batchId=240)
> TestReplicationScenariosAcidTables - did not produce a TEST-*.xml file 
(likely timed out) (batchId=240)
> TestSemanticAnalyzerHookLoading - did not produce a TEST-*.xml file 
(likely timed out) (batchId=240)
> TestSparkStatistics - did not produce a TEST-*.xml file (likely timed 
out) (batchId=240)
> 
> 
>  From the Hive QA homepage, the last stable build was 12444 whereas the 
current run is 12473. I looked at some of the runs in between and it looks like 
most of the runs are failing due to the above batch of unit tests.
> 
> Regards,
> Deepak
> 

[jira] [Created] (HIVE-20121) investigate issues with TestReplicationScenariosAcidTables

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-20121:
---

 Summary: investigate issues with TestReplicationScenariosAcidTables
 Key: HIVE-20121
 URL: https://issues.apache.org/jira/browse/HIVE-20121
 Project: Hive
  Issue Type: Bug
Reporter: Zoltan Haindrich


[~djaiswal] have noticed that somehow ptest execution are getting more and more 
problems lately... it seems to me that these problems are coming from this test

{code}
2018-07-08T22:07:33,461 DEBUG [main] metastore.HiveMetaStoreClient: Unable to 
shutdown metastore client. Will try closing transport directly.
org.apache.thrift.transport.TTransportException: Cannot write to null 
outputStream
{code}


some links to more or less recent logs:
http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/maven-test.txt
the hive.log is ~200M:
http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/logs/hive.log



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Hive QA batches timing out

[Zoltan Haindrich]

Hello

Thank you Deepak for taking a closer look! from what you've found I've noticed that the runtime of TestReplicationScenariosAcidTables have jumped up to ~2000sec in the 
runs which have failedit seems like this problem is there for a long time now; I've found jira tickets in which this test was "timed out" and the HiveQA comment was 
date at April 03so it's not entirely new...


The problem which prohibits this test from completing successfully seems like 
that it has difficulties closing down the metastore client - which goes on for 
a while ...
I don't know if this is an acid/replication/metastore/? issue...but it seems intermittent - I've a hunch that somehow it might happen more reliably with this test...I've 
opened HIVE-20121 to investigate this...


2018-07-08T22:07:33,461 DEBUG [main] metastore.HiveMetaStoreClient: Unable to 
shutdown metastore client. Will try closing transport directly.
org.apache.thrift.transport.TTransportException: Cannot write to null 
outputStream

some links to more or less recent logs:
http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/maven-test.txt
the hive.log is ~200M:
http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/logs/hive.log


cheers,
Zoltan

On 07/08/2018 06:49 PM, Deepak Jaiswal wrote:

I am seeing tests timing out in my latest ptest run,

https://builds.apache.org/job/PreCommit-HIVE-Build/12468/testReport
https://builds.apache.org/job/PreCommit-HIVE-Build/12468/console

TestAlterTableMetadata - did not produce a TEST-*.xml file (likely timed out) 
(batchId=240)
TestAutoPurgeTables - did not produce a TEST-*.xml file (likely timed out) 
(batchId=240)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out) 
(batchId=240)
TestReplicationScenariosAcidTables - did not produce a TEST-*.xml file (likely 
timed out) (batchId=240)
TestSemanticAnalyzerHookLoading - did not produce a TEST-*.xml file (likely 
timed out) (batchId=240)
TestSparkStatistics - did not produce a TEST-*.xml file (likely timed out) 
(batchId=240)


 From the Hive QA homepage, the last stable build was 12444 whereas the current 
run is 12473. I looked at some of the runs in between and it looks like most of 
the runs are failing due to the above batch of unit tests.

Regards,
Deepak