[jira] [Created] (HIVE-15170) Flaky test: TestEmbeddedThriftBinaryCLIService.testTaskStatus

2016-11-09 Thread Barna Zsombor Klara (JIRA) 
Barna Zsombor Klara created HIVE-15170:
--

 Summary: Flaky test: 
TestEmbeddedThriftBinaryCLIService.testTaskStatus
 Key: HIVE-15170
 URL: https://issues.apache.org/jira/browse/HIVE-15170
 Project: Hive
  Issue Type: Sub-task
Reporter: Barna Zsombor Klara


Test failed in:
https://builds.apache.org/job/PreCommit-HIVE-Build/2043/
https://issues.apache.org/jira/browse/HIVE-12891 (Test results have been 
deleted but the test failed in two subsequent runs without a change to the 
patch file)

Logs:
{code}
2016-11-08T18:39:31,295 ERROR [167769e0-ccac-41b0-b271-4175e0c37a79 main] 
SessionState: PREHOOK: query: CREATE TABLE TEST_EXEC_ASYNC(ID STRING)
PREHOOK: type: CREATETABLE
2016-11-08T18:39:31,295 ERROR [167769e0-ccac-41b0-b271-4175e0c37a79 main] 
SessionState: PREHOOK: type: CREATETABLE
PREHOOK: Output: database:default
2016-11-08T18:39:31,295 ERROR [167769e0-ccac-41b0-b271-4175e0c37a79 main] 
SessionState: PREHOOK: Output: database:default
PREHOOK: Output: default@TEST_EXEC_ASYNC
2016-11-08T18:39:31,295 ERROR [167769e0-ccac-41b0-b271-4175e0c37a79 main] 
SessionState: PREHOOK: Output: default@TEST_EXEC_ASYNC

2016-11-08T18:39:31,369 ERROR [167769e0-ccac-41b0-b271-4175e0c37a79 main] 
SessionState: POSTHOOK: query: CREATE TABLE TEST_EXEC_ASYNC(ID STRING)
POSTHOOK: type: CREATETABLE
2016-11-08T18:39:31,370 ERROR [167769e0-ccac-41b0-b271-4175e0c37a79 main] 
SessionState: POSTHOOK: type: CREATETABLE
POSTHOOK: Output: database:default
2016-11-08T18:39:31,370 ERROR [167769e0-ccac-41b0-b271-4175e0c37a79 main] 
SessionState: POSTHOOK: Output: database:default
POSTHOOK: Output: default@TEST_EXEC_ASYNC
2016-11-08T18:39:31,370 ERROR [167769e0-ccac-41b0-b271-4175e0c37a79 main] 
SessionState: POSTHOOK: Output: default@TEST_EXEC_ASYNC
...
2016-11-08T18:39:31,711 ERROR [HiveServer2-Background-Pool: Thread-4824] 
SessionState: PREHOOK: query: select a.id, b.id from (SELECT ID + ' ' `ID` FROM 
TEST_EXEC_ASYNC) a full outer join (SELECT ID + ' ' `ID` FROM TEST_EXEC_ASYNC) 
b on a.ID=b.ID
PREHOOK: type: QUERY
2016-11-08T18:39:31,711 ERROR [HiveServer2-Background-Pool: Thread-4824] 
SessionState: PREHOOK: type: QUERY
PREHOOK: Input: default@test_exec_async
2016-11-08T18:39:31,711 ERROR [HiveServer2-Background-Pool: Thread-4824] 
SessionState: PREHOOK: Input: default@test_exec_async
PREHOOK: Output: 
file:/home/hiveptest/54.177.123.16-hiveptest-1/apache-github-source-source/itests/hive-unit/target/tmp/localscratchdir/167769e0-ccac-41b0-b271-4175e0c37a79/hive_2016-11-08_18-39-31_373_2208326672728744152-1/-mr-10001
2016-11-08T18:39:31,712 ERROR [HiveServer2-Background-Pool: Thread-4824] 
SessionState: PREHOOK: Output: 
file:/home/hiveptest/54.177.123.16-hiveptest-1/apache-github-source-source/itests/hive-unit/target/tmp/localscratchdir/167769e0-ccac-41b0-b271-4175e0c37a79/hive_2016-11-08_18-39-31_373_2208326672728744152-1/-mr-10001
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15169) Flaky test: TestMiniLlapLocalCliDriver.testCliDriver.columnstats_part_coltype

2016-11-09 Thread Barna Zsombor Klara (JIRA) 
Barna Zsombor Klara created HIVE-15169:
--

 Summary: Flaky test: 
TestMiniLlapLocalCliDriver.testCliDriver.columnstats_part_coltype
 Key: HIVE-15169
 URL: https://issues.apache.org/jira/browse/HIVE-15169
 Project: Hive
  Issue Type: Sub-task
Reporter: Barna Zsombor Klara


Flaky runs:
https://builds.apache.org/job/PreCommit-HIVE-Build/2043/
https://issues.apache.org/jira/browse/HIVE-12891 (Test report missing, but test 
failed in a second run without a new patch file being uploaded)

Error Message
Unexpected exception java.lang.AssertionError: Client execution failed with 
error code = 4 running 
analyze table partcolstatsnum partition (tint=100, sint=1000, bint=100) 
compute statistics for columnsfname=columnstats_part_coltype.q



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15168) Flaky test: TestSparkClient.testJobSubmission (still flaky)

2016-11-09 Thread Barna Zsombor Klara (JIRA) 
Barna Zsombor Klara created HIVE-15168:
--

 Summary: Flaky test: TestSparkClient.testJobSubmission (still 
flaky)
 Key: HIVE-15168
 URL: https://issues.apache.org/jira/browse/HIVE-15168
 Project: Hive
  Issue Type: Sub-task
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara


[HIVE-14910|https://issues.apache.org/jira/browse/HIVE-14910] already addressed 
one source of flakyness bud sadly not all it seems.
In JobHandleImpl the listeners are registered after the job has been submitted.
This may end up in a racecondition.
{code}
 // Link the RPC and the promise so that events from one are propagated to the 
other as
  // needed.
  rpc.addListener(new 
GenericFutureListener() {
@Override
public void operationComplete(io.netty.util.concurrent.Future f) {
  if (f.isSuccess()) {
handle.changeState(JobHandle.State.QUEUED);
  } else if (!promise.isDone()) {
promise.setFailure(f.cause());
  }
}
  });
  promise.addListener(new GenericFutureListener() {
@Override
public void operationComplete(Promise p) {
  if (jobId != null) {
jobs.remove(jobId);
  }
  if (p.isCancelled() && !rpc.isDone()) {
rpc.cancel(true);
  }
}
  });
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 53609: HIVE-15168: Flaky test: TestSparkClient.testJobSubmission (still flaky)

2016-11-09 Thread Barna Zsombor Klara 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53609/
---

Review request for hive, Rui Li and Xuefu Zhang.


Repository: hive-git


Description
---

HIVE-15168: Flaky test: TestSparkClient.testJobSubmission (still flaky)


Diffs
-

  spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java 
863aaa877769094164c209bd5861515d91ed250c 
  spark-client/src/main/java/org/apache/hive/spark/client/rpc/Rpc.java 
b2f133b6aa3abdb74e83eab8b14cd827b0de60dc 
  spark-client/src/test/java/org/apache/hive/spark/client/rpc/TestRpc.java 
77c3d020529d60539e3044aaa36a7fb69583ea4b 

Diff: https://reviews.apache.org/r/53609/diff/


Testing
---

Ran modified unit tests / tests of the modified classes.
Ran simple queries with spark as the execution engine.
Tested that the jobSubmission test is passing if the thread is blocked for 3 
seconds in the ClientProtocol#submit.


Thanks,

Barna Zsombor Klara

Re: Review Request 53609: HIVE-15168: Flaky test: TestSparkClient.testJobSubmission (still flaky)

2016-11-09 Thread Peter Vary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53609/#review155444
---



Thanks Zsombor for the fix! One more down, but wy much to go :)
Just minor nits.


spark-client/src/main/java/org/apache/hive/spark/client/rpc/Rpc.java (line 255)


nit: Please change the comment to match the new signature



spark-client/src/main/java/org/apache/hive/spark/client/rpc/Rpc.java (line 263)


nit: Please change the comment to match the new signature



spark-client/src/main/java/org/apache/hive/spark/client/rpc/Rpc.java (line 280)


nit: Please change the comment to match the new signature



spark-client/src/test/java/org/apache/hive/spark/client/rpc/TestRpc.java (line 
39)


nit: Is this import is used after this change?


- Peter Vary


On Nov. 9, 2016, 3:20 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53609/
> ---
> 
> (Updated Nov. 9, 2016, 3:20 p.m.)
> 
> 
> Review request for hive, Rui Li and Xuefu Zhang.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15168: Flaky test: TestSparkClient.testJobSubmission (still flaky)
> 
> 
> Diffs
> -
> 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java 
> 863aaa877769094164c209bd5861515d91ed250c 
>   spark-client/src/main/java/org/apache/hive/spark/client/rpc/Rpc.java 
> b2f133b6aa3abdb74e83eab8b14cd827b0de60dc 
>   spark-client/src/test/java/org/apache/hive/spark/client/rpc/TestRpc.java 
> 77c3d020529d60539e3044aaa36a7fb69583ea4b 
> 
> Diff: https://reviews.apache.org/r/53609/diff/
> 
> 
> Testing
> ---
> 
> Ran modified unit tests / tests of the modified classes.
> Ran simple queries with spark as the execution engine.
> Tested that the jobSubmission test is passing if the thread is blocked for 3 
> seconds in the ClientProtocol#submit.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

[jira] [Created] (HIVE-15171) set SparkTask's jobID with application id

2016-11-09 Thread zhihai xu (JIRA) 
zhihai xu created HIVE-15171:


 Summary: set SparkTask's jobID with application id
 Key: HIVE-15171
 URL: https://issues.apache.org/jira/browse/HIVE-15171
 Project: Hive
  Issue Type: Improvement
  Components: Spark
Affects Versions: 2.1.0
Reporter: zhihai xu
Assignee: zhihai xu


set SparkTask's jobID with application id, The information will be useful to 
monitor the Spark Application in hook



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15172) Flaky test: TestSparkCliDriver.testCliDriver[limit_pushdown]

2016-11-09 Thread Jason Dere (JIRA) 
Jason Dere created HIVE-15172:
-

 Summary: Flaky test: 
TestSparkCliDriver.testCliDriver[limit_pushdown]
 Key: HIVE-15172
 URL: https://issues.apache.org/jira/browse/HIVE-15172
 Project: Hive
  Issue Type: Sub-task
  Components: Tests
Reporter: Jason Dere


Looks like this has been failing on recent precommit tests



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52912: HIVE-13557: Make interval keyword optional while specifying DAY in interval arithmetic

2016-11-09 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52912/#review155521
---




ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g (line 294)


I would rather put the new rewrite rule here for intervalLiteral. Here you 
can branch out, and say it can also match the new requirement. 
for example, expr ... etc.



ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g (line 347)


Then "select date '2012-01-01' + 1 days from t;" can not be supported as 
there is no parenthese.


- pengcheng xiong


On Oct. 15, 2016, 9:27 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52912/
> ---
> 
> (Updated Oct. 15, 2016, 9:27 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-13557
> https://issues.apache.org/jira/browse/HIVE-13557
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> * added intervalExpression
> * syntax: INTERVAL (c) DAY
> * added days/months/seconds/years
> * kept the existing (interval '8' days) as is
> * INTERVAL keyword usage is optional
> * moving this into an udf seemed like a good thing, this way: it may accept 
> column refs or expressions as input (sql2011)
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
> 0dbbc1d6226330bcb8271abba8b2675adaeb6f0f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 
> 025ea10cda5b126f46226809884088caadc0a3d0 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 
> 13f68796e4fe262098eaf6c5f09bc7f963d0f8cf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInternalInterval.java
>  PRE-CREATION 
>   
> ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFInternalInterval.java
>  PRE-CREATION 
>   ql/src/test/queries/clientpositive/interval_alt.q PRE-CREATION 
>   ql/src/test/results/clientpositive/interval_alt.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/52912/diff/
> 
> 
> Testing
> ---
> 
> * small unit test to check exceptions
> * added qtest
> 
> 
> Thanks,
> 
> Zoltan Haindrich
> 
>

[jira] [Created] (HIVE-15173) Allow dec as an alias for decimal

2016-11-09 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-15173:
---

 Summary: Allow dec as an alias for decimal
 Key: HIVE-15173
 URL: https://issues.apache.org/jira/browse/HIVE-15173
 Project: Hive
  Issue Type: Sub-task
  Components: Parser
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


Standard allows dec as an alias for decimal



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15174) Respect auth_to_local rules from hdfs configs (core-site.xml) for LDAP authentication too

2016-11-09 Thread Ruslan Dautkhanov (JIRA) 
Ruslan Dautkhanov created HIVE-15174:


 Summary: Respect auth_to_local rules from hdfs configs 
(core-site.xml) for LDAP authentication too
 Key: HIVE-15174
 URL: https://issues.apache.org/jira/browse/HIVE-15174
 Project: Hive
  Issue Type: Bug
  Components: Authentication, HiveServer2, Security
Affects Versions: 2.1.0, 1.2.1, 1.1.1
 Environment: Hive 1.1; Hadoop 2.6
Reporter: Ruslan Dautkhanov


Hive has implemented Kerberos principal mapping for authentication; the same 
should be implemented for LDAP authentication.
Both Kerberos and LDAP are using Active Directory as a backend to store 
principals (in many cases), so it's naturally to think this should work for 
LDAP too.
Fact that IMPALA-2660 works only for Kerberos and not for LDAP principals, 
breaks authentication in our organization.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 53593: HIVE-14089 complex type support in LLAP IO is broken

2016-11-09 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53593/#review155539
---




llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 


this should not be removed



ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
(line 210)


should be removed



storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
 (line 84)


stale comment


- Sergey Shelukhin


On Nov. 9, 2016, 2:34 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53593/
> ---
> 
> (Updated Nov. 9, 2016, 2:34 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  a27266f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  2fa68a9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  eb8ee6c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
>  c9b0a4d 
>   orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c347181 
>   orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 245e3ce 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java a9dbc3e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ccb39da 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 
> 4d09dcd 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> bcb54d6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
>  ebbdf8d 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 4405232 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 5cc3663 
>   ql/src/test/queries/clientpositive/vector_complex_all.q 91a7368 
>   ql/src/test/results/clientpositive/llap/vector_complex_all.q.out 08d49bc 
>   ql/src/test/results/clientpositive/vector_complex_all.q.out a54a371 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 76a1286 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
>  907181e 
> 
> Diff: https://reviews.apache.org/r/53593/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[GitHub] hive pull request #113: HIVE-15112 Implement Parquet vectorization reader fo...

2016-11-09 Thread winningsix 
GitHub user winningsix opened a pull request:

https://github.com/apache/hive/pull/113

HIVE-15112 Implement Parquet vectorization reader for Struct type

Patch includes:
1. support for struct type
2. UT refine

To be done:
QTest for struct type

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/winningsix/hive complex_types

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/113.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #113


commit 37f50c7629b5ef2a8fb6e9f63caaec6223abf308
Author: Ferdinand Xu 
Date:   2016-09-01T22:15:31Z

HIVE-14815: Support vectorization for Parquet

clean code and add qtest

Refine code

Clean code

Clean up code

Clean up

clean up code

Update qfile output files

Clean up code

Address comments

Avoid creating new HiveDecimalWritable object

Address more comments

Remove unused imports

Address further comments

Fix NPE

Fix for failed cases

commit 891b219838e4978f2eb4d41c0016214d44cc1bb7
Author: Ferdinand Xu 
Date:   2016-11-07T06:10:16Z

HIVE-15112: Implement Parquet vectorization reader for Complex types

commit 26e513a2ac67dcfb05875e6ad7ba07f158be9073
Author: Ferdinand Xu 
Date:   2016-11-09T19:49:46Z

Refactor UT




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Review Request 52800: HIVE-13966: DbNotificationListener: can loose DDL operation notifications

2016-11-09 Thread Mohit Sabharwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52800/
---

(Updated Nov. 10, 2016, 3:43 a.m.)


Review request for hive.


Changes
---

Incorp. review feedback


Bugs: HIVE-13966
https://issues.apache.org/jira/browse/HIVE-13966


Repository: hive-git


Description
---

Metadata event and associated notification should be committed in the same
JDO transaction.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
7ce29e10798730061d02fadb55423bea3b69c6ac 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 PRE-CREATION 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
 81ce67bdc8fdaf11ff4fec3f255ed0021a4752c7 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java
 af16f75e63c372c37bfd73567b777bba53f94db3 
  metastore/src/java/org/apache/hadoop/hive/metastore/AlterHandler.java 
dedd4497adfcc9d57090a943f6bb4f35ea87fa61 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
40b337a9e40ea04a37f108146853d2d1f42dcd29 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
60e462fd06a3f84d5b87cd335afb49768cb27562 

Diff: https://reviews.apache.org/r/52800/diff/


Testing
---

Enhanced TestDbNotificationListener


Thanks,

Mohit Sabharwal

Re: Review Request 52800: HIVE-13966: DbNotificationListener: can loose DDL operation notifications

2016-11-09 Thread Mohit Sabharwal 


> On Nov. 8, 2016, 4:13 p.m., Chaoyu Tang wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 790
> > 
> >
> > What kind of events need to be invoked in a separate JDO transactions? 
> > or did you mean the events that do not require a JDO transaction?

There are few scenarios where it may make sense to still use separate JDO 
transactions (and hence the old config), like:

(1) When the listener method invocation is expensive. For example, say an 
external authorization service has it's own listener implementation
that makes an RPC call to an (external) server. This call may block forever. To 
avoid rolling back original metadata from HMS db, user may want 
to do these in separate transactions -- with the understanding that the two may 
get inconsistent.

If, on the other hand, the listener is doing a relatively predictable operation 
like adding notification to a queue or persisting to the 
HMS db schema (like DbNotificationListener), it may be reasonable to expect 
that both happen in the same transaction. This is the motivation
for the new config.

(2) When the notification needs to be logged irrespective of whether the 
metadata event failed or succeeds. Here the notification acts more
like a audit.

(3) For backward compatibility.


> On Nov. 8, 2016, 4:13 p.m., Chaoyu Tang wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 794
> > 
> >
> > I wonder why we need a separate transactional event listener in Hive. 
> > Can it be deferred to listener implementation? For example, a listener 
> > which needs be transactional first checks if it is under a transaction 
> > context, joins (or enlists) itself to the transaction if it is. This way 
> > might also be applicable to the cases where multiple backends invovle (e.g. 
> > distributed transaction). It is just my thought, and may not necessarily be 
> > right or feasible.

Any implementation of MetaStoreEventListener interface can be configured so 
that it's invoked in the same 
transaction as the metadata event. This patch does not introduce any separate 
transactional event listener.

If user wants both the metadata event and notification to happen in the same 
JDO transaction, they 
configure hive.metastore.transactional.event.listeners, otherwise they 
configure hive.metastore.event.listeners.

This way the interface implementation does not need to be changed. 

Only the configuration needs to be changed.


> On Nov. 8, 2016, 4:13 p.m., Chaoyu Tang wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/AlterHandler.java, line 
> > 86
> > 
> >
> > I wonder if the API changes will cause some backward compatibility 
> > issue, if the HiveMetaStore is currently the only consumer of the 
> > AlterHandler, it should be fine.

That's a good point. I have created new versions of these methods. But adding 
new methods makes it backward compatibility as well, right ? I'm not sure there 
is a good solution here.


- Mohit


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52800/#review155256
---


On Nov. 7, 2016, 11:25 p.m., Mohit Sabharwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52800/
> ---
> 
> (Updated Nov. 7, 2016, 11:25 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-13966
> https://issues.apache.org/jira/browse/HIVE-13966
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Metadata event and associated notification should be committed in the same
> JDO transaction.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 80cd5ada060331797a603848e268c7d2a78a679c 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  PRE-CREATION 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
>  81ce67bdc8fdaf11ff4fec3f255ed0021a4752c7 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java
>  af16f75e63c372c37bfd73567b777bba53f94db3 
>   metastore/src/java/org/apache/hadoop/hive/metastore/AlterHandler.java 
> dedd4497adfcc9d57090a943f6bb4f35ea87fa61 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
> 40b337a9e40ea04a37f108146853d2d1f42dcd29 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> 60e462fd06a3f84d5b87cd335afb49768cb27562 
> 
> Diff:

[jira] [Created] (HIVE-15175) NOT IN condition is not handled correctly with predicate push down

2016-11-09 Thread Teruyoshi Zenmyo (JIRA) 
Teruyoshi Zenmyo created HIVE-15175:
---

 Summary: NOT IN condition is not handled correctly with predicate 
push down
 Key: HIVE-15175
 URL: https://issues.apache.org/jira/browse/HIVE-15175
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.1.0
Reporter: Teruyoshi Zenmyo


With predicate pushdown enabled, NOT IN conditions are treated as FALSE.

Example (pkey is a partition key)
{code}
hive> select * from test;
OK
test.keytest.valtest.pkey
a   1   a
b   2   a
c   3   a
a   1   b
b   2   b
c   3   b
Time taken: 0.171 seconds, Fetched: 6 row(s)
hive> set hive.optimize.ppd=false;
hive> select * from test where not pkey in ('a');
OK
test.keytest.valtest.pkey
a   1   b
b   2   b
c   3   b
Time taken: 0.237 seconds, Fetched: 3 row(s)
hive> set hive.optimize.ppd=true;
hive> select * from test where not pkey in ('a');
OK
test.keytest.valtest.pkey
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)