[jira] [Created] (HIVE-17738) CommitTransaction/rollbackTransaction may throw exceptions

2017-10-06 Thread Alexander Kolbasov (JIRA) 
Alexander Kolbasov created HIVE-17738:
-

 Summary: CommitTransaction/rollbackTransaction may throw exceptions
 Key: HIVE-17738
 URL: https://issues.apache.org/jira/browse/HIVE-17738
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.3.0, 3.0.0
Reporter: Alexander Kolbasov


The code in ObjectStore assumes that commitTransaction/rollbackTransaction 
never throws exceptions when, in fact, they do. As a result all the callers 
down the chain are not doing anything with these which causes potential 
problems.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17737) ObjectStore.getNotificationEventsCount may cause NPE

2017-10-06 Thread Alexander Kolbasov (JIRA) 
Alexander Kolbasov created HIVE-17737:
-

 Summary: ObjectStore.getNotificationEventsCount may cause NPE
 Key: HIVE-17737
 URL: https://issues.apache.org/jira/browse/HIVE-17737
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.3.0, 3.0.0
Reporter: Alexander Kolbasov


In ObjectStore.getNotificationEventsCount():

{code}
 public NotificationEventsCountResponse 
getNotificationEventsCount(NotificationEventsCountRequest rqst) {
Long result = 0L;
try {
  openTransaction();
  long fromEventId = rqst.getFromEventId();
  String inputDbName = rqst.getDbName();
  String queryStr = "select count(eventId) from " + 
MNotificationLog.class.getName()
+ " where eventId > fromEventId && dbName == inputDbName";
  query = pm.newQuery(queryStr);
  query.declareParameters("java.lang.Long fromEventId, java.lang.String 
inputDbName");
  result = (Long) query.execute(fromEventId, inputDbName); // <- Here
  commited = commitTransaction();
  return new NotificationEventsCountResponse(result.longValue());
}
  }
{code}

It is possible that query.execute will return null in which case 
rsult.longValue() may throw NPE.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17736) ObjectStore transaction handling can be simplified

2017-10-06 Thread Alexander Kolbasov (JIRA) 
Alexander Kolbasov created HIVE-17736:
-

 Summary: ObjectStore transaction handling can be simplified
 Key: HIVE-17736
 URL: https://issues.apache.org/jira/browse/HIVE-17736
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Affects Versions: 3.0.0
Reporter: Alexander Kolbasov
Assignee: Alexander Kolbasov


There are many places in ObjectStore that do something like this:

{code}
boolean commited = false;
   try {
  openTransaction();
  commited = commitTransaction();
} finally {
  if (!commited) {
rollbackTransaction();
  }
}
{code}

We can simplify this in two ways:

1) Create a wrapper that calls given piece of code inside the block of code 
above. This is similar to TransactionManager in Sentry.

2) Create a special auto-closeable object that does the check and rollback on 
close.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17735) ObjectStore.addNotificationEvent is leaking queries

2017-10-06 Thread Alexander Kolbasov (JIRA) 
Alexander Kolbasov created HIVE-17735:
-

 Summary: ObjectStore.addNotificationEvent is leaking queries
 Key: HIVE-17735
 URL: https://issues.apache.org/jira/browse/HIVE-17735
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Alexander Kolbasov
Assignee: Alexander Kolbasov


In ObjectStore.addNotificationEvent():

{code}
  Query objectQuery = pm.newQuery(MNotificationNextId.class);
  Collection ids = (Collection) objectQuery.execute();
{code}

The query is never closed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: does anyone care about list bucketing stored as directories?

2017-10-06 Thread Sergey Shelukhin 
Looks like nobody does… I’ll file a ticket to remove it shortly.

From: Sergey Shelukhin >
Date: Tuesday, October 3, 2017 at 12:59
To: "u...@hive.apache.org" 
>, 
"dev@hive.apache.org" 
>
Subject: does anyone care about list bucketing stored as directories?

1) There seem to be some bugs and limitations in LB (e.g. incorrect cleanup - 
https://issues.apache.org/jira/browse/HIVE-14886) and nobody appears to as much 
as watch JIRAs ;) Does anyone actually use this stuff? Should we nuke it in 
3.0, and by 3.0 I mean I’ll remove it from master in a few weeks? :)

2) I actually wonder, on top of the same SQL syntax, wouldn’t it be much easier 
to add logic to partitioning to write skew values into partitions and non-skew 
values into a new type of default partition? It won’t affect nearly as many low 
level codepaths in obscure and unobvious ways, instead keeping all the logic in 
metastore and split generation, and would integrate with Hive features like PPD 
automatically.
Esp. if we are ok with the same limitations - e.g. if you add a new skew value 
right now, I’m not sure what happens to the rows with that value already 
sitting in the non-skew directories, but I don’t expect anything reasonable...

[jira] [Created] (HIVE-17734) Add trigger type to WM_TRIGGER table

2017-10-06 Thread Prasanth Jayachandran (JIRA) 
Prasanth Jayachandran created HIVE-17734:


 Summary: Add trigger type to WM_TRIGGER table
 Key: HIVE-17734
 URL: https://issues.apache.org/jira/browse/HIVE-17734
 Project: Hive
  Issue Type: Sub-task
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran


Following up with HIVE-17566, it will be useful to add a 'type' to WM_TRIGGER 
table to represent the type of the trigger. HIVE-17508 added 'execution' 
triggers but there could be other triggers that can be added later. This will 
avoid fetching all triggers and filtering at client side.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] hive pull request #258: HIVE-17733

2017-10-06 Thread alanfgates 
GitHub user alanfgates opened a pull request:

https://github.com/apache/hive/pull/258

HIVE-17733

Moved RawStore, implementations, and some classes they depend on.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alanfgates/hive hive17733

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/258.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #258


commit c6221d3305e0be74b63ec13902bff90cb355ccd1
Author: Alan Gates 
Date:   2017-08-24T00:04:18Z

Moved Filter.g and ExpressionTree.  As part of this I had to move 
PARTITION_DATE_FORMAT from HiveMetaStore to MetaStoreUtils.  Without this the 
move would have had to wait until I moved HiveMetaStore.  But this caused a 
cascade of other things to have to wait, so that I would have ended up with one 
huge patch that moves everything.

commit b7a555b27e993250ec2f7c53916d8aaf4d56e495
Author: Alan Gates 
Date:   2017-08-24T22:53:20Z

Moved NDV classes in prep for moving Stats mergers.

commit fa7602a290f5691242319cf74fbd021808cc088b
Author: Alan Gates 
Date:   2017-08-24T23:12:36Z

Moved ColumnStatsMerger and implementors.

commit 7089ee49ee579e0ef034d73543e96db207c871f1
Author: Alan Gates 
Date:   2017-08-25T00:08:34Z

Moved MetaStoreDirectSql, StatObjectConverter, and ColumnStatsAggregator 
and implementations.  Combined all of these together because there was a 
circular dependence.

commit e1d74d08237e7ddcdbb64a90915771ccd5d20199
Author: Alan Gates 
Date:   2017-08-25T23:37:33Z

Moved PartFilterExprUtil

commit fd6b77ee78dece36dd239510dc4c9623cd81370c
Author: Alan Gates 
Date:   2017-08-31T00:00:05Z

Moved stats data inspectors

commit 9fb4cede04a302f90641aec94a1f91c91a120b79
Author: Alan Gates 
Date:   2017-08-26T01:36:15Z

Moved RawStore and friends.

commit 416d88ae19c5c51bc97ba5c12d3dd03bc58e36f7
Author: Alan Gates 
Date:   2017-08-28T22:16:02Z

Moved RawStoreProxy and associated test.

commit 6aac3bb41b5923e179b7c508a0c0373e2c217f6f
Author: Alan Gates 
Date:   2017-10-06T21:51:23Z

Changes to make it compile after rebase.  Also split TestObjectStore so 
that I could move the majority of it to standalone-metastore (since ObjectStore 
is now there) but left a couple of tests in metastore since they reference 
EventMessage which hasn't moved yet.




---

[jira] [Created] (HIVE-17733) Move RawStore to standalone metastore

2017-10-06 Thread Alan Gates (JIRA) 
Alan Gates created HIVE-17733:
-

 Summary: Move RawStore to standalone metastore
 Key: HIVE-17733
 URL: https://issues.apache.org/jira/browse/HIVE-17733
 Project: Hive
  Issue Type: Sub-task
  Components: Metastore
Reporter: Alan Gates
Assignee: Alan Gates


This includes moving implementations of RawStore (like ObjectStore), 
MetastoreDirectSql, and stats related classes like ColumnStatsAggregator and 
the NDV classes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17732) org.apache.hive.hcatalog.data.JsonSerDe.java

2017-10-06 Thread BELUGA BEHR (JIRA) 
BELUGA BEHR created HIVE-17732:
--

 Summary: org.apache.hive.hcatalog.data.JsonSerDe.java
 Key: HIVE-17732
 URL: https://issues.apache.org/jira/browse/HIVE-17732
 Project: Hive
  Issue Type: Improvement
  Components: Serializers/Deserializers
Affects Versions: 3.0.0
Reporter: BELUGA BEHR
Priority: Trivial


Some simple improvements for org.apache.hive.hcatalog.data.JsonSerDe

Remove superfluous logging, cut down on object instantiation 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17731) add a backward compat option for external users to HIVE-11985

2017-10-06 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-17731:
---

 Summary: add a backward compat option for external users to 
HIVE-11985
 Key: HIVE-17731
 URL: https://issues.apache.org/jira/browse/HIVE-17731
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


See HIVE-11985.
Some external callers (e.g. Presto) do not appear to process types from 
deserializer correctly, relying on DB types. Ideally, it should be resolved via 
HIVE-17714, hiding the custom SerDe logic from users.
For now we can add a backward compatibility config for such cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17730) Queries can be closed automatically

2017-10-06 Thread Alexander Kolbasov (JIRA) 
Alexander Kolbasov created HIVE-17730:
-

 Summary: Queries can be closed automatically
 Key: HIVE-17730
 URL: https://issues.apache.org/jira/browse/HIVE-17730
 Project: Hive
  Issue Type: Bug
Reporter: Alexander Kolbasov


HIVE-16213 made QueryWrapper AutoCloseable, but queries are still closed 
manually and not by using try-with-resource. And now Query itself is auto 
closeable, so we don't need the wrapper at all.

So we should get rid of QueryWrapper and use try-with-resource to create 
queries.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 62816: HIVE-17729 Add Database & Explain related blobstore tests

2017-10-06 Thread Rentao Wu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62816/
---

Review request for hive and Sergio Pena.


Bugs: HIVE-17729
https://issues.apache.org/jira/browse/HIVE-17729


Repository: hive-git


Description
---

HIVE-17729 Add Database & Explain related blobstore tests


This patch introduces the following regression tests into the hive-blobstore 
qtest module:
create_database.q -> tests tables with location inherited from database
multiple_db.q -> tests query spanning multiple databases
explain.q -> tests EXPLAIN INSERT OVERWRITE command


Diffs
-

  data/files/single_int.txt PRE-CREATION 
  itests/hive-blobstore/src/test/queries/clientpositive/create_database.q 
PRE-CREATION 
  itests/hive-blobstore/src/test/queries/clientpositive/explain.q PRE-CREATION 
  itests/hive-blobstore/src/test/queries/clientpositive/multiple_db.q 
PRE-CREATION 
  itests/hive-blobstore/src/test/results/clientpositive/create_database.q.out 
PRE-CREATION 
  itests/hive-blobstore/src/test/results/clientpositive/explain.q.out 
PRE-CREATION 
  itests/hive-blobstore/src/test/results/clientpositive/multiple_db.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62816/diff/1/


Testing
---

under /itests/hive-blobstore/

$ mvn clean test -Dtest=TestBlobstoreCliDriver 
-Dqfile=create_database.q,explain.q,multiple_db.q

---
 T E S T S
---
Running org.apache.hadoop.hive.cli.TestBlobstoreCliDriver
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.4 sec - in 
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver

Results :

Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 01:12 min
[INFO] Finished at: 2017-10-06T11:29:41-07:00
[INFO] Final Memory: 88M/1003M
[INFO] 


Thanks,

Rentao Wu

Re: Review Request 62453: HIVE-17371 : Move tokenstores to metastore module

2017-10-06 Thread Vihang Karajgaonkar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62453/
---

(Updated Oct. 6, 2017, 6:33 p.m.)


Review request for hive, Aihua Xu, Alan Gates, and Thejas Nair.


Changes
---

rebased and resolved conflicts.


Bugs: HIVE-17371
https://issues.apache.org/jira/browse/HIVE-17371


Repository: hive-git


Description
---

HIVE-17371 : Move tokenstores to metastore module


Diffs (updated)
-

  beeline/src/test/org/apache/hive/beeline/ProxyAuthTest.java 
318857dddbff22615fcf660988b7dc9f25fbec65 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 
81804cfb2e309387051fb1762ff1e5959b9f9ce2 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/Security.java 
9b621952e63ab43a0a907e97d451a0d8bb190940 
  
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java
 f4c4b76c8127c4be02eb1cfb0622ef7a07b08470 
  
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestHiveAuthFactory.java
 e3a51909df14d90a538426163f332a6378f35f40 
  
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithDBTokenStore.java
 d690aaa673a50785561750f4f461ec867b6f0abc 
  
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java
 256262d6e488237950bb42b4b93e72f1fe82efb3 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/thrift/TestDBTokenStore.java
 4bfa22419cb1ea5abe56773c1c76186b01996e03 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/thrift/TestZooKeeperTokenStore.java
 7800416a60239246e96497229d592736f2a9d79d 
  jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 
a9a4f2cf66f649c67e09a53f73e04951c1bb9e57 
  jdbc/src/java/org/apache/hive/jdbc/Utils.java 
574fb7e1d2c46d3e51278143b74ca853bbe7769a 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
5617e1ce5748c85831c570d3e4c09fd6cdc17bc8 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
6bc45b6c7b4e209e2214f0aafddc062660b23d77 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
a147a2590d6de1fe161c7b02f043f179243cf83c 
  service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java 
0f4a4d8d545cfaf6869b706db828e768f1d53d4d 
  service/src/java/org/apache/hive/service/auth/HttpAuthUtils.java 
8b5661a0508cf0b68d7040e3de85f4694cb9c3c5 
  service/src/java/org/apache/hive/service/auth/KerberosSaslHelper.java 
ea2c689a4fb2e2ba6867b276044cbca0ac6899d3 
  
service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
 00a7e742cabd2fc36faa464b29250b5a6a9b1159 
  service/src/java/org/apache/hive/service/cli/session/SessionUtils.java 
PRE-CREATION 
  
shims/0.23/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge23.java
 5e21c9f187977a8bf556f76cbdc0c49f3a531641 
  shims/common/src/main/java/org/apache/hadoop/hive/shims/ShimLoader.java 
f15e7ff53fa340ab384db6071146b95529022fed 
  shims/common/src/main/java/org/apache/hadoop/hive/shims/Utils.java 
3c93186082ae363497dc94ca1303db4f746fd8b5 
  shims/common/src/main/java/org/apache/hadoop/hive/thrift/DBTokenStore.java 
d6dc0796e77591d3afca8dbd29c3aa0eff255dd0 
  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier.java
 4ca3c0beefa6944407f4b1d0aebee79e86b4ab84 
  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java
 5299e18743aa45c539287b335f95e8ce8df0fc35 
  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector.java
 f6e2420eb20f58a22f636976fd53a98c294ddfe2 
  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenStore.java
 867b4ed98859f681770fccfe7478ab519c823924 
  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge.java
 2d39bea4be27d7c99285f53b8288d141b5acedc0 
  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/HiveDelegationTokenManager.java
 b3e4a7608282be603e79d1d101679e239a5219b0 
  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/MemoryTokenStore.java 
9d837b8fa19ed53546c4a95944484cc1f06a21e6 
  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/TokenStoreDelegationTokenSecretManager.java
 4719b85e8d2f9045ae61e0e86c6624b865aa2653 
  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/ZooKeeperTokenStore.java
 885ec56d543e1c46e74dc148c73f249803b7604c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/DelegationTokenStore.java
 0cafeff89614c94ba551a5f1ba9c4d892ee5720a 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/MemoryTokenStore.java
 c484cd3132d08c8c0493fa98e79fa449e84ef9b8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/MetastoreDelegationTokenManager.java
 2b0110fe0badcb571144067539031fb6ac81276b

[jira] [Created] (HIVE-17729) Add Database & Explain related blobstore tests

2017-10-06 Thread Rentao Wu (JIRA) 
Rentao Wu created HIVE-17729:


 Summary: Add Database & Explain related blobstore tests
 Key: HIVE-17729
 URL: https://issues.apache.org/jira/browse/HIVE-17729
 Project: Hive
  Issue Type: Test
  Components: Tests
Reporter: Rentao Wu
Assignee: Rentao Wu


This patch introduces the following regression tests into the hive-blobstore 
qtest module:
* create_database.q  -> tests tables with location inherited from database
* multiple_db.q  -> tests query spanning multiple databases
* explain.q -> tests EXPLAIN INSERT OVERWRITE command
 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] hive pull request #257: HIVE-17728: TestHCatClient should configure hive.met...

2017-10-06 Thread sankarh 
GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/257

HIVE-17728: TestHCatClient should configure 
hive.metastore.transactional.event.listeners as per recommendation.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-17728

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/257.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #257


commit e5be3c09e18398824b74c7294f1d5af881a83637
Author: Sankar Hariappan 
Date:   2017-10-06T16:31:43Z

HIVE-17728: TestHCatClient should configure 
hive.metastore.transactional.event.listeners as per recommendation.




---

[jira] [Created] (HIVE-17728) TestHCatClient should configure hive.metastore.transactional.event.listeners as per recommendation.

2017-10-06 Thread Sankar Hariappan (JIRA) 
Sankar Hariappan created HIVE-17728:
---

 Summary: TestHCatClient should configure 
hive.metastore.transactional.event.listeners as per recommendation.
 Key: HIVE-17728
 URL: https://issues.apache.org/jira/browse/HIVE-17728
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 3.0.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan
 Fix For: 3.0.0


Currently, TestHCatClient.java uses hive.metastore.event.listeners to enable 
notification events logging. But the recommended configuration for the same is 
hive.metastore.transactional.event.listeners. 
So, need to update the same.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17727) HoS Queries Print "Starting task [Stage-x:MAPRED] in serial mode"

2017-10-06 Thread Sahil Takiar (JIRA) 
Sahil Takiar created HIVE-17727:
---

 Summary: HoS Queries Print "Starting task [Stage-x:MAPRED] in 
serial mode"
 Key: HIVE-17727
 URL: https://issues.apache.org/jira/browse/HIVE-17727
 Project: Hive
  Issue Type: Improvement
  Components: Spark
Reporter: Sahil Takiar


Whenever a HoS query is run something like "Starting task [Stage-3:MAPRED] in 
serial mode" in printed out for each {{SparkTask}}, which is confusing because 
this isn't a MAPRED job. We should change {{StageType}} to include a {{SPARK}} 
type.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 62810: HIVE-17300 WebUI query plan graphs

2017-10-06 Thread Peter Vary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62810/
---

Review request for hive, Karen Coppage, Xuefu Zhang, and Xuefu Zhang.


Bugs: HIVE-17300
https://issues.apache.org/jira/browse/HIVE-17300


Repository: hive-git


Description
---

Moving the review here, since could not change Karen's original one


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/LogUtils.java 0a3e0c7 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d2afc2c 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1943c6d 
  ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 4b60514 
  ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java bf6cb91 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 
3c07197 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 41a1ef1 
  service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon ff7476e 
  service/src/resources/hive-webapps/static/css/query-plan-graph.css 
PRE-CREATION 
  service/src/resources/hive-webapps/static/js/query-plan-graph.js PRE-CREATION 
  service/src/resources/hive-webapps/static/js/vis.min.js PRE-CREATION 


Diff: https://reviews.apache.org/r/62810/diff/1/


Testing
---


Thanks,

Peter Vary

Re: Review Request 61663: WebUI query plan graphs

2017-10-06 Thread Peter Vary 


> On Aug. 30, 2017, 4:47 a.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java
> > Lines 127 (patched)
> > 
> >
> > What happens if this is a map-only task?

In the next patch I split the if statement


> On Aug. 30, 2017, 4:47 a.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java
> > Lines 132 (patched)
> > 
> >
> > It might be better if the null check is put in getCountersJson() method.

I think we should keep this, as it is - it is better to omit the "counter" 
element, than adding an empty one? What do you think?


> On Aug. 30, 2017, 4:47 a.m., Xuefu Zhang wrote:
> > service/src/resources/hive-webapps/static/css/query-plan-graph.css
> > Lines 1 (patched)
> > 
> >
> > Apache license header if possible.

Done


> On Aug. 30, 2017, 4:47 a.m., Xuefu Zhang wrote:
> > service/src/resources/hive-webapps/static/js/query-plan-graph.js
> > Lines 1 (patched)
> > 
> >
> > I think we need apache license header.

Done


> On Aug. 30, 2017, 4:47 a.m., Xuefu Zhang wrote:
> > service/src/resources/hive-webapps/static/js/vis.min.js
> > Lines 1 (patched)
> > 
> >
> > Apache license header.

Done


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61663/#review184121
---


On Aug. 16, 2017, 1:55 p.m., Karen Coppage wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61663/
> ---
> 
> (Updated Aug. 16, 2017, 1:55 p.m.)
> 
> 
> Review request for hive, Peter Vary and Xuefu Zhang.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> I’m working on a feature of the Hive WebUI Query Plan tab that would provide 
> the option to display the query plan as a nice graph (scroll down for 
> screenshots). If you click on one of the graph’s stages, the plan for that 
> stage appears as text below.
> Stages are color-coded if they have a status (Success, Error, Running), and 
> the rest are grayed out. Coloring is based on status already available in the 
> WebUI, under the Stages tab.
> There is an additional option to display stats for MapReduce tasks. This 
> includes the job’s ID, tracking URL (where the logs are found), and mapper 
> and reducer numbers/progress, among other info.
> The library I’m using for the graph is called vis.js (http://visjs.org/). It 
> has an Apache license, and the only necessary file to be included from this 
> library is about 700 KB.
> I tried to keep server-side changes minimal, and graph generation is taken 
> care of by the client. Plans with more than a given number of stages 
> (default: 25) won't be displayed in order to preserve resources.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/LogUtils.java 0a3e0c7201 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3c158a6692 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4e7c80f184 
>   ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 4b6051485e 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java bf6cb91745 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 
> 3c0719717c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 1bd4db7805 
>   service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon ff7476ee02 
>   service/src/resources/hive-webapps/static/css/query-plan-graph.css 
> PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/query-plan-graph.js 
> PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/vis.min.js PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/61663/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Karen Coppage
> 
>

[jira] [Created] (HIVE-17726) Using exists may lead to incorrect results

2017-10-06 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-17726:
---

 Summary: Using exists may lead to incorrect results
 Key: HIVE-17726
 URL: https://issues.apache.org/jira/browse/HIVE-17726
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Zoltan Haindrich


{code}
drop table if exists tx1;
create table tx1 (a integer,b integer);
insert into tx1 values  (1, 1),
(1, 2),
(1, 3);

select count(*) as result,3 as expected from tx1 u
where exists (select * from tx1 v where u.a=v.a and u.b <> v.b);

select count(*) as result,3 as expected from tx1 u
where exists (select * from tx1 v where u.a=v.a and u.b <> v.b limit 1);
{code}

current results are 6 and 2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17725) Fix misnamed tests which are not run during precommit runs.

2017-10-06 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-17725:
---

 Summary: Fix misnamed tests which are not run during precommit 
runs. 
 Key: HIVE-17725
 URL: https://issues.apache.org/jira/browse/HIVE-17725
 Project: Hive
  Issue Type: Bug
  Components: Test
Reporter: Zoltan Haindrich



I've just seen a testfailure for jdk9; but the test is not even executed during 
precommit runs:
{{TaskTrackerTest}}
I think the test classes name should match the {{**/Test*}} to be executed 
during test runs...however there seems to be quite a few... {{find . -name 
'*Test.java'}} returns a few abstracts ; but there are real test ; which are 
just misnamed...




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62782: HIVE-17706 Add a possibility to run the BeeLine tests on the default database

2017-10-06 Thread Peter Vary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62782/
---

(Updated Oct. 6, 2017, 10:43 a.m.)


Review request for hive and Barna Zsombor Klara.


Changes
---

Rebased the patch


Bugs: HIVE-17706
https://issues.apache.org/jira/browse/HIVE-17706


Repository: hive-git


Description
---

- Added a cleanup method which removed unkonwn databases, tables and view. This 
makes it is possible to run the tests sequentially using 'default' database.
- Renamed the test.rewrite.source.tables parameter to test.beeline.run.parallel 
so it makes more sense.
- Copied some masking from QTestUtil
- Enhanced the table name rewriting regexp a little, so mixed case tablenames 
are kept as it is
- In the QFile made it possible to not call create/drop database command, if 
not needed.

I would value any advice where the cleanup logic should be kept.
- I this solution when the tests are parallel then the cleanup is in 
QFileBeeLineClient (create/drop database), when the tests are sequential then 
the cleanup is in the CoreBeeLineDriver.

Would it be a good idea to move every cleanup related stuff to 
CoreBeeLineDriver? Like:
- QFileBeeLineClient.beforeExecute
- QFileBeeLineClient.afterExecute
Both of these are need QFile specific info, and an existing BeeLineClient, 
which we currently do not have in hand in the CoreBeeLineDriver, and more 
refactoring is needed

Or would it be a good idea to move every cleanup related stuff to 
QFileBeeLineClient? Like:
- CoreBeeLineDriver.runCleanup
This is really cleanup stuff

Or we should leave as it is :)

Any comments are welcome.
Thanks,
Peter


Diffs (updated)
-

  
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java
 1fdce17 
  itests/util/src/main/java/org/apache/hive/beeline/QFile.java 38b0d91 
  itests/util/src/main/java/org/apache/hive/beeline/QFileBeeLineClient.java 
2f91834 


Diff: https://reviews.apache.org/r/62782/diff/3/

Changes: https://reviews.apache.org/r/62782/diff/2-3/


Testing
---

Run the tests manually with, and without the test.beeline.run.parallel parameter


Thanks,

Peter Vary

[jira] [Created] (HIVE-17724) Cast from string to float gives strange result

2017-10-06 Thread David Scarlatti (JIRA) 
David Scarlatti created HIVE-17724:
--

 Summary: Cast from string to float gives strange result
 Key: HIVE-17724
 URL: https://issues.apache.org/jira/browse/HIVE-17724
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.1.0
 Environment: Hive 1.1.0-cdh5.10.0
Subversion 
file:///data0/jenkins/workspace/generic-package-ubuntu64-14-04/CDH5.10.0-Packaging-Hive-2017-01-20_11-52-43/hive-1.1.0+cdh5.10.0+859-1.cdh5.10.0.p0.71~trusty
 -r Unknown

Reporter: David Scarlatti


this query:"select CAST ('1507284611' as int), CAST ('1507284611' as bigint),  
CAST ('1507284611' as float),   CAST ('1507284611' as double),   CAST 
('1507284611' as decimal)"
returns
1507284611  1507284611  1507284608  1507284611  1507284611

cast as float is the only one changing the numeric value.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17723) Update Accumulo drive q.out files

2017-10-06 Thread Peter Vary (JIRA) 
Peter Vary created HIVE-17723:
-

 Summary: Update Accumulo drive q.out files
 Key: HIVE-17723
 URL: https://issues.apache.org/jira/browse/HIVE-17723
 Project: Hive
  Issue Type: Bug
  Components: Test
Affects Versions: 3.0.0
Reporter: Peter Vary
Assignee: Peter Vary






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17722) Execution of selectDistinctStar.q breaks stats in optimize_nullscan.q

2017-10-06 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-17722:
---

 Summary: Execution of selectDistinctStar.q breaks stats in 
optimize_nullscan.q
 Key: HIVE-17722
 URL: https://issues.apache.org/jira/browse/HIVE-17722
 Project: Hive
  Issue Type: Bug
  Components: Test
Reporter: Zoltan Haindrich



{code}
M_OPTS+=" -q -T9 -Dmaven.surefire.plugin.version=2.20.1"
M_OPTS+=" -Pitests -DskipSparkTests"
M_OPTS+=" -Dtest=TestMiniLlapLocalCliDriver"
M_OPTS+=" -pl itests/qtest"
M_OPTS+=" install"

#fail
mvn $M_OPTS -Dqfile=selectDistinctStar.q,optimize_nullscan.q

#pass
mvn $M_OPTS -Dqfile=optimize_nullscan.q
mvn $M_OPTS -Dqfile=selectDistinctStar.q

{code}

my guess is that something have "happend" with the sacred src table...or that 
view might cause some trouble?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62782: HIVE-17706 Add a possibility to run the BeeLine tests on the default database

2017-10-06 Thread Barna Zsombor Klara 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62782/#review187240
---


Ship it!




Ship It!

- Barna Zsombor Klara


On Oct. 5, 2017, 4:39 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62782/
> ---
> 
> (Updated Oct. 5, 2017, 4:39 p.m.)
> 
> 
> Review request for hive and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-17706
> https://issues.apache.org/jira/browse/HIVE-17706
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> - Added a cleanup method which removed unkonwn databases, tables and view. 
> This makes it is possible to run the tests sequentially using 'default' 
> database.
> - Renamed the test.rewrite.source.tables parameter to 
> test.beeline.run.parallel so it makes more sense.
> - Copied some masking from QTestUtil
> - Enhanced the table name rewriting regexp a little, so mixed case tablenames 
> are kept as it is
> - In the QFile made it possible to not call create/drop database command, if 
> not needed.
> 
> I would value any advice where the cleanup logic should be kept.
> - I this solution when the tests are parallel then the cleanup is in 
> QFileBeeLineClient (create/drop database), when the tests are sequential then 
> the cleanup is in the CoreBeeLineDriver.
> 
> Would it be a good idea to move every cleanup related stuff to 
> CoreBeeLineDriver? Like:
> - QFileBeeLineClient.beforeExecute
> - QFileBeeLineClient.afterExecute
> Both of these are need QFile specific info, and an existing BeeLineClient, 
> which we currently do not have in hand in the CoreBeeLineDriver, and more 
> refactoring is needed
> 
> Or would it be a good idea to move every cleanup related stuff to 
> QFileBeeLineClient? Like:
> - CoreBeeLineDriver.runCleanup
> This is really cleanup stuff
> 
> Or we should leave as it is :)
> 
> Any comments are welcome.
> Thanks,
> Peter
> 
> 
> Diffs
> -
> 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java
>  9dfc253 
>   itests/util/src/main/java/org/apache/hive/beeline/QFile.java e70ac38 
>   itests/util/src/main/java/org/apache/hive/beeline/QFileBeeLineClient.java 
> 2f91834 
> 
> 
> Diff: https://reviews.apache.org/r/62782/diff/2/
> 
> 
> Testing
> ---
> 
> Run the tests manually with, and without the test.beeline.run.parallel 
> parameter
> 
> 
> Thanks,
> 
> Peter Vary
> 
>