Re: Review Request 65172: HIVE-18450

2018-01-19 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65172/
---

(Updated Jan. 20, 2018, 2:02 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-18450
https://issues.apache.org/jira/browse/HIVE-18450


Repository: hive-git


Description
---

HIVE-18450


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java 
b5c7be43e9c4d541599790389c36f1a3bcce5e9b 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveTableFunctionScan.java
 a416ca35c0ba4b703ca5384e71c6accc7c71ccfe 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
f639a403303b75729597b8c9ae06d7cc6416ef5f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/FromClauseParser.g 
b038e8185de5e0457ff02cdbf7372f03f6a2a9d9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
d0fe8ab322db7e5bfafe71b1dd3406cc1a9e27e9 
  ql/src/test/queries/clientpositive/tablevalues.q 
501f532eb7fe9bbe8dfe013b8af5883823b3e0c9 
  ql/src/test/results/clientpositive/tablevalues.q.out 
247c5647a3614bcf031586665752dfc69fd051ce 


Diff: https://reviews.apache.org/r/65172/diff/2/

Changes: https://reviews.apache.org/r/65172/diff/1-2/


Testing
---


Thanks,

Jesús Camacho Rodríguez

[jira] [Created] (HIVE-18503) MM/ACID tables: make tests that will never be compatible with acid use non-txn tables explicitly

2018-01-19 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-18503:
---

 Summary: MM/ACID tables: make tests that will never be compatible 
with acid use non-txn tables explicitly
 Key: HIVE-18503
 URL: https://issues.apache.org/jira/browse/HIVE-18503
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


Some tests do stuff that will simply never work with ACID tables, e.g. delete 
table files.
They should be changed to use external table, or explicitly set 
transactional=false





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65172: HIVE-18450

2018-01-19 Thread Jesús Camacho Rodríguez 


> On Jan. 20, 2018, 1:01 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
> > Lines 2701 (patched)
> > 
> >
> > Whats PTF node? Is it over clause? If so, can we add some tests for it.

This is similar to what I found in SELECT clause. I have added a test.


> On Jan. 20, 2018, 1:01 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
> > Lines 2718-2723 (patched)
> > 
> >
> > Can be written with lambda syntax. More readably.

Nice! I need to get used to it :)


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65172/#review195863
---


On Jan. 15, 2018, 11:52 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65172/
> ---
> 
> (Updated Jan. 15, 2018, 11:52 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-18450
> https://issues.apache.org/jira/browse/HIVE-18450
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18450
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java 
> 69a750bb3c88f976396f5306d2d39d305754b7a5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveTableFunctionScan.java
>  bf4896d61d594dfc076611b4a9c422fee8c18383 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 572b95f9603ba7570734cff8172dfe6a49b9b37d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/FromClauseParser.g 
> b038e8185de5e0457ff02cdbf7372f03f6a2a9d9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> c558356a4e9cca628e7590e48c0d6ec503bc594e 
>   ql/src/test/queries/clientpositive/tablevalues.q 
> 501f532eb7fe9bbe8dfe013b8af5883823b3e0c9 
>   ql/src/test/results/clientpositive/tablevalues.q.out 
> 247c5647a3614bcf031586665752dfc69fd051ce 
> 
> 
> Diff: https://reviews.apache.org/r/65172/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 65172: HIVE-18450

2018-01-19 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65172/#review195863
---




ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java
Lines 63 (patched)


More readable name: CBOSupportedLateralViews and then flip meaning of 
boolean.



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Lines 880 (patched)


Better name: CBOsupportedLateralView ?



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Lines 2701 (patched)


Whats PTF node? Is it over clause? If so, can we add some tests for it.



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Lines 2718-2723 (patched)


Can be written with lambda syntax. More readably.



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Lines 2734 (patched)


with instead of and in sentence ?


- Ashutosh Chauhan


On Jan. 15, 2018, 11:52 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65172/
> ---
> 
> (Updated Jan. 15, 2018, 11:52 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-18450
> https://issues.apache.org/jira/browse/HIVE-18450
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18450
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java 
> 69a750bb3c88f976396f5306d2d39d305754b7a5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveTableFunctionScan.java
>  bf4896d61d594dfc076611b4a9c422fee8c18383 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 572b95f9603ba7570734cff8172dfe6a49b9b37d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/FromClauseParser.g 
> b038e8185de5e0457ff02cdbf7372f03f6a2a9d9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> c558356a4e9cca628e7590e48c0d6ec503bc594e 
>   ql/src/test/queries/clientpositive/tablevalues.q 
> 501f532eb7fe9bbe8dfe013b8af5883823b3e0c9 
>   ql/src/test/results/clientpositive/tablevalues.q.out 
> 247c5647a3614bcf031586665752dfc69fd051ce 
> 
> 
> Diff: https://reviews.apache.org/r/65172/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Review Request 65250: HIVE-18387

2018-01-19 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65250/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-18387
https://issues.apache.org/jira/browse/HIVE-18387


Repository: hive-git


Description
---

HIVE-18387


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/Constants.java 
4e0c50a66256271507a377f8f7fa4ab7cb9dc279 
  ql/src/java/org/apache/hadoop/hive/ql/QueryLifeTimeHookRunner.java 
53d716bceb98c2ced3a3ba3f0cd607766447dfd9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
96fc33016b469980421d3ba25bb9a4e47559ebca 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewUpdateRegistryTask.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewUpdateRegistryWork.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 
85cef8664674db72cd69929d4ad96f1bd85279da 
  
ql/src/java/org/apache/hadoop/hive/ql/hooks/MaterializedViewRegistryUpdateHook.java
 e886399d5371e4b8d9692811998f6cc8ca3c2b7b 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
923372d5b6da42446997051d0758e9aab4881e2e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
d0fe8ab322db7e5bfafe71b1dd3406cc1a9e27e9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 
0c1c4e09d67f25c307e693bc220015f8b47f7ce1 
  ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java 
9b45f430261331a3d3af2fd1706d5892a76d84e4 
  ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java 
b48379013d74c56df245bb9e292e45bb298367da 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite_4.q 
efc65c4061c608fac5ba308a9a8238aca443555f 
  ql/src/test/results/clientpositive/materialized_view_create.q.out 
89523042a9acac42dbfe865cc6f091ae6aea4a6e 
  ql/src/test/results/clientpositive/materialized_view_create_rewrite.q.out 
95b1d61f94dd54a5419ca16d3ec5ec073a50428e 
  ql/src/test/results/clientpositive/materialized_view_create_rewrite_3.q.out 
0d8d238e8b43fdf453f3c11cd1cdd0f1aa8764bc 
  ql/src/test/results/clientpositive/materialized_view_create_rewrite_4.q.out 
8ab151718668daa8915575ffed5845d62e15b775 
  
ql/src/test/results/clientpositive/materialized_view_create_rewrite_multi_db.q.out
 d7ee468b49af904da93a74c86f0898c310970cab 
  ql/src/test/results/clientpositive/materialized_view_describe.q.out 
efd60d4345d68fe627f11ea8fd4b82ea5404e1eb 


Diff: https://reviews.apache.org/r/65250/diff/1/


Testing
---


Thanks,

Jesús Camacho Rodríguez

[jira] [Created] (HIVE-18502) base_-9223372036854775808 is confusing

2018-01-19 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-18502:
-

 Summary: base_-9223372036854775808 is confusing
 Key: HIVE-18502
 URL: https://issues.apache.org/jira/browse/HIVE-18502
 Project: Hive
  Issue Type: Improvement
Affects Versions: 0.13.0
Reporter: Eugene Koifman


start with no acid table

write some data

Alter Table to acid table

run major compaction

It will create base_-9223372036854775808  (if there are no delta dirs)

Nothing wrong with it but confuses users.  Should probably make it base_0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18501) Typo in beeline code

2018-01-19 Thread Naveen Gangam (JIRA) 
Naveen Gangam created HIVE-18501:


 Summary: Typo in beeline code
 Key: HIVE-18501
 URL: https://issues.apache.org/jira/browse/HIVE-18501
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Affects Versions: 3.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


[https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L744]

the string literal used here should be "silent", not "slient". There is no 
functional bug here just a silly typo.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Switched PreCommit-Admin over to Apache Yetus

2018-01-19 Thread Allen Wittenauer 

> On Jan 19, 2018, at 11:15 AM, Eric Wohlstadter  wrote:
> Is it possible for you to check whether Tez has been affected?
> https://issues.apache.org/jira/projects/TEZ/

There weren't any (that I saw at least) in the queue when I fired off 
of the new build, but Tez does use precommit-admin.

> For the last few days my patches aren't triggering any submission hook.
> Here is an example: https://issues.apache.org/jira/browse/TEZ-3888

They actually are triggering. (Here's TEZ-3888: 
https://builds.apache.org/view/PreCommit%20Builds/job/PreCommit-TEZ-Build/2715/ 
) The problem is that all of the old, deprecated features that Tez' test-patch 
depends upon have been removed.

I'd obviously recommend switching to Yetus' test-patch (esp 0.7.0 after 
it gets released hopefully by the end of the week since it fixes another bug 
with the new JIRA install), but I'm obviously a bit biased there. ;)

> In exchange, I can offer a Sharks NHL fidget spinner.

lol

[jira] [Created] (HIVE-18500) annoying exceptions from LLAP Jmx view in the logs

2018-01-19 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-18500:
---

 Summary: annoying exceptions from LLAP Jmx view in the logs
 Key: HIVE-18500
 URL: https://issues.apache.org/jira/browse/HIVE-18500
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


This can show for multiple things; should be handled better 
{noformat}
 - /jmx ()] org.apache.hive.http.JMXJsonServlet: getting attribute 
UsageThreshold of java.lang:type=MemoryPool,name=G1 Survivor Space threw an 
exception
javax.management.RuntimeMBeanException: 
java.lang.UnsupportedOperationException: Usage threshold is not supported
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
 ~[?:1.8.0_25]
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
 ~[?:1.8.0_25]
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
 ~[?:1.8.0_25]
at 
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678) 
~[?:1.8.0_25]
at 
org.apache.hive.http.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:322) 
~[hive-exec-2.1.0.2.6.3.0-235.jar:2.1.0.2.6.3.0-235]
at 
org.apache.hive.http.JMXJsonServlet.listBeans(JMXJsonServlet.java:300) 
~[hive-exec-2.1.0.2.6.3.0-235.jar:2.1.0.2.6.3.0-235]
at org.apache.hive.http.JMXJsonServlet.doGet(JMXJsonServlet.java:194) 
~[hive-exec-2.1.0.2.6.3.0-235.jar:2.1.0.2.6.3.0-235]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) 
~[servlet-api-2.5.jar:2.5]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) 
~[servlet-api-2.5.jar:2.5]
at 
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) 
~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) 
~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) 
~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:521) 
~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
 ~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
 ~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) 
~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
 ~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
 ~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) 
~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) 
~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:312)
 ~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
 ~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) 
~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at org.eclipse.jetty.server.Server.handle(Server.java:345) 
~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
 ~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:910)
 ~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634) 
~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) 
~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
 ~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
 ~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at 
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
 ~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
at

Re: Switched PreCommit-Admin over to Apache Yetus

2018-01-19 Thread Eric Wohlstadter 
Hi Allen,
 Thanks for providing the update.

Is it possible for you to check whether Tez has been affected?
https://issues.apache.org/jira/projects/TEZ/

For the last few days my patches aren't triggering any submission hook.
Here is an example: https://issues.apache.org/jira/browse/TEZ-3888

In exchange, I can offer a Sharks NHL fidget spinner.

--Eric

On Tue, Jan 16, 2018 at 7:40 PM, Vihang Karajgaonkar 
wrote:

> Thanks Allen for fixing the precommit-admin job.
>
> All,
>
> The issue with precommit hive job reported in
> https://issues.apache.org/jira/browse/HIVE-18461 is fixed as well. So
> automated precommit tests should be back for Hive. Incase your patch didn't
> make the precommit tests during this downtime, you will need to reattach
> the patch again so that it gets triggered in the next run.
>
> Thanks,
> Vihang
>
>
> On Tue, Jan 16, 2018 at 6:15 PM, Allen Wittenauer  wrote:
>
> > bcc: bui...@apache.org, d...@hbase.apache.org, dev@hive.apache.org,
> > common-...@hadoop.apache.org, d...@phoenix.apache.org,
> d...@oozie.apache.org
> >
> > (These are all of the projects that had pending issues.  This is
> > not all of the groups that actually rely upon this code...)
> >
> > The recent JIRA upgrade broke PreCommit-Admin.
> >
> > This breakage was obviously an unintended consequence.  This
> > python code hasn’t been touched in a very long time.  In fact, it was
> still
> > coming from Hadoop SVN tree as it had never been migrated to git.  So it
> > isn’t too surprising that after all this time something finally broke it.
> >
> > Luckily, Apache Yetus was already in the progress of adopting it
> > for a variety of reasons that I won't go into here.  With the breakage,
> > this work naturally became more urgent.  With the help of the Apache
> Yetus
> > community doing some quick reviews, I just switched PreCommit-Admin over
> to
> > using the master version of the equivalent code in the Yetus source tree.
> > As soon as the community can get a 0.7.0 release out, we’ll switch it
> over
> > from master to 0.7.0 so that it can follow our regular release cadence.
> > This also means that JIRA issues can be filled against Yetus for bugs
> seen
> > in the code base or for feature requests.  [Hopefully with code and docs
> > attached. :) ]
> >
> > In any case, with the re-activation of this job, all unprocessed
> > jobs just kicked off.  So don't be too surprised by the influx of
> feedback.
> >
> > As a sidenote, there are some other sticky issues with regards to
> > precommit setups on Jenkins.  I'll be sending another note in the future
> on
> > that though. I've had enough excitement for today. :)
> >
> > Thanks!
> >
> >
>

[jira] [Created] (HIVE-18499) Amend point lookup tests to check for data

2018-01-19 Thread Janaki Lahorani (JIRA) 
Janaki Lahorani created HIVE-18499:
--

 Summary: Amend point lookup tests to check for data
 Key: HIVE-18499
 URL: https://issues.apache.org/jira/browse/HIVE-18499
 Project: Hive
  Issue Type: Bug
Reporter: Janaki Lahorani
Assignee: Janaki Lahorani






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65219: HIVE-18486: Create tests to cover methods for adding Partitions

2018-01-19 Thread Adam Szita via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65219/#review195825
---



Looks good! I added small issues to fix, similarly with the dropPartitions 
change: some helper methods could be static (the ones that don't depend on HMS 
client certainly) and some could use Lists.newArrayList


standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
Lines 68 (patched)


Again this may be -1 instead, when MAX as a value is not used for any other 
reason than returning all partitions



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
Lines 672 (patched)


This is found multiple times in the code, should we create a method for it?


- Adam Szita


On Jan. 18, 2018, 4:58 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65219/
> ---
> 
> (Updated Jan. 18, 2018, 4:58 p.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18486
> https://issues.apache.org/jira/browse/HIVE-18486
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The following methods of IMetaStoreClient are covered by this test.
> - Partition add_partition(Partition)
> - int add_partitions(List)
> - List add_partitions(List, boolean, boolean)
> 
> 
> The test covers not just the happy pathes, but the edge cases as well.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65219/diff/1/
> 
> 
> Testing
> ---
> 
> Run the tests
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Review Request 65241: HIVE-18496: Create tests to cover add/alter/drop index methods

2018-01-19 Thread Marta Kuczora via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65241/
---

Review request for hive, Peter Vary and Adam Szita.


Bugs: HIVE-18496
https://issues.apache.org/jira/browse/HIVE-18496


Repository: hive-git


Description
---

The following methods of IMetaStoreClient are covered by this test.
- void createIndex(Index, Table)
- boolean dropIndex(String, String, String, boolean)
- void alter_index(String, String, String, Index)

The test covers not just the happy pathes, but the edge cases as well.


Diffs
-

  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddAlterDropIndexes.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/65241/diff/1/


Testing
---

Run the tests


Thanks,

Marta Kuczora

Review Request 65240: HIVE-18498: Create tests to cover get and list index methods

2018-01-19 Thread Marta Kuczora via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65240/
---

Review request for hive, Peter Vary and Adam Szita.


Bugs: HIVE-18498
https://issues.apache.org/jira/browse/HIVE-18498


Repository: hive-git


Description
---

The following methods of IMetaStoreClient are covered by this test.
- Index getIndex(String, String, String)
- List listIndexes(String, String, short)
- List listIndexNames(String, String, short)

The test covers not just the happy pathes, but the edge cases as well.


Diffs
-

  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestGetListIndexes.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/65240/diff/1/


Testing
---

Run the tests


Thanks,

Marta Kuczora

Re: Review Request 65217: Create tests to cover alterPartition and renamePartition methods

2018-01-19 Thread Adam Szita via Review Board 


> On Jan. 19, 2018, 11:41 a.m., Peter Vary wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
> > Lines 122 (patched)
> > 
> >
> > nit: Why static?

I like my helper methods static :)


> On Jan. 19, 2018, 11:41 a.m., Peter Vary wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
> > Lines 150 (patched)
> > 
> >
> > nit: Can we stick to one notation? I do not like _ in function names :)

fixed


> On Jan. 19, 2018, 11:41 a.m., Peter Vary wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
> > Lines 450 (patched)
> > 
> >
> > Question: Maybe in setup?

Some tests depend on not having a table in HMS.


- Adam


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65217/#review195795
---


On Jan. 19, 2018, 2:29 p.m., Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65217/
> ---
> 
> (Updated Jan. 19, 2018, 2:29 p.m.)
> 
> 
> Review request for hive, Marta Kuczora and Peter Vary.
> 
> 
> Bugs: HIVE-18468
> https://issues.apache.org/jira/browse/HIVE-18468
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Adding IMetastoreClient API tests that cover alterPartition ande 
> renamePartition methods
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65217/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adam Szita
> 
>

Re: Review Request 65217: Create tests to cover alterPartition and renamePartition methods

2018-01-19 Thread Adam Szita via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65217/
---

(Updated Jan. 19, 2018, 2:29 p.m.)


Review request for hive, Marta Kuczora and Peter Vary.


Changes
---

Fixed smaller issues


Bugs: HIVE-18468
https://issues.apache.org/jira/browse/HIVE-18468


Repository: hive-git


Description
---

Adding IMetastoreClient API tests that cover alterPartition ande 
renamePartition methods


Diffs (updated)
-

  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/65217/diff/2/

Changes: https://reviews.apache.org/r/65217/diff/1-2/


Testing
---


Thanks,

Adam Szita

[jira] [Created] (HIVE-18498) Create tests to cover get and list index methods

2018-01-19 Thread Marta Kuczora (JIRA) 
Marta Kuczora created HIVE-18498:


 Summary: Create tests to cover get and list index methods
 Key: HIVE-18498
 URL: https://issues.apache.org/jira/browse/HIVE-18498
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Marta Kuczora
Assignee: Marta Kuczora


The following methods of IMetaStoreClient are covered in this Jira:
{code:java}
- Index getIndex(String, String, String)
- List listIndexes(String, String, short)
- List listIndexNames(String, String, short){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 65237: HIVE-18496: Create tests to cover add/alter/drop index methods

2018-01-19 Thread Marta Kuczora via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65237/
---

Review request for hive, Peter Vary and Adam Szita.


Bugs: HIVE-18496
https://issues.apache.org/jira/browse/HIVE-18496


Repository: hive-git


Description
---

The following methods of IMetaStoreClient are covered by this test.
- void createIndex(Index, Table)
- boolean dropIndex(String, String, String, boolean)
- void alter_index(String, String, String, Index)

The test covers not just the happy pathes, but the edge cases as well.


Diffs
-

  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestIndexes.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/65237/diff/1/


Testing
---

Run the tests


Thanks,

Marta Kuczora

Re: Review Request 65219: HIVE-18486: Create tests to cover methods for adding Partitions

2018-01-19 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65219/#review195811
---



Thanks Marta for the patch.
Some questions below.
Also I think it would be nice to have test for the default values when adding a 
partition.

Thanks,
Peter


standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
Lines 236 (patched)


There are some interesting validation in the SD. Maybe we can add test for 
it here too.



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
Lines 282 (patched)


Is it worth to create a test for an External Table too?



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
Lines 700 (patched)


Like above - validation inside the SD might be important as well



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
Lines 749 (patched)


Maybe for external table too. And maybe check the locations



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
Lines 1117 (patched)


I do not understand this check


- Peter Vary


On Jan. 18, 2018, 4:58 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65219/
> ---
> 
> (Updated Jan. 18, 2018, 4:58 p.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18486
> https://issues.apache.org/jira/browse/HIVE-18486
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The following methods of IMetaStoreClient are covered by this test.
> - Partition add_partition(Partition)
> - int add_partitions(List)
> - List add_partitions(List, boolean, boolean)
> 
> 
> The test covers not just the happy pathes, but the edge cases as well.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65219/diff/1/
> 
> 
> Testing
> ---
> 
> Run the tests
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

[jira] [Created] (HIVE-18497) JDBC connection parameter to control socket read timeout

2018-01-19 Thread Amruth S (JIRA) 
Amruth S created HIVE-18497:
---

 Summary: JDBC connection parameter to control socket read timeout
 Key: HIVE-18497
 URL: https://issues.apache.org/jira/browse/HIVE-18497
 Project: Hive
  Issue Type: Improvement
  Components: Clients, JDBC
Reporter: Amruth S


Hive server failures are making the JDBC client get stuck in socketRead.

Users should be able to configure socket read timeout to fail fast in case of 
server failures.

*Proposed solution*

Add a Jdbc connection param **

*hive.client.read.socket.timeoutmillis*

This can control the socket read timeout, connect timeout in TCP as well as 
HTTP mode.

Let me know your thoughts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18496) Create tests to cover add/alter/drop index methods

2018-01-19 Thread Marta Kuczora (JIRA) 
Marta Kuczora created HIVE-18496:


 Summary: Create tests to cover add/alter/drop index methods
 Key: HIVE-18496
 URL: https://issues.apache.org/jira/browse/HIVE-18496
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Marta Kuczora
Assignee: Marta Kuczora






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65213: HIVE-18479: Create tests to cover methods for dropping Partitions

2018-01-19 Thread Adam Szita via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65213/#review195796
---



Overall looks good, I just have small observations.

One more nitpick: some helper methods could be static e.g. getYearAndMonthCols


standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDropPartitions.java
Lines 64 (patched)


Is there any reason to use 10? Supplying -1 would return all partitions 
which I think was the original intent in the test cases below (in 
listPartitions calls).



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDropPartitions.java
Lines 165 (patched)


createTable is called here twice because it's also in @Before. Does this 
mean it overrides the existing table with the supplied tableParams?



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDropPartitions.java
Lines 539-553 (patched)


Guava lib has a handy method for these use cases:

 ArrayList com.google.common.collect.Lists.newArrayList(E... elements)

...which is doing exactly the same + optimising on initial list size.


- Adam Szita


On Jan. 18, 2018, 1:11 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65213/
> ---
> 
> (Updated Jan. 18, 2018, 1:11 p.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18479
> https://issues.apache.org/jira/browse/HIVE-18479
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The following methods of IMetaStoreClient are covered by this test.
> - boolean dropPartition(String, String, List, boolean)
> - boolean dropPartition(String, String, List, PartitionDropOptions)
> - boolean dropPartition(String, String, String, boolean)
> 
> The test covers not just the happy pathes, but the edge cases as well.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDropPartitions.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65213/diff/1/
> 
> 
> Testing
> ---
> 
> Run the tests
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

[jira] [Created] (HIVE-18495) JUnit rule to enable Driver level testing

2018-01-19 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-18495:
---

 Summary: JUnit rule to enable Driver level testing
 Key: HIVE-18495
 URL: https://issues.apache.org/jira/browse/HIVE-18495
 Project: Hive
  Issue Type: Sub-task
  Components: Testing Infrastructure
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


I've tried to write a case for a sophisticated check...it worked so well that 
I've started using it and eventually created a junit rule to make it easier to 
reuse

Currently it takes ~15-25sec to run a test case with this framework (from which 
most of the time is the launch time of all kind of stuff which are needed to 
run a driver command).

* enable to write JUnit tests which has access to the {{IDriver}} level
* leave out the cli-driver; it sometimes causes problems
* write tests at the {{ql}} module
* it should also work from the IDE without changing anything

Note: JUnit 5 would be great for this task; but unfortunately junit5 needs 
maven-surefire 2.19.1 ; which causes all kinds of problems for hive devs using 
idea...so that's not an option.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65213: HIVE-18479: Create tests to cover methods for dropping Partitions

2018-01-19 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65213/#review195809
---



Thanks Marta for the patch.
Just one question.

Peter


standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDropPartitions.java
Lines 341 (patched)


Question: Does this behave the same way for external tables too?


- Peter Vary


On Jan. 18, 2018, 1:11 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65213/
> ---
> 
> (Updated Jan. 18, 2018, 1:11 p.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18479
> https://issues.apache.org/jira/browse/HIVE-18479
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The following methods of IMetaStoreClient are covered by this test.
> - boolean dropPartition(String, String, List, boolean)
> - boolean dropPartition(String, String, List, PartitionDropOptions)
> - boolean dropPartition(String, String, String, boolean)
> 
> The test covers not just the happy pathes, but the edge cases as well.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDropPartitions.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65213/diff/1/
> 
> 
> Testing
> ---
> 
> Run the tests
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Re: Review Request 65217: Create tests to cover alterPartition and renamePartition methods

2018-01-19 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65217/#review195795
---



Thanks for the patch Adam.
Questions, and some minor comments.

Thanks,
Peter


standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
Lines 52 (patched)


nit: We should import everything which is needed



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
Lines 122 (patched)


nit: Why static?



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
Lines 150 (patched)


nit: Can we stick to one notation? I do not like _ in function names :)



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
Lines 181 (patched)


These are the only parameters which can be changed?
If so should not we check the other paramteres are not changed in 
assertPartitionChanged?



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
Lines 298 (patched)


nit: We might want to add TODO comments to places where we think we want to 
change



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
Lines 450 (patched)


Question: Maybe in setup?


- Peter Vary


On Jan. 18, 2018, 2:26 p.m., Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65217/
> ---
> 
> (Updated Jan. 18, 2018, 2:26 p.m.)
> 
> 
> Review request for hive, Marta Kuczora and Peter Vary.
> 
> 
> Bugs: HIVE-18468
> https://issues.apache.org/jira/browse/HIVE-18468
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Adding IMetastoreClient API tests that cover alterPartition ande 
> renamePartition methods
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65217/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adam Szita
> 
>

[jira] [Created] (HIVE-18494) Regression: from HIVE-18069, the metastore directsql is getting disabled

2018-01-19 Thread Gopal V (JIRA) 
Gopal V created HIVE-18494:
--

 Summary: Regression: from HIVE-18069, the metastore directsql is 
getting disabled
 Key: HIVE-18494
 URL: https://issues.apache.org/jira/browse/HIVE-18494
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


{code:java}
018-01-19T02:12:34,457  WARN [pool-10-thread-1] metastore.ObjectStore: Falling 
back to ORM path due to direct SQL failure (this is not an error): 
java.lang.String cannot be cast to [Ljava.lang.Object; at

org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getTables(MetaStoreDirectSql.java:412)
 at 
org.apache.hadoop.hive.metastore.ObjectStore$2.getSqlResult(ObjectStore.java:1336)

2018-01-19T02:12:34,457 DEBUG [pool-10-thread-1] metastore.ObjectStore: Full 
DirectSQL callstack for debugging (note: this is not an error)

java.lang.ClassCastException: java.lang.String cannot be cast to 
[Ljava.lang.Object;

        at 
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getTables(MetaStoreDirectSql.java:412)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]

        at 
org.apache.hadoop.hive.metastore.ObjectStore$2.getSqlResult(ObjectStore.java:1336)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]

        at 
org.apache.hadoop.hive.metastore.ObjectStore$2.getSqlResult(ObjectStore.java:1332)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]

        at 
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3200)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]

        at 
org.apache.hadoop.hive.metastore.ObjectStore.getTablesInternal(ObjectStore.java:1344)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]

        at 
org.apache.hadoop.hive.metastore.ObjectStore.getTables(ObjectStore.java:1323) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
{code}

The DataNucleus layer returns List for this case, when exactly one 
column has been selected.

And MetastoreDirectSQL is disabled for all further queries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)