date:20180307

[jira] [Created] (HIVE-18906) Lower Logging for "Using direct SQL"

2018-03-07 Thread BELUGA BEHR (JIRA)

BELUGA BEHR created HIVE-18906:
--

 Summary: Lower Logging for "Using direct SQL"
 Key: HIVE-18906
 URL: https://issues.apache.org/jira/browse/HIVE-18906
 Project: Hive
  Issue Type: Improvement
Reporter: BELUGA BEHR


https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L181

{code}
2018-02-26 14:32:50,620  INFO  
org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]: Using 
direct SQL, underlying DB is MYSQL
2018-02-26 14:33:09,566  INFO  
org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]: Using 
direct SQL, underlying DB is MYSQL
2018-02-26 14:33:43,170  INFO  
org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]: Using 
direct SQL, underlying DB is MYSQL
2018-02-26 14:33:46,301  INFO  
org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]: Using 
direct SQL, underlying DB is MYSQL
2018-02-26 14:34:16,966  INFO  
org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]: Using 
direct SQL, underlying DB is MYSQL
2018-02-26 14:34:18,072  INFO  
org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-196]: Using 
direct SQL, underlying DB is MYSQL
2018-02-26 14:34:25,064  INFO  
org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]: Using 
direct SQL, underlying DB is MYSQL
2018-02-26 14:34:51,781  INFO  
org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]: Using 
direct SQL, underlying DB is MYSQL
2018-02-26 14:35:04,685  INFO  
org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]: Using 
direct SQL, underlying DB is MYSQL
2018-02-26 14:35:26,689  INFO  
org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]: Using 
direct SQL, underlying DB is MYSQL
{code}

I wouldn't mind knowing this information when the service starts up and detects 
that there is a Direct SQL compatible database, but I don't need to know about 
this every time.

Please lower to _debug_ level logging and parameterize with SLF4J.  Direct SQL 
is a performance thing and not something I care about during the normal course 
of operation (unless there is an error which would presumably be covered by an 
_error_ logging message).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65952: HIVE-18898: Fix NPEs in HiveMetastore.dropPartition method

2018-03-07 Thread Alexander Kolbasov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65952/#review198847
---



Afrew with Sahil - the checks should be on the server side.

- Alexander Kolbasov


On March 7, 2018, 3:48 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65952/
> ---
> 
> (Updated March 7, 2018, 3:48 p.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18898
> https://issues.apache.org/jira/browse/HIVE-18898
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The TestDropPartitions tests revealed that NPE is thrown if the 
> dropPartition(String db_name, String tbl_name, List part_vals, 
> PartitionDropOptions options) method is called with null options and with a 
> part_vals list which contains null elements.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  3128089 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDropPartitions.java
>  4d94ebf 
> 
> 
> Diff: https://reviews.apache.org/r/65952/diff/1/
> 
> 
> Testing
> ---
> 
> Run the TestDropPartitions tests.
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Re: Review Request 65943: HIVE-18888: Replace synchronizedMap with ConcurrentHashMap

2018-03-07 Thread Alexander Kolbasov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65943/
---

(Updated March 8, 2018, 1:19 a.m.)


Review request for hive, Aihua Xu, Andrew Sherman, Janaki Lahorani, Peter Vary, 
Sahil Takiar, and Vihang Karajgaonkar.


Changes
---

Removed unused imports.


Bugs: HIVE-1
https://issues.apache.org/jira/browse/HIVE-1


Repository: hive-git


Description
---

HIVE-1: Replace synchronizedMap with ConcurrentHashMap


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicValueRegistryTez.java 
0bed22a8f805ed64ba1af260434a40667004e051 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
c0be51e0b2c085896739a14096146822e4599d41 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 662de9a66767f27f31998f14c68f854e59993ab6 


Diff: https://reviews.apache.org/r/65943/diff/2/

Changes: https://reviews.apache.org/r/65943/diff/1-2/


Testing
---


Thanks,

Alexander Kolbasov

[jira] [Created] (HIVE-18905) HS2: SASL auth loads HiveConf

2018-03-07 Thread Gopal V (JIRA)

Gopal V created HIVE-18905:
--

 Summary: HS2: SASL auth loads HiveConf
 Key: HIVE-18905
 URL: https://issues.apache.org/jira/browse/HIVE-18905
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


SASL authentication filter does a new HiveConf() for no good reason.

{code}
  public static PasswdAuthenticationProvider 
getAuthenticationProvider(AuthMethods authMethod)
throws AuthenticationException {
return getAuthenticationProvider(authMethod, new HiveConf());
  }
{code}

The session HiveConf is not needed to do this operation & it can't be changed 
after the HS2 starts up (today).

{code}
org.apache.hadoop.hive.conf.HiveConf.() HiveConf.java:4404
org.apache.hive.service.auth.AuthenticationProviderFactory.getAuthenticationProvider(AuthenticationProviderFactory$AuthMethods)
 AuthenticationProviderFactory.java:61
org.apache.hive.service.auth.PlainSaslHelper$PlainServerCallbackHandler.handle(Callback[])
 PlainSaslHelper.java:106
org.apache.hive.service.auth.PlainSaslServer.evaluateResponse(byte[]) 
PlainSaslServer.java:103
org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(byte[])
 TSaslTransport.java:539
org.apache.thrift.transport.TSaslTransport.open() TSaslTransport.java:283
org.apache.thrift.transport.TSaslServerTransport.open() 
TSaslServerTransport.java:41
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TTransport)
 TSaslServerTransport.java:216
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run() 
TThreadPoolServer.java:269
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) 
ThreadPoolExecutor.java:1142
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:617
java.lang.Thread.run() Thread.java:745
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reload of aux jars on HiveServer2 failing

2018-03-07 Thread Shashank Pedamallu

Hi,

I have a jar that is configured under hive.reloadable.aux.jars.path. Upon
updating the jar and successful execution of *RELOAD;* command via Beeline,
the latest jar is not getting picked up. These are the steps I followed:
0. Started hs2 process with --hiveconf hive.reloadable.aux.jars.path=

1. Removed the *desired_jar*
2. Added an updated jar (that has some log statements) with same name in
the same location as *desired_jar_absolute_path*
3. Connected to HS2 process via Beeline and issued a RELOAD; command
4. Ran a query that should have emitted a log statement as per updated jar.
But it was not.

Please note that if I did a restart of hs2 process, the latest jar is being
picked and the log statements are getting printed.

Could someone please help me if I'm not using the RELOAD command right or
if there is a known issue surrounding it.

Thanks in advance!
-Shashank

[jira] [Created] (HIVE-18904) HS2: Static Loggers in hive-exec classes are being initialized per-thread

2018-03-07 Thread Gopal V (JIRA)

Gopal V created HIVE-18904:
--

 Summary: HS2: Static Loggers in hive-exec classes are being 
initialized per-thread
 Key: HIVE-18904
 URL: https://issues.apache.org/jira/browse/HIVE-18904
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Gopal V


Thread Class loaders shouldn't apply to any class which is part of the install.

{code}
HiveServer2-Handler-Pool: Thread-1001 <--- Frozen for at least 11m 25s
java.util.zip.ZipFile.open(String, int, long, boolean) ZipFile.java (native)
java.util.zip.ZipFile.(File, int, Charset) ZipFile.java:219
java.util.zip.ZipFile.(File, int) ZipFile.java:149
java.util.jar.JarFile.(File, boolean, int) JarFile.java:166
java.util.jar.JarFile.(String) JarFile.java:103
sun.misc.URLClassPath$JarLoader.getJarFile(URL) URLClassPath.java:893
sun.misc.URLClassPath$JarLoader.access$700(URLClassPath$JarLoader, URL) 
URLClassPath.java:756
sun.misc.URLClassPath$JarLoader$1.run() URLClassPath.java:838
sun.misc.URLClassPath$JarLoader$1.run() URLClassPath.java:831
java.security.AccessController.doPrivileged(PrivilegedExceptionAction) 
AccessController.java (native)
sun.misc.URLClassPath$JarLoader.ensureOpen() URLClassPath.java:830
sun.misc.URLClassPath$JarLoader.(URL, URLStreamHandler, HashMap) 
URLClassPath.java:803
sun.misc.URLClassPath$3.run() URLClassPath.java:530
sun.misc.URLClassPath$3.run() URLClassPath.java:520
java.security.AccessController.doPrivileged(PrivilegedExceptionAction) 
AccessController.java (native)
sun.misc.URLClassPath.getLoader(URL) URLClassPath.java:519
sun.misc.URLClassPath.getLoader(int) URLClassPath.java:492
sun.misc.URLClassPath.getNextLoader(int[], int) URLClassPath.java:457
sun.misc.URLClassPath.getResource(String, boolean) URLClassPath.java:211
java.net.URLClassLoader$1.run() URLClassLoader.java:365
java.net.URLClassLoader$1.run() URLClassLoader.java:362
java.security.AccessController.doPrivileged(PrivilegedExceptionAction, 
AccessControlContext) AccessController.java (native)
java.net.URLClassLoader.findClass(String) URLClassLoader.java:361
java.lang.ClassLoader.loadClass(String, boolean) ClassLoader.java:424
java.lang.ClassLoader.loadClass(String) ClassLoader.java:357
org.apache.logging.log4j.util.LoaderUtil.loadClass(String) LoaderUtil.java:163
org.apache.logging.slf4j.Log4jLogger.createConverter() Log4jLogger.java:416
org.apache.logging.slf4j.Log4jLogger.(ExtendedLogger, String) 
Log4jLogger.java:54
org.apache.logging.slf4j.Log4jLoggerFactory.newLogger(String, LoggerContext) 
Log4jLoggerFactory.java:37
org.apache.logging.slf4j.Log4jLoggerFactory.newLogger(String, LoggerContext) 
Log4jLoggerFactory.java:29
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(String) 
AbstractLoggerAdapter.java:52
org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(String) 
Log4jLoggerFactory.java:29
org.slf4j.LoggerFactory.getLogger(String) LoggerFactory.java:281
org.slf4j.LoggerFactory.getLogger(Class) LoggerFactory.java:301
org.apache.hadoop.hive.ql.parse.TableMask.(SemanticAnalyzer, HiveConf, 
boolean) TableMask.java:42
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(ASTNode, 
SemanticAnalyzer$PlannerContext) SemanticAnalyzer.java:11558
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(ASTNode, 
SemanticAnalyzer$PlannerContextFactory) SemanticAnalyzer.java:11665
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(ASTNode) 
CalcitePlanner.java:304
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(ASTNode, Context) 
BaseSemanticAnalyzer.java:273
org.apache.hadoop.hive.ql.Driver.compile(String, boolean, boolean) 
Driver.java:614
org.apache.hadoop.hive.ql.Driver.compileInternal(String, boolean) 
Driver.java:1545
org.apache.hadoop.hive.ql.Driver.compileAndRespond(String) Driver.java:1498
org.apache.hive.service.cli.operation.SQLOperation.prepare(QueryState) 
SQLOperation.java:198
org.apache.hive.service.cli.operation.SQLOperation.runInternal() 
SQLOperation.java:284
org.apache.hive.service.cli.operation.Operation.run() Operation.java:243
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(String,
 Map, boolean, long) HiveSessionImpl.java:541
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(String,
 Map, long) HiveSessionImpl.java:527
org.apache.hive.service.cli.CLIService.executeStatementAsync(SessionHandle, 
String, Map, long) CLIService.java:311
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(TExecuteStatementReq)
 ThriftCLIService.java:564
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService$Iface,
 TCLIService$ExecuteStatement_args) TCLIService.java:1557
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(Object,
 TBase) TCLIService.java:1542
org.apache.thrift.ProcessFunction.process(int, TProtocol, TProtocol, Object)

[jira] [Created] (HIVE-18903) Lower Logging Level for ObjectStore

2018-03-07 Thread BELUGA BEHR (JIRA)

BELUGA BEHR created HIVE-18903:
--

 Summary: Lower Logging Level for ObjectStore
 Key: HIVE-18903
 URL: https://issues.apache.org/jira/browse/HIVE-18903
 Project: Hive
  Issue Type: Improvement
  Components: Standalone Metastore
Affects Versions: 3.0.0, 2.4.0
Reporter: BELUGA BEHR


[https://github.com/apache/hive/blob/7c22d74c8d0eb0650adf6e84e0536127c103e46c/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java]

 
{code:java}
2018-03-01 06:51:58,051  INFO  org.apache.hadoop.hive.metastore.ObjectStore: 
[pool-4-thread-13]: ObjectStore, initialize called
2018-03-01 06:51:58,052  INFO  org.apache.hadoop.hive.metastore.ObjectStore: 
[pool-4-thread-13]: Initialized ObjectStore
{code}

Noting actionable or all that useful here.  Please lower to _debug_ or _trace_ 
level logging.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65943: HIVE-18888: Replace synchronizedMap with ConcurrentHashMap

2018-03-07 Thread Peter Vary via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65943/#review198825
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicValueRegistryTez.java
Lines 22-23 (original), 22-23 (patched)


Remove unused imports


- Peter Vary


On March 7, 2018, 6:08 a.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65943/
> ---
> 
> (Updated March 7, 2018, 6:08 a.m.)
> 
> 
> Review request for hive, Aihua Xu, Andrew Sherman, Janaki Lahorani, Peter 
> Vary, Sahil Takiar, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-1
> https://issues.apache.org/jira/browse/HIVE-1
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-1: Replace synchronizedMap with ConcurrentHashMap
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicValueRegistryTez.java 
> 0bed22a8f805ed64ba1af260434a40667004e051 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> c0be51e0b2c085896739a14096146822e4599d41 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  2be018ba0fd88daf484d8e20ddb64087958bed55 
> 
> 
> Diff: https://reviews.apache.org/r/65943/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

[jira] [Created] (HIVE-18902) Lower Logging Level for Cleaning Up "local RawStore"

2018-03-07 Thread BELUGA BEHR (JIRA)

BELUGA BEHR created HIVE-18902:
--

 Summary: Lower Logging Level for Cleaning Up "local RawStore"
 Key: HIVE-18902
 URL: https://issues.apache.org/jira/browse/HIVE-18902
 Project: Hive
  Issue Type: Improvement
  Components: Standalone Metastore
Affects Versions: 3.0.0, 2.4.0
Reporter: BELUGA BEHR


[https://github.com/apache/hive/blob/7c22d74c8d0eb0650adf6e84e0536127c103e46c/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L7756-L7768]

 
{code:java}
private static void cleanupRawStore() {
  try {
RawStore rs = HMSHandler.getRawStore();
if (rs != null) {
  HMSHandler.logInfo("Cleaning up thread local RawStore...");
  rs.shutdown();
}
  } finally {
HMSHandler handler = HMSHandler.threadLocalHMSHandler.get();
if (handler != null) {
  handler.notifyMetaListenersOnShutDown();
}
HMSHandler.threadLocalHMSHandler.remove();
HMSHandler.threadLocalConf.remove();
HMSHandler.threadLocalModifiedConfig.remove();
HMSHandler.removeRawStore();
HMSHandler.logInfo("Done cleaning up thread local RawStore");
  }
}
{code}

{code}
2018-03-03 17:21:49,832  INFO  org.apache.hadoop.hive.metastore.HiveMetaStore: 
[pool-4-thread-21]: 19: Cleaning up thread local RawStore...
2018-03-03 17:21:49,834  INFO  org.apache.hadoop.hive.metastore.HiveMetaStore: 
[pool-4-thread-21]: 19: Done cleaning up thread local RawStore
{code}

Not very helpful logging.  Please change logging levels to _debug_ or even 
_trace_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18901) Lower ResourceDownloader Logging to Debug

2018-03-07 Thread BELUGA BEHR (JIRA)

BELUGA BEHR created HIVE-18901:
--

 Summary: Lower ResourceDownloader Logging to Debug
 Key: HIVE-18901
 URL: https://issues.apache.org/jira/browse/HIVE-18901
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 3.0.0, 2.4.0
Reporter: BELUGA BEHR


https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/util/ResourceDownloader.java#L98

{code}
  LOG.info("converting to local {}", srcUri);
{code}

# Set to _debug_ level
# Uppercase the "C" to "Converting" to match all other Hive logging



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18900) Remove WARN for "Hadoop command-line option parsing not performed"

2018-03-07 Thread BELUGA BEHR (JIRA)

BELUGA BEHR created HIVE-18900:
--

 Summary: Remove WARN for "Hadoop command-line option parsing not 
performed"
 Key: HIVE-18900
 URL: https://issues.apache.org/jira/browse/HIVE-18900
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 3.0.0, 2.4.0
Reporter: BELUGA BEHR


2018-03-05 07:23:11,002  WARN  
[org.apache.hadoop.mapreduce.JobResourceUploader|https://csi.infra.cloudera.com/synergy/cluster_stats/services/hive/roles/hive-HIVESERVER2-40fe7dfa9435f6d3681cdaa7a7113014/roleType/HIVESERVER2/browse?cluster=Nielsen-Watch-Digital-Production=1520256759000=00a921de-7668-4149-896a-1fd08d33e4c7=org.apache.hadoop.mapreduce.JobResourceUploader]:
 [HiveServer2-Background-Pool: Thread-426416]: Hadoop command-line option 
parsing not performed. Implement the Tool interface and execute your 
application with ToolRunner to remedy this.
How do we remove this WARN message from appearing in the logs?  It's not clear 
to me, as an admin, what I should be doing in response to this message.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18899) Separate FetchWork required for each query that uses the results cache

2018-03-07 Thread Jason Dere (JIRA)

Jason Dere created HIVE-18899:
-

 Summary: Separate FetchWork required for each query that uses the 
results cache
 Key: HIVE-18899
 URL: https://issues.apache.org/jira/browse/HIVE-18899
 Project: Hive
  Issue Type: Sub-task
Reporter: Jason Dere
Assignee: Jason Dere


[~gopalv] found issues when running lots of concurrent queries against HS2 with 
the query cache. Looks like the FetchWork held by the results cache cannot be 
shared between multiple queries because it contains a ListSinkOperator that is 
used to hold the results of a fetch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65952: HIVE-18898: Fix NPEs in HiveMetastore.dropPartition method

2018-03-07 Thread Sahil Takiar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65952/#review198796
---




standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
Lines 996 (patched)


Why do this validation in `HiveMetaStoreClient` rather than `HiveMetaStore`?



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
Lines 1006 (patched)


Do the other parameters need to be checked?


- Sahil Takiar


On March 7, 2018, 3:48 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65952/
> ---
> 
> (Updated March 7, 2018, 3:48 p.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18898
> https://issues.apache.org/jira/browse/HIVE-18898
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The TestDropPartitions tests revealed that NPE is thrown if the 
> dropPartition(String db_name, String tbl_name, List part_vals, 
> PartitionDropOptions options) method is called with null options and with a 
> part_vals list which contains null elements.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  3128089 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDropPartitions.java
>  4d94ebf 
> 
> 
> Diff: https://reviews.apache.org/r/65952/diff/1/
> 
> 
> Testing
> ---
> 
> Run the TestDropPartitions tests.
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Review Request 65952: HIVE-18898: Fix NPEs in HiveMetastore.dropPartition method

2018-03-07 Thread Marta Kuczora via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65952/
---

Review request for hive, Peter Vary and Adam Szita.


Bugs: HIVE-18898
https://issues.apache.org/jira/browse/HIVE-18898


Repository: hive-git


Description
---

The TestDropPartitions tests revealed that NPE is thrown if the 
dropPartition(String db_name, String tbl_name, List part_vals, 
PartitionDropOptions options) method is called with null options and with a 
part_vals list which contains null elements.


Diffs
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 3128089 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDropPartitions.java
 4d94ebf 


Diff: https://reviews.apache.org/r/65952/diff/1/


Testing
---

Run the TestDropPartitions tests.


Thanks,

Marta Kuczora

[jira] [Created] (HIVE-18898) Fix NPEs in HiveMetastore.dropPartition method

2018-03-07 Thread Marta Kuczora (JIRA)

Marta Kuczora created HIVE-18898:


 Summary: Fix NPEs in HiveMetastore.dropPartition method
 Key: HIVE-18898
 URL: https://issues.apache.org/jira/browse/HIVE-18898
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Reporter: Marta Kuczora
Assignee: Marta Kuczora


The TestDropPartitions tests revealed that NPE is thrown if the 
dropPartition(String db_name, String tbl_name, List part_vals, 
PartitionDropOptions options) method is called with null options and with a 
part_vals list which contains null elements.

Example: NPE is thrown in the following test cases
 * testDropPartitionNullPartDropOptions
 * testDropPartitionNullVal



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18897) Hive is Double-Logging Invalid UDF Error

2018-03-07 Thread BELUGA BEHR (JIRA)

BELUGA BEHR created HIVE-18897:
--

 Summary: Hive is Double-Logging Invalid UDF Error
 Key: HIVE-18897
 URL: https://issues.apache.org/jira/browse/HIVE-18897
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 3.0.0, 2.4.0
Reporter: BELUGA BEHR


It logs the "invalid" function twice.  Once at ERROR level and once at WARN 
level.  Please change so that the error is logged once, at the WARN level.  The 
stack trace seems overkill here as well for such a trivial error... usually a 
user typo or the function needs to be registered.

{code:java}
2018-03-05 07:50:44,473  ERROR org.apache.hadoop.hive.ql.Driver: 
[HiveServer2-Handler-Pool: Thread-43]: FAILED: SemanticException [Error 10011]: 
Line 1:7 Invalid function 'aes_encrypt'
org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:7 Invalid function 
'aes_encrypt'
at 
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:836)
at 
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1176)
at 
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
at 
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:193)
at 
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:146)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:10422)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:10378)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3771)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3550)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8830)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8785)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9652)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9545)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10018)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10029)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9909)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:488)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1274)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1261)
at 
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:143)
at 
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:215)
at 
org.apache.hive.service.cli.operation.Operation.run(Operation.java:337)
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:425)
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:402)
at 
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:258)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:500)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:746)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at

[jira] [Created] (HIVE-18896) Do Not Log User Errors in SQLOperation Class

2018-03-07 Thread BELUGA BEHR (JIRA)

BELUGA BEHR created HIVE-18896:
--

 Summary: Do Not Log User Errors in SQLOperation Class
 Key: HIVE-18896
 URL: https://issues.apache.org/jira/browse/HIVE-18896
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 3.0.0, 2.4.0
Reporter: BELUGA BEHR


[https://github.com/apache/hive/blob/f7dea1060247dddccd69112e24200ec84d2847a3/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L344]
{code:java}
2018-03-05 07:00:25,920  ERROR org.apache.hive.service.cli.operation.Operation: 
[HiveServer2-Background-Pool: Thread-2412]: Error running hive query: 
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: 
FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. new table 
sales.customer_acct already exists
at 
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:399)
at 
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:189)
at 
org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:76)
at 
org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:241)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1714)
at 
org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:254)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter 
table. new table sales.customer_acct already exists
at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:556)
at org.apache.hadoop.hive.ql.exec.DDLTask.alterTable(DDLTask.java:3379)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:340)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1969)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1682)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1419)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1203)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1198)
at 
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:187)
... 11 more
Caused by: InvalidOperationException(message:new table sales.customer_acct 
already exists)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_cascade_result$alter_table_with_cascade_resultStandardScheme.read(ThriftHiveMetastore.java:42276)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_cascade_result$alter_table_with_cascade_resultStandardScheme.read(ThriftHiveMetastore.java:42262)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_cascade_result.read(ThriftHiveMetastore.java:42204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_alter_table_with_cascade(ThriftHiveMetastore.java:1410)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.alter_table_with_cascade(ThriftHiveMetastore.java:1394)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:350)
at 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.alter_table(SessionHiveMetaStoreClient.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:105)
at com.sun.proxy.$Proxy11.alter_table(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at

[jira] [Created] (HIVE-18895) Address TODO Comments in SQLOperation Class

2018-03-07 Thread BELUGA BEHR (JIRA)

BELUGA BEHR created HIVE-18895:
--

 Summary: Address TODO Comments in SQLOperation Class
 Key: HIVE-18895
 URL: https://issues.apache.org/jira/browse/HIVE-18895
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 3.0.0, 2.4.0
Reporter: BELUGA BEHR


https://github.com/apache/hive/blob/f7dea1060247dddccd69112e24200ec84d2847a3/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java

There are a couple of TODO comments in this class.  Please review, address, and 
remove.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18894) investigate stats8.q

2018-03-07 Thread Zoltan Haindrich (JIRA)

Zoltan Haindrich created HIVE-18894:
---

 Summary: investigate stats8.q
 Key: HIVE-18894
 URL: https://issues.apache.org/jira/browse/HIVE-18894
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich


rawdatasize 5312 seems to be an underestimation...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18893) Estimate avgrowsize for stats calc in mixed case

2018-03-07 Thread Zoltan Haindrich (JIRA)

Zoltan Haindrich created HIVE-18893:
---

 Summary: Estimate avgrowsize for stats calc in mixed case
 Key: HIVE-18893
 URL: https://issues.apache.org/jira/browse/HIVE-18893
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich


after HIVE-18140; in case there is at least 1 partition with COMPLETE stats 
state; deser ratio can be estimated using that to provide better estimations 
for partitions which doesn't have any



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 65947: HIVE-18892: Fix NPEs in HiveMetastore.exchange_partitions method

2018-03-07 Thread Marta Kuczora via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65947/
---

Review request for hive, Peter Vary and Adam Szita.


Bugs: HIVE-18892
https://issues.apache.org/jira/browse/HIVE-18892


Repository: hive-git


Description
---

The TestExchangePartitions tests revealed that NPE is thrown if the 
exchange_partitions method is called with null, empty or non-existing DB and 
table names. These NPEs could be prevented with a simple null check and a 
MetaException with a proper error message should be thrown instead.


Diffs
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 662de9a 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestExchangePartitions.java
 5b21491 


Diff: https://reviews.apache.org/r/65947/diff/1/


Testing
---

Run the TestExchangePartitions tests.


Thanks,

Marta Kuczora

[jira] [Created] (HIVE-18892) Fix NPEs in HiveMetastore.exchange_partitions method

2018-03-07 Thread Marta Kuczora (JIRA)

Marta Kuczora created HIVE-18892:


 Summary: Fix NPEs in HiveMetastore.exchange_partitions method
 Key: HIVE-18892
 URL: https://issues.apache.org/jira/browse/HIVE-18892
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Reporter: Marta Kuczora
Assignee: Marta Kuczora


The TestExchangePartitions tests revealed that NPE is thrown if the 
exchange_partitions method is called with null, empty or non-existing DB and 
table names. These NPEs could be prevented with a simple null check and a 
MetaException with a proper error message should be thrown instead.
Example: NPE is thrown in the following test cases
 * testExchangePartitionsNonExistingSourceTable
 * testExchangePartitionsNonExistingSourceDB
 * testExchangePartitionsNonExistingDestTable
 * testExchangePartitionsNonExistingDestDB
 * testExchangePartitionsEmptySourceTable
 * testExchangePartitionsEmptySourceDB
 * testExchangePartitionsEmptyDestTable
 * testExchangePartitionsEmptyDestDB
 * testExchangePartitionsNullSourceTable
 * testExchangePartitionsNullSourceDB
 * testExchangePartitionsNullDestTable
 * testExchangePartitionsNullDestDB
 * testExchangePartitionsNullPartSpec



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18906) Lower Logging for "Using direct SQL"

Re: Review Request 65952: HIVE-18898: Fix NPEs in HiveMetastore.dropPartition method

Re: Review Request 65943: HIVE-18888: Replace synchronizedMap with ConcurrentHashMap

[jira] [Created] (HIVE-18905) HS2: SASL auth loads HiveConf

Reload of aux jars on HiveServer2 failing

[jira] [Created] (HIVE-18904) HS2: Static Loggers in hive-exec classes are being initialized per-thread

[jira] [Created] (HIVE-18903) Lower Logging Level for ObjectStore

Re: Review Request 65943: HIVE-18888: Replace synchronizedMap with ConcurrentHashMap

[jira] [Created] (HIVE-18902) Lower Logging Level for Cleaning Up "local RawStore"

[jira] [Created] (HIVE-18901) Lower ResourceDownloader Logging to Debug

[jira] [Created] (HIVE-18900) Remove WARN for "Hadoop command-line option parsing not performed"

[jira] [Created] (HIVE-18899) Separate FetchWork required for each query that uses the results cache

Re: Review Request 65952: HIVE-18898: Fix NPEs in HiveMetastore.dropPartition method

Review Request 65952: HIVE-18898: Fix NPEs in HiveMetastore.dropPartition method

[jira] [Created] (HIVE-18898) Fix NPEs in HiveMetastore.dropPartition method

[jira] [Created] (HIVE-18897) Hive is Double-Logging Invalid UDF Error

[jira] [Created] (HIVE-18896) Do Not Log User Errors in SQLOperation Class

[jira] [Created] (HIVE-18895) Address TODO Comments in SQLOperation Class

[jira] [Created] (HIVE-18894) investigate stats8.q

[jira] [Created] (HIVE-18893) Estimate avgrowsize for stats calc in mixed case

Review Request 65947: HIVE-18892: Fix NPEs in HiveMetastore.exchange_partitions method

[jira] [Created] (HIVE-18892) Fix NPEs in HiveMetastore.exchange_partitions method

22 matches

Site Navigation

Mail list logo

Footer information