[jira] [Created] (HIVE-24171) Support HDFS reads from observer NameNodes

2020-09-15 Thread Aihua Xu (Jira)
Aihua Xu created HIVE-24171:
---

 Summary: Support HDFS reads from observer NameNodes
 Key: HIVE-24171
 URL: https://issues.apache.org/jira/browse/HIVE-24171
 Project: Hive
  Issue Type: New Feature
  Components: Hive
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


HDFS-12943 introduces the consistent reads from observer NameNodes which can 
boost the read performance and reduces the overloads on active NameNodes.

To take advantage of this feature, the clients are required to make a msync() 
call after writing the files or before reading the files since observer 
NameNodes could have the stale data for a small window. 

Hive needs to make msync() call to HDFS in some places, e.g., 1) after 
generating the plan files - map.xml and reduce.xml so they can get used later 
by executors; 2) after the intermediate files are generated so they can get 
used by later stages or HS2. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-24170) Add the UDF jar explicitely to the classpath while handling drop function event during repl load.

2020-09-15 Thread Pravin Sinha (Jira)
Pravin Sinha created HIVE-24170:
---

 Summary: Add the UDF jar explicitely to the classpath while 
handling drop function event during repl load.
 Key: HIVE-24170
 URL: https://issues.apache.org/jira/browse/HIVE-24170
 Project: Hive
  Issue Type: Bug
Reporter: Pravin Sinha
Assignee: Pravin Sinha






--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-24169) HiveServer2 UDF cache

2020-09-15 Thread Sam An (Jira)
Sam An created HIVE-24169:
-

 Summary: HiveServer2 UDF cache
 Key: HIVE-24169
 URL: https://issues.apache.org/jira/browse/HIVE-24169
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 4.0.0
Reporter: Sam An
Assignee: Sam An


UDF is cache per session. This optional feature can help speed up UDF access in 
S3 scenario.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-24168) Disable hdfsEncryptionShims cache during query-based compaction

2020-09-15 Thread Karen Coppage (Jira)
Karen Coppage created HIVE-24168:


 Summary: Disable hdfsEncryptionShims cache during query-based 
compaction
 Key: HIVE-24168
 URL: https://issues.apache.org/jira/browse/HIVE-24168
 Project: Hive
  Issue Type: Bug
Reporter: Karen Coppage
Assignee: Karen Coppage


Hive keeps a cache of encryption shims in SessionState (Map hdfsEncryptionShims). Each encryption shim in 
the cache stores a FileSystem object.
After compaction where the session user is not the same user as the owner of 
the partition/table directory, we close all FileSystem objects associated with 
the user running the compaction, possibly closing an FS stored in the 
encryption shim cache. The next time query-based compaction is run on a 
table/partition owned by the same user, compaction will fail in MoveTask[1] 
since the FileSystem stored in the cache was closed.
This change disables the cache during query-based compaction (optionally; 
default: disabled).

[1] Error:
{code:java}
2020-09-08 11:23:50,170 ERROR org.apache.hadoop.hive.ql.Driver: 
[rncdpdev-2.fyre.ibm.com-27]: FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Filesystem 
closed. org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: 
Filesystem closed
at org.apache.hadoop.hive.ql.metadata.Hive.needToCopy(Hive.java:4637)
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:4147)
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:4694)
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3120)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:423)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:477)
at 
org.apache.hadoop.hive.ql.DriverUtils.runOnDriver(DriverUtils.java:70)
at 
org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor.runCompactionQueries(QueryCompactor.java:116)
at 
org.apache.hadoop.hive.ql.txn.compactor.MmMajorQueryCompactor.runCompaction(MmMajorQueryCompactor.java:72)
at 
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:232)
at org.apache.hadoop.hive.ql.txn.compactor.Worker$1.run(Worker.java:221)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:218)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-24167) NPE in query 14 while generating plan for sub query predicate

2020-09-15 Thread Stamatis Zampetakis (Jira)
Stamatis Zampetakis created HIVE-24167:
--

 Summary: NPE in query 14 while generating plan for sub query 
predicate
 Key: HIVE-24167
 URL: https://issues.apache.org/jira/browse/HIVE-24167
 Project: Hive
  Issue Type: Bug
  Components: CBO
Reporter: Stamatis Zampetakis


TPC-DS query 14 (cbo_query14.q and query4.q) fail with NPE on the metastore 
with the partitioned TPC-DS 30TB dataset while generating the plan for sub 
query predicate. 

The problem can be reproduced using the PR in HIVE-23965.

The current stacktrace shows that the NPE appears while trying to display the 
debug message but even if this line didn't exist it would fail again later on.

{noformat}
java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:10867)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11765)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11622)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11649)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11622)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11649)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11635)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlanForSubQueryPredicate(SemanticAnalyzer.java:3375)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:3473)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:10819)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11765)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11622)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11625)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11625)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11649)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11622)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11649)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11635)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:12417)
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:718)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12519)
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:443)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:301)
at 
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:171)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:301)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:173)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:414)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:363)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:357)
at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:129)
at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:231)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355)
at 
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:740)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:710)
at 
org.apache.hadoop.hive.cli.control.CorePerfCliDriver.runTest(CorePerfCliDriver.java:103)
at 
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
at 
org.apache.hadoop.hive.cli.TestTezTPCDS30TBPerfCliDriver.testCliDriver(TestTezTPCDS30TBPerfCliDriver.java:83)
{noformat}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (HIVE-24166) use uppercase database name cause alias error

2020-09-15 Thread zhaolong (Jira)
zhaolong created HIVE-24166:
---

 Summary: use uppercase database name cause alias error
 Key: HIVE-24166
 URL: https://issues.apache.org/jira/browse/HIVE-24166
 Project: Hive
  Issue Type: Bug
Affects Versions: 4.0.0
Reporter: zhaolong
Assignee: zhaolong






--
This message was sent by Atlassian Jira
(v8.3.4#803005)