[jira] [Created] (HIVE-22910) CBO fails when subquery with rank left joined

2020-02-18 Thread Krisztian Kasa (Jira) 
Krisztian Kasa created HIVE-22910:
-

 Summary: CBO fails when subquery with rank left joined
 Key: HIVE-22910
 URL: https://issues.apache.org/jira/browse/HIVE-22910
 Project: Hive
  Issue Type: Bug
Reporter: Krisztian Kasa
Assignee: Krisztian Kasa


*Repro*
{code}
CREATE TABLE table1(a int, b int);

ANALYZE TABLE table1 COMPUTE STATISTICS FOR COLUMNS;


EXPLAIN CBO
SELECT sub1.r FROM
(
SELECT
RANK() OVER (ORDER BY t1.b desc) as r
FROM table1 t1
JOIN table1 t2 ON t1.a = t2.b
) sub1
LEFT OUTER JOIN table1 t3
ON sub1.r = t3.a;
{code}

{code}
See ./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log, or 
check ./ql/target/surefire-reports or ./itests/qtest/target/surefire-reports/ 
for specific test cases logs.
 org.apache.hadoop.hive.ql.parse.SemanticException: Line 0:-1 Invalid column 
reference 'b': (possible column names are: $hdt$_0.a, $hdt$_0.b, $hdt$_1.b)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:13089)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:13031)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:12999)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genJoinKeys(SemanticAnalyzer.java:9248)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genJoinOperator(SemanticAnalyzer.java:9409)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genJoinPlan(SemanticAnalyzer.java:9624)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11781)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11661)
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:534)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12547)
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:361)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
at 
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:171)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:219)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:103)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:594)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:540)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:534)
at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:249)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:193)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:415)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:346)
at 
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:709)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:679)
at 
org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:169)
at 
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
at 
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:135)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4Class

[jira] [Created] (HIVE-22909) Vectorized row_number() returns incorrect results in case it is called multiple times with different constant expression

2020-02-18 Thread Shubham Chaurasia (Jira) 
Shubham Chaurasia created HIVE-22909:


 Summary: Vectorized row_number() returns incorrect results in case 
it is called multiple times with different constant expression
 Key: HIVE-22909
 URL: https://issues.apache.org/jira/browse/HIVE-22909
 Project: Hive
  Issue Type: Bug
Reporter: Shubham Chaurasia
Assignee: Shubham Chaurasia


Vectorized row_number() returns incorrect results in case it is called multiple 
times in the same query with different constant expressions.

Example
{code}
select row_number() over(partition by 1) r1, row_number() over(partition by 2) 
r2, t from over10k_n8 limit 1100;
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22908) AM caching connections to LLAP based on hostname and port does not work in kubernetes

2020-02-18 Thread Prasanth Jayachandran (Jira) 
Prasanth Jayachandran created HIVE-22908:


 Summary: AM caching connections to LLAP based on hostname and port 
does not work in kubernetes
 Key: HIVE-22908
 URL: https://issues.apache.org/jira/browse/HIVE-22908
 Project: Hive
  Issue Type: Bug
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


AM is caching all connections to LLAP services using combination of hostname 
and port which does not work in kubernetes environment where hostname of pod 
and port can be same with statefulset. This causes AM to talk to old LLAP which 
could have died or OOM/Pod kill etc. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22907) Break up DDLSemanticAnalyzer - extract rest of the alter Table analyzers

2020-02-18 Thread Miklos Gergely (Jira) 
Miklos Gergely created HIVE-22907:
-

 Summary: Break up DDLSemanticAnalyzer - extract rest of the alter 
Table analyzers
 Key: HIVE-22907
 URL: https://issues.apache.org/jira/browse/HIVE-22907
 Project: Hive
  Issue Type: Sub-task
Reporter: Miklos Gergely
Assignee: Miklos Gergely


DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is to 
refactor it in order to have everything cut into more handleable classes under 
the package  org.apache.hadoop.hive.ql.exec.ddl:
 * have a separate class for each analyzers
 * have a package for each operation, containing an analyzer, a description, 
and an operation, so the amount of classes under a package is more manageable

Step #14: extract the table storage related analyzers from DDLSemanticAnalyzer, 
and move them under the new package.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72139: Break up DDLSemanticAnalyzer - extract Table storage analyzers

2020-02-18 Thread Miklos Gergely 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72139/
---

(Updated Feb. 18, 2020, 6:46 p.m.)


Review request for hive and Zoltan Haindrich.


Bugs: HIVE-22895
https://issues.apache.org/jira/browse/HIVE-22895


Repository: hive-git


Description
---

DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is to 
refactor it in order to have everything cut into more handleable classes under 
the package  org.apache.hadoop.hive.ql.exec.ddl:

have a separate class for each analyzers
have a package for each operation, containing an analyzer, a description, and 
an operation, so the amount of classes under a package is more manageable
Step #14: extract the table storage related analyzers from DDLSemanticAnalyzer, 
and move them under the new package.


Diffs (updated)
-

  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzer.java
 8277d34731 
  ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLSemanticAnalyzerFactory.java 
119b555d58 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/alter/location/AlterDatabaseSetLocationAnalyzer.java
 e22fd924ef 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/alter/owner/AlterDatabaseSetOwnerAnalyzer.java
 4e9f079184 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/alter/poperties/AlterDatabaseSetPropertiesAnalyzer.java
 e23293598d 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/create/CreateDatabaseAnalyzer.java
 eb37193376 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/desc/DescDatabaseAnalyzer.java
 50ff0159e9 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/drop/DropDatabaseAnalyzer.java
 5823b1d54b 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/lock/LockDatabaseAnalyzer.java
 498e3aba3e 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/show/ShowDatabasesAnalyzer.java
 425205ff1e 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/showcreate/ShowCreateDatabaseAnalyzer.java
 e8f028d2ad 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/unlock/UnlockDatabaseAnalyzer.java
 e9f3d9b2a9 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/use/SwitchDatabaseAnalyzer.java
 134b67183a 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/function/create/CreateFunctionAnalyzer.java
 40dc4dc15d 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/function/desc/DescFunctionAnalyzer.java
 50454f4bee 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/function/drop/DropFunctionAnalyzer.java
 7ab172a7cf 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/function/macro/create/CreateMacroAnalyzer.java
 a2177e0291 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/function/macro/drop/DropMacroAnalyzer.java
 15ce4750c9 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/function/reload/ReloadFunctionsAnalyzer.java
 06c6622dba 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/function/show/ShowFunctionsAnalyzer.java
 3926585c64 
  ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/conf/ShowConfAnalyzer.java 
a7c88a52b7 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/metadata/CacheMetadataAnalyzer.java
 c040b40d93 
  ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/msck/MsckAnalyzer.java 
3d8fb584eb 
  ql/src/java/org/apache/hadoop/hive/ql/ddl/privilege/grant/GrantAnalyzer.java 
c16609bc51 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/privilege/revoke/RevokeAnalyzer.java 
8fe16119b9 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/privilege/role/create/CreateRoleAnalyzer.java
 d4975d16a9 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/privilege/role/drop/DropRoleAnalyzer.java
 0b2a848d1c 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/privilege/role/grant/GrantRoleAnalyzer.java
 d845c43411 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/privilege/role/revoke/RevokeRoleAnalyzer.java
 21f71cb630 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/privilege/role/set/SetRoleAnalyzer.java
 39d887c7a5 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/privilege/role/show/ShowCurrentRoleAnalyzer.java
 f4f8437d47 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/privilege/role/show/ShowRolesAnalyzer.java
 20860fd882 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/privilege/show/grant/ShowGrantAnalyzer.java
 8b0f51c53c 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/privilege/show/principals/ShowPrincipalsAnalyzer.java
 7cdeef1abb 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/privilege/show/rolegrant/ShowRoleGrantAnalyzer.java
 2e093e7a74 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/abort/AbortTransactionsAnalyzer.java
 21116a8f2b 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/kill/KillQueriesAnalyzer.java 
71b1e0478b 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsAnalyzer.java
 99b9f5d5b3 
  
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/transactions/ShowTransactionsAnalyzer.java
 5bb259597f 
  
ql/src/java/org/apache/hadoop/h

[jira] [Created] (HIVE-22906) Redundant checkLock Mutex blocks concurrent Lock requests

2020-02-18 Thread Denys Kuzmenko (Jira) 
Denys Kuzmenko created HIVE-22906:
-

 Summary: Redundant checkLock Mutex blocks concurrent Lock requests
 Key: HIVE-22906
 URL: https://issues.apache.org/jira/browse/HIVE-22906
 Project: Hive
  Issue Type: Improvement
Reporter: Denys Kuzmenko






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22905) Transaction is not aborted when query cancelled, only when session is closed

2020-02-18 Thread Aron Hamvas (Jira) 
Aron Hamvas created HIVE-22905:
--

 Summary: Transaction is not aborted when query cancelled, only 
when session is closed
 Key: HIVE-22905
 URL: https://issues.apache.org/jira/browse/HIVE-22905
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 3.1.2
Reporter: Aron Hamvas
Assignee: Aron Hamvas


Reproduction:
# Start HMS
# Start HS2
# Start beeline
# Execute a query and cancel query after transaction is started and before 
locks are acquired (yeah, that will take a debugger with breakpoints in 
DbLockManager)
# Do not terminate Hive session

DbLockManager will keep spinning in the checkLock loop as long as the Hive 
session is not closed since abortTxns is not invoked on HMS, and until the lock 
is acquired (even though the query could have been canceled long time ago).

Driver only checks in close() method if there are locks acquired but it does 
not check if a txn is open. 




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22904) Compaction cleaner cannot find COMPACTION_QUEUE table using postgres db

2020-02-18 Thread Jira 
László Pintér created HIVE-22904:


 Summary: Compaction cleaner cannot find COMPACTION_QUEUE table 
using postgres db
 Key: HIVE-22904
 URL: https://issues.apache.org/jira/browse/HIVE-22904
 Project: Hive
  Issue Type: Bug
Reporter: László Pintér
Assignee: László Pintér


In CompactionTxnHandler 
{code:java}
delete from COMPACTION_QUEUE where cq_id = ?
{code}
fails with 
{code:java}
org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does not 
exist
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22903) Vectorized row_number() resets the row number after one batch in case of constant expression in partition clause

2020-02-18 Thread Shubham Chaurasia (Jira) 
Shubham Chaurasia created HIVE-22903:


 Summary: Vectorized row_number() resets the row number after one 
batch in case of constant expression in partition clause
 Key: HIVE-22903
 URL: https://issues.apache.org/jira/browse/HIVE-22903
 Project: Hive
  Issue Type: Bug
  Components: UDF, Vectorization
Affects Versions: 4.0.0
Reporter: Shubham Chaurasia
Assignee: Shubham Chaurasia


Vectorized row number implementation resets the row number when constant 
expression is passed in partition clause.

Repro Query
{code}
select row_number() over(partition by 1) r1, t from over10k_n8;

Or

select row_number() over() r1, t from over10k_n8;
{code}
where table over10k_n8 contains more than 1024 records.

This happens because currently in VectorPTFOperator, we reset evaluators if 
only partition clause is there.
{code:java}
// If we are only processing a PARTITION BY, reset our evaluators.
if (!isPartitionOrderBy) {
  groupBatches.resetEvaluators();
}
{code}

To resolve, we should also check if the entire partition clause is a constant 
expression, if it is so then we should not do {{groupBatches.resetEvaluators()}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22902) Incorrect Spark/SparkOnYarn result for auto_sortmerge_join_16.q

2020-02-18 Thread Karen Coppage (Jira) 
Karen Coppage created HIVE-22902:


 Summary: Incorrect Spark/SparkOnYarn result for 
auto_sortmerge_join_16.q
 Key: HIVE-22902
 URL: https://issues.apache.org/jira/browse/HIVE-22902
 Project: Hive
  Issue Type: Bug
Reporter: Karen Coppage


In files
{code:java}
auto_sortmerge_join_16.q.out_spark [TestMiniSparkOnYarnCliDriver]
auto_sortmerge_join_16.q.out [TestSparkCliDriver]
{code}
at the first run of:
{code:java}
select a.key , a.value , b.value , 'day1' as day, 1 as pri
from
( select key, value
  from bucket_big_n17 where day='day1' ) a
left outer join
( select key, value
  from bucket_small_n17
  where pri between 1 and 2 ) b
on
(a.key = b.key)
{code}
the output is (beginning line 444):
{code:java}
0   val_0   val_0   day11
0   val_0   val_0   day11
0   val_0   val_0   day11
0   val_0   val_0   day11
0   val_0   val_0   day11
0   val_0   val_0   day11
103 val_103 val_103 day11
103 val_103 val_103 day11
103 val_103 val_103 day11
103 val_103 val_103 day11
169 val_169 NULLday11
172 val_172 val_172 day11
172 val_172 val_172 day11
172 val_172 val_172 day11
172 val_172 val_172 day11
374 val_374 NULLday11
{code}
Result should not include NULLs. It should match 
llap/auto_sortmerge_join_16.q.out, beginning line 461:
{code:java}
0   val_0   val_0   day11
0   val_0   val_0   day11
0   val_0   val_0   day11
0   val_0   val_0   day11
0   val_0   val_0   day11
0   val_0   val_0   day11
103 val_103 val_103 day11
103 val_103 val_103 day11
103 val_103 val_103 day11
103 val_103 val_103 day11
169 val_169 val_169 day11
169 val_169 val_169 day11
169 val_169 val_169 day11
169 val_169 val_169 day11
169 val_169 val_169 day11
169 val_169 val_169 day11
169 val_169 val_169 day11
169 val_169 val_169 day11
172 val_172 val_172 day11
172 val_172 val_172 day11
172 val_172 val_172 day11
172 val_172 val_172 day11
374 val_374 val_374 day11
374 val_374 val_374 day11
{code}
Looks like this was changed in HIVE-20915.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22901) Variable substitution can lead to OOM on circular references

2020-02-18 Thread Daniel Voros (Jira) 
Daniel Voros created HIVE-22901:
---

 Summary: Variable substitution can lead to OOM on circular 
references
 Key: HIVE-22901
 URL: https://issues.apache.org/jira/browse/HIVE-22901
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 3.1.2
Reporter: Daniel Voros
Assignee: Daniel Voros


{{SystemVariables#substitute()}} is dealing with circular references between 
variables by only doing the substitution 40 times by default. If the 
substituted part is sufficiently large though, it's possible that the 
substitution will produce a string bigger than the heap size within the 40 
executions.

Take the following test case that fails with OOM in current master (third round 
of execution would need 10G heap, while running with only 2G):

{code}
@Test
public void testSubstitute() {
String randomPart = RandomStringUtils.random(100_000);
String reference = "${hiveconf:myTestVariable}";

StringBuilder longStringWithReferences = new StringBuilder();
for(int i = 0; i < 10; i ++) {
longStringWithReferences.append(randomPart).append(reference);
}

SystemVariables uut = new SystemVariables();

HiveConf conf = new HiveConf();
conf.set("myTestVariable", longStringWithReferences.toString());

uut.substitute(conf, longStringWithReferences.toString(), 40);
}
{code}

Produces:
{code}
java.lang.OutOfMemoryError: Java heap space

at java.util.Arrays.copyOf(Arrays.java:3332)
at 
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at 
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuilder.append(StringBuilder.java:136)
at 
org.apache.hadoop.hive.conf.SystemVariables.substitute(SystemVariables.java:110)
at 
org.apache.hadoop.hive.conf.SystemVariablesTest.testSubstitute(SystemVariablesTest.java:27)
{code}

We should check the size of the substituted query and bail out earlier.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22900) Predicate Push Down Of Like Filter While Fetching Partition Data From MetaStore

2020-02-18 Thread Syed Shameerur Rahman (Jira) 
Syed Shameerur Rahman created HIVE-22900:


 Summary: Predicate Push Down Of Like Filter While Fetching 
Partition Data From MetaStore
 Key: HIVE-22900
 URL: https://issues.apache.org/jira/browse/HIVE-22900
 Project: Hive
  Issue Type: New Feature
Reporter: Syed Shameerur Rahman
Assignee: Syed Shameerur Rahman
 Fix For: 4.0.0


Currently PPD is disabled for like filter while fetching partition data from 
metastore. The following patch covers all the test cases mentioned in HIVE-5134



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Moving Hive site from Apache CMS/SVN to Git

2020-02-18 Thread Roy Lenferink 
Anyone any thoughts on this? Otherwise I'll start a lazy vote for moving over.

On 2020/02/13 18:33:13, Roy Lenferink  wrote: 
> Hello Hive community,
> 
> I noticed Hive is still serving its site from SVN with help of the Apache 
> CMS. At the Apache Celix 
> project we decided to move away from the CMS and move towards git for serving 
> our website [1]. 
> This because the CMS was introduced in 2010 and is currently not accepting 
> any new projects. With 
> the move to Hugo we're sure that we still can update the site when the ASF 
> CMS will be 
> decommissioned.
> 
> The master branch contains the actual website sources whereas the generated 
> website is served 
> from the asf-site branch. Hugo is being used as static website generator. 
> Content is written in 
> Markdown.
> 
> I had a look at Hive its site and it was quite easy to migrate Hive its site 
> to use Hugo as well. An 
> example of Hugo its output is (for demo purposes only) available on [2]. 
> These are the existing
> markdown files with a slightly changed header.
> 
> Moving to git can enlarge the visibility of how projects are functioning. 
> Next to that, new contributors 
> can simply create a pull request against the website repository if they find 
> anything they want to 
> improve. An 'Edit on GitHub' button can be added as well to the site, e.g. an 
> example of a page on
> the Celix website [3].
> 
> If the community is interested in this move what I propose:
> - Someone from the PMC to request a new git repository for the website (e.g. 
> hive-site) via [4]
> - Me creating the pull request from the repository I am temporarily using for 
> the website contents [5] 
> to the official hive-site repository
> - Create a Jenkins job to automatically build the site after changes happen 
> on the master branch.
> - When the pull request is reviewed and merged ask INFRA to move over from 
> the current svnpubsub
> to the gitpubsub approach and remove MINA from the Apache CMS.
> 
> Next to that Hive is also having javadocs which are only part of the 
> production site (won't be included
> when people clone the Hive website SVN repo). These can still be served from 
> e.g. a release-docs
> branch. In the README we can then mention that the hive-site repo needs to be 
> cloned using the
> --single-branch option. An alternative would be serving the javadocs from a 
> separate repository (e.g.
> hive-release-docs).
> 
> I'd like to hear everyone's opinion on this :)
> 
> Best regards,
> Roy
> 
> [1] https://github.com/apache/celix-site
> [2] http://hive.roylenferink.nl/
> [3] http://celix.apache.org/contributing/releasing.html
> [4] https://gitbox.apache.org/setup/newrepo.html
> [5] https://github.com/rlenferink/hive-site
> 
>

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-18 Thread Marta Kuczora via Review Board 


> On Feb. 4, 2020, 3:49 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
> > Lines 1732-1737 (patched)
> > 
> >
> > What about using lambda here?
> 
> Marta Kuczora wrote:
> Fixed it.

At the end this code part got removed.


- Marta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/#review219487
---


On Feb. 18, 2020, 12:21 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71904/
> ---
> 
> (Updated Feb. 18, 2020, 12:21 p.m.)
> 
> 
> Review request for hive, Gopal V and Peter Vary.
> 
> 
> Bugs: HIVE-21164
> https://issues.apache.org/jira/browse/HIVE-21164
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the original patch with saving the task attempt ids in the file 
> names and also fixed some bugs in the original patch.
> With this fix, inserting into an ACID table would not use move task to place 
> the generated files into the final directory. It will inserts every files to 
> the final directory and then clean up the files which are not needed (like 
> written by failed task attempts).
> Also fixed the replication tests which failed for the original patch as well.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d3cb60b790 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  da677c7977 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 056cd27496 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
>  31d15fdef9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  c2aa73b5f1 
>   itests/src/test/resources/testconfiguration.properties 1b1bf1147a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 9a3258115b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 
> 8980a6292a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 
> c4c56f8477 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> b8a0f0465c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
> 398698ec06 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  2543dc6fc4 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1eb9c12cc8 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 73ca658d9c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 33d3beba46 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
> c102a69f8f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 739f2b654b 
>   ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289583 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnAddPartition.java c9cb6692df 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 842140815d 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java e56d83158f 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 908ceb43fc 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnConcatenate.java 8676e0db11 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java 66b2b2768b 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java bb55d9fd79 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java ea6b1d9bec 
>   ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
> af14e628b3 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 83db48e758 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
> 2c4b69b2fe 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 48e9afc496 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/C

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-18 Thread Marta Kuczora via Review Board 


> On Feb. 4, 2020, 10:16 p.m., Rajesh Balamohan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
> > Line 4382 (original), 4397 (patched)
> > 
> >
> > Is this needed for direct insert?. In objectstores, we could have calls 
> > getting throttled.

That's a really good question, I was thinking about it a lot. I think it is not 
needed. This method does two things: removes the temporarily and duplicated 
files and returns the emptyBuckets list. This list contains elements if the 
number of buckets are bigger than the number of files. In this case, for MM 
tables,  empty files will be created. But this is not the case for ACID tables, 
there won't be any empty files created for ACID tables. I want to revisit this 
topic whether or not we need these empty files, but for now, I would go with 
the same behaviour as for ACID tables. 
About the temp file removal, when the direct insert is finished all files which 
are not committed (meaning not in the manifest files) will be deleted prior to 
this call. So there shouldn't be any unnecessary files left at this point. 
I remove this call, and upload a patch to see the result of the pre-commit 
tests. If everything passes, I think it is safe to remove this call in case of 
direct insert.


- Marta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/#review219494
---


On Feb. 18, 2020, 12:21 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71904/
> ---
> 
> (Updated Feb. 18, 2020, 12:21 p.m.)
> 
> 
> Review request for hive, Gopal V and Peter Vary.
> 
> 
> Bugs: HIVE-21164
> https://issues.apache.org/jira/browse/HIVE-21164
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extended the original patch with saving the task attempt ids in the file 
> names and also fixed some bugs in the original patch.
> With this fix, inserting into an ACID table would not use move task to place 
> the generated files into the final directory. It will inserts every files to 
> the final directory and then clean up the files which are not needed (like 
> written by failed task attempts).
> Also fixed the replication tests which failed for the original patch as well.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d3cb60b790 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  da677c7977 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 056cd27496 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
>  31d15fdef9 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
>  c2aa73b5f1 
>   itests/src/test/resources/testconfiguration.properties 1b1bf1147a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
> 9a3258115b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 
> 8980a6292a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 
> c4c56f8477 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> b8a0f0465c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
> 398698ec06 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  2543dc6fc4 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1eb9c12cc8 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 73ca658d9c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 33d3beba46 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
> c102a69f8f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 739f2b654b 
>   ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289583 
>

Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction

2020-02-18 Thread Marta Kuczora via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71904/
---

(Updated Feb. 18, 2020, 12:21 p.m.)


Review request for hive, Gopal V and Peter Vary.


Bugs: HIVE-21164
https://issues.apache.org/jira/browse/HIVE-21164


Repository: hive-git


Description
---

Extended the original patch with saving the task attempt ids in the file names 
and also fixed some bugs in the original patch.
With this fix, inserting into an ACID table would not use move task to place 
the generated files into the final directory. It will inserts every files to 
the final directory and then clean up the files which are not needed (like 
written by failed task attempts).
Also fixed the replication tests which failed for the original patch as well.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d3cb60b790 
  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 da677c7977 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
056cd27496 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
 31d15fdef9 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java
 c2aa73b5f1 
  itests/src/test/resources/testconfiguration.properties 1b1bf1147a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
9a3258115b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 8980a6292a 
  ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java c4c56f8477 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
b8a0f0465c 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 398698ec06 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 2543dc6fc4 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1eb9c12cc8 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
73ca658d9c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 33d3beba46 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
c102a69f8f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
739f2b654b 
  ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289583 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnAddPartition.java c9cb6692df 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 842140815d 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java e56d83158f 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 908ceb43fc 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnConcatenate.java 8676e0db11 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java 66b2b2768b 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java bb55d9fd79 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java ea6b1d9bec 
  ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java af14e628b3 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 83db48e758 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 
2c4b69b2fe 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
48e9afc496 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/CompactorTest.java 
cfd7290762 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestWorker.java 
70ae85c458 
  ql/src/test/queries/clientpositive/tez_acid_union_dynamic_partition.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/tez_acid_union_dynamic_partition_2.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/tez_acid_union_multiinsert.q PRE-CREATION 
  ql/src/test/results/clientpositive/acid_subquery.q.out 1dc1775557 
  ql/src/test/results/clientpositive/create_transactional_full_acid.q.out 
e324d5ec43 
  
ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_dynamic.q.out
 61b0057adb 
  ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out fbf4e481f1 
  ql/src/test/results/clientpositive/llap/insert_overwrite.q.out fbc3326b39 
  ql/src/test/results/clientpositive/llap/mm_al

[jira] [Created] (HIVE-22899) Make sure qtests clean up copied files from test directories

2020-02-18 Thread Zoltan Chovan (Jira) 
Zoltan Chovan created HIVE-22899:


 Summary: Make sure qtests clean up copied files from test 
directories
 Key: HIVE-22899
 URL: https://issues.apache.org/jira/browse/HIVE-22899
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Chovan
Assignee: Zoltan Chovan


Several qtest files are copying schema or test files to the test directories 
(such as ${system:test.tmp.dir} and ${hiveconf:hive.metastore.warehouse.dir}), 
many times without changing the name of the copied file. When the same files is 
copied by another qtest to the same directory the copy and hence the test 
fails. This can lead to flaky tests when any two of these qtests gets scheduled 
to the same batch.

 

In order to avoid these failures, we should make sure the files copied to the 
test dirs have unique names and we should make sure these files are cleaned up 
by the same qtest files that copies the file.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22898) CharsetDecoder race condition in OrcRecordUpdater

2020-02-18 Thread Antal Sinkovits (Jira) 
Antal Sinkovits created HIVE-22898:
--

 Summary: CharsetDecoder race condition in OrcRecordUpdater 
 Key: HIVE-22898
 URL: https://issues.apache.org/jira/browse/HIVE-22898
 Project: Hive
  Issue Type: Bug
Affects Versions: 4.0.0
Reporter: Antal Sinkovits
Assignee: Antal Sinkovits


Instances of CharsetDecoder are not thread safe, causing race condition in 
OrcRecordUpdater



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72119: Extract ValidTxnManager from Driver

2020-02-18 Thread Miklos Gergely 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72119/
---

(Updated Feb. 18, 2020, 8:40 a.m.)


Review request for hive and Zoltan Haindrich.


Summary (updated)
-

Extract ValidTxnManager from Driver


Bugs: HIVE-22868
https://issues.apache.org/jira/browse/HIVE-22868


Repository: hive-git


Description
---

~250 lines in driver are dedicated for the specific task of validating the 
transaction id list, and writing the valid write ids to the configuration. 
These should be put into a separate class.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1f8bc12fde 
  ql/src/java/org/apache/hadoop/hive/ql/ValidTxnManager.java PRE-CREATION 


Diff: https://reviews.apache.org/r/72119/diff/1/


Testing
---

Tests are still passing.


Thanks,

Miklos Gergely

Re: Review Request 72139: Break up DDLSemanticAnalyzer - extract Table storage analyzers

2020-02-18 Thread Zoltan Haindrich 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72139/#review219607
---




ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLSemanticAnalyzerFactory.java
Line 52 (original), 52 (patched)


I don't really like the 2 type/types with defaults annotation.

I think we can remove the default-s and the 2 methods for the same thing
```
  @Retention(RetentionPolicy.RUNTIME)
  public @interface DDLType {
int[] types();
  }

  // and then the following 2 is valid:
  @DDLType(types = 2)
  class a {
  }

  @DDLType(types = { 2, 3 })
  class b {
  }

```
 
another alternative would be to introduce an annotation to encapsulate 
multiple DDLTypes

https://stackoverflow.com/questions/1554112/multiple-annotations-of-the-same-type-on-one-element



ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLSemanticAnalyzerFactory.java
Line 79 (original), 81 (patched)


add duplicate detection; if by mistake somehow a re-registeration may occur 
that could lead to odd behaviour


- Zoltan Haindrich


On Feb. 16, 2020, 7:22 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72139/
> ---
> 
> (Updated Feb. 16, 2020, 7:22 p.m.)
> 
> 
> Review request for hive and Zoltan Haindrich.
> 
> 
> Bugs: HIVE-22895
> https://issues.apache.org/jira/browse/HIVE-22895
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is 
> to refactor it in order to have everything cut into more handleable classes 
> under the package  org.apache.hadoop.hive.ql.exec.ddl:
> 
> have a separate class for each analyzers
> have a package for each operation, containing an analyzer, a description, and 
> an operation, so the amount of classes under a package is more manageable
> Step #14: extract the table storage related analyzers from 
> DDLSemanticAnalyzer, and move them under the new package.
> 
> 
> Diffs
> -
> 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzer.java
>  8277d34731 
>   ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLSemanticAnalyzerFactory.java 
> 119b555d58 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/AbstractAlterTableAnalyzer.java
>  81800fe000 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableArchiveDesc.java
>  9dd6c8ec7d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableArchiveOperation.java
>  248fe0f5a3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableArchiveUtils.java
>  c285405522 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableClusteredByDesc.java
>  a9a4724ea7 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableClusteredByOperation.java
>  c232e66760 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableCompactDesc.java
>  3a512ba6e4 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableCompactOperation.java
>  8e576fa446 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableConcatenateDesc.java
>  5f5bbe4a0d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableConcatenateOperation.java
>  ea21e026bf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableIntoBucketsDesc.java
>  c8d1a599db 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableIntoBucketsOperation.java
>  f7d224bddd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableNotClusteredDesc.java
>  37005f658d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableNotClusteredOperation.java
>  9b4fb32881 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableNotSkewedDesc.java
>  016c18c60e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableNotSkewedOperation.java
>  cb4632d309 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableNotSortedDesc.java
>  30614f2dce 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableNotSortedOperation.java
>  3d3996d0d3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableSetFileFormatDesc.java
>  78ac94b3b5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableSetFileFormatOperation.java
>  5b7c5acbe4 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableSetLocationDesc.java
>  d79a8e4751 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/Al