[jira] [Created] (HIVE-18553) VectorizedParquetReader fails after adding a new column to table

[Vihang Karajgaonkar (JIRA)]

Vihang Karajgaonkar created HIVE-18553:
--

 Summary: VectorizedParquetReader fails after adding a new column 
to table
 Key: HIVE-18553
 URL: https://issues.apache.org/jira/browse/HIVE-18553
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 2.3.2, 3.0.0, 2.4.0
Reporter: Vihang Karajgaonkar


VectorizedParquetReader throws an exception when trying to reading from a 
parquet table on which new columns are added. Steps to reproduce below:

{code}
0: jdbc:hive2://localhost:1/default> desc test_p;
+---++--+
| col_name  | data_type  | comment  |
+---++--+
| t1| tinyint|  |
| t2| tinyint|  |
| i1| int|  |
| i2| int|  |
+---++--+
0: jdbc:hive2://localhost:1/default> set hive.fetch.task.conversion=none;
0: jdbc:hive2://localhost:1/default> set 
hive.vectorized.execution.enabled=true;
0: jdbc:hive2://localhost:1/default> alter table test_p add columns (ts 
timestamp);
0: jdbc:hive2://localhost:1/default> select * from test_p;
Error: Error while processing statement: FAILED: Execution Error, return code 2 
from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2)
{code}

Following exception is seen in the logs

{code}
Caused by: java.lang.IllegalArgumentException: [ts] BINARY is not in the store: 
[[i1] INT32, [i2] INT32, [t1] INT32, [t2] INT32] 3
at 
org.apache.parquet.hadoop.ColumnChunkPageReadStore.getPageReader(ColumnChunkPageReadStore.java:160)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:479)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:432)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:393)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:345)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:88)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:167)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:52)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:229)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:142)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
 ~[hadoop-mapreduce-client-core-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
at 
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185) 
~[hadoop-mapreduce-client-core-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52) 
~[hadoop-mapreduce-client-core-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459) 
~[hadoop-mapreduce-client-core-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) 
~[hadoop-mapreduce-client-core-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
at 
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
 ~[hadoop-mapreduce-client-common-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[?:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
~[?:1.8.0_121]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[?:1.8.0_121]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
~[?:1.8.0_121]
at 

[jira] [Created] (HIVE-18552) Split hive.strict.checks.large.query into two configs, enable partition filter check by default

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-18552:
---

 Summary: Split hive.strict.checks.large.query into two configs, 
enable partition filter check by default
 Key: HIVE-18552
 URL: https://issues.apache.org/jira/browse/HIVE-18552
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Reporter: Sahil Takiar


{{hive.strict.checks.large.query}} controls the strict checks for restricting 
order bys with no limits, and scans of a partitioned table without a filter on 
the partition table.

While both checks prevent "large" queries from being run, they both control 
very different behavior. It would be better if users could control these 
restrictions separately.

Furthermore, many users make the mistake of abusing partitioned tables and 
often end up in a situation where they are running queries that are doing 
full-table scans of partitioned tables. This can lead to lots of issues for 
Hive - e.g. OOM issues because so many partitions are loaded in memory. So it 
would be good if we enabled this restriction by default.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18551) Vectorization: VectorMapOperator tries to write too many vector columns for Hybrid Grace

[Matt McCline (JIRA)]

Matt McCline created HIVE-18551:
---

 Summary: Vectorization: VectorMapOperator tries to write too many 
vector columns for Hybrid Grace
 Key: HIVE-18551
 URL: https://issues.apache.org/jira/browse/HIVE-18551
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Matt McCline
Assignee: Matt McCline
 Fix For: 3.0.0


Code incorrectly uses projectedColumns.length instead of singleRow.length



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18550) Keep the hbase table name property as hbase.table.name

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-18550:
---

 Summary: Keep the hbase table name property as hbase.table.name
 Key: HIVE-18550
 URL: https://issues.apache.org/jira/browse/HIVE-18550
 Project: Hive
  Issue Type: Sub-task
  Components: HBase Handler
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


With hbase 2.0 support, I made some changes to the hbase table name property 
change in HIVE-18366 and HIVE-18202. By checking the logic, seems the change is 
not necessary since hbase.table.name is internal to hive hbase handler. We just 
need to map hbase.table.name to hbase.mapreduce.hfileoutputformat.table.name 
for HiveHFileOutputFormat. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65264: HIVE-18481: Create tests for table related methods (get, list, exists)

[Sahil Takiar]

> On Jan. 24, 2018, 5:50 p.m., Sahil Takiar wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesGetListExists.java
> > Lines 63 (patched)
> > 
> >
> > A lot of the setup and teardown code for all these classes looks pretty 
> > similar. Is it worth creating an abstract class (something like 
> > `AbstractMetaStoreClientTest`) to encapsulate all the common setup / 
> > teardown logic?
> > 
> > We don't have to do this for all the JIRAs in HIVE-18371, since most of 
> > them are already done. But would be good for future tests.
> 
> Peter Vary wrote:
> When we have a junit release with this commit:
> 
> https://github.com/junit-team/junit4/commit/1bf8438b65858565dbb64736bfe13aae9cfc1b5a
> 
> Then we can change the code to something like this:
> ```
>   @Parameterized.Parameters(name = "{0}")
>   public static List getMetaStoreToTest() throws Exception {
> return MetaStoreFactoryForTests.getMetaStores();
>   }
> 
>   public TestTablesList(String name, AbstractMetaStoreService metaStore) 
> throws Exception {
> this.metaStore = metaStore;
> this.metaStore.start();
>   }
> 
>   @AfterParam
>   public static void stopMetaStores() throws Exception {
> metaStore.stop();
>   }
> ```
> Until then we need a way to stop the metastore services at the end of the 
> runs.
> Sidenote: Currently metaStore.stop() does not do anything, since there is 
> no clean way to stop a metastore. So really it is only there to remind us, 
> that we should do something like this.
> 
> Do you think we should remove the extra code, and file a jira to do this, 
> when we have either the junit change in a release, or a way to stop a 
> metastore?
> Thanks,
> Peter

I think its fine for now, its more of a nit. Regardless of the junit fix, can't 
all this logic still be pulled into an abstract class?

Isn't stopping a remote metastore consist of just shutting down the process 
running HMS?


> On Jan. 24, 2018, 5:50 p.m., Sahil Takiar wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesGetListExists.java
> > Lines 218-221 (patched)
> > 
> >
> > It's a bit weird that the embedded metastore and remote metastore throw 
> > different exceptions (speaking broadly here since this applies to a lot of 
> > the tests).
> > 
> > Is there any way to fix that?
> 
> Peter Vary wrote:
> Yeah, this "API" does many weird things :)
> Since the consensus on the dev list was, that we do not want to change 
> the API, I do not see how can we solve this issue:
> - Embedded MetaStore should not throw TProtocolException
> - Remote MetaStore API is defined by the hive_metastore.thrift definition 
> which defines this field as "required"
> We can change the IMetaStoreClient implementation, to check this before 
> sending it to the MetaStore server, but that solution is even worse :(
> 
> So I would skip this for now. What do you think?

Yeah, I was just curious. Its out of scope for this RB. I think making the 
making the two consistent is justifiable, even if it is backwards incompatible. 
We can make the change in Hive 3.0.0. Even if it comes after the HMS separation 
work, would still be good to have a JIRA to track it.


- Sahil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65264/#review196144
---


On Jan. 25, 2018, 1:01 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65264/
> ---
> 
> (Updated Jan. 25, 2018, 1:01 p.m.)
> 
> 
> Review request for hive, Marta Kuczora, Adam Szita, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-18481
> https://issues.apache.org/jira/browse/HIVE-18481
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Create IMetaStoreClient tests to cover the table query methods
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesGetExists.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesList.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65264/diff/4/
> 
> 
> Testing
> ---
> 
> Run the created tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 65264: HIVE-18481: Create tests for table related methods (get, list, exists)

[Sahil Takiar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65264/#review196299
---


Ship it!




Ship It!

- Sahil Takiar


On Jan. 25, 2018, 1:01 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65264/
> ---
> 
> (Updated Jan. 25, 2018, 1:01 p.m.)
> 
> 
> Review request for hive, Marta Kuczora, Adam Szita, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-18481
> https://issues.apache.org/jira/browse/HIVE-18481
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Create IMetaStoreClient tests to cover the table query methods
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesGetExists.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesList.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65264/diff/4/
> 
> 
> Testing
> ---
> 
> Run the created tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

[jira] [Created] (HIVE-18549) Option to create null session handle for hive server.

[Purshotam Shah (JIRA)]

Purshotam Shah created HIVE-18549:
-

 Summary: Option to create null session handle for hive server.
 Key: HIVE-18549
 URL: https://issues.apache.org/jira/browse/HIVE-18549
 Project: Hive
  Issue Type: Bug
Reporter: Purshotam Shah






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18548) Fix log4j import

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-18548:
--

 Summary: Fix log4j import
 Key: HIVE-18548
 URL: https://issues.apache.org/jira/browse/HIVE-18548
 Project: Hive
  Issue Type: Bug
Reporter: Vineet Garg
Assignee: Vineet Garg
 Fix For: 3.0.0


LLAP server code could run into invalid reference to log4j.MDC due to code 
importing {{org.apache.log4j.MDC}} instead of {{org.slf4j.MDC }}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Alternative fix for DB notifications (HIVE-16886)

[Alexander Kolbasov]

While looking at the fix for HIVE-16886, I noticed that DataNucleus
provides a generic way of doing SELECT FOR UPDATE so it is possible to
achieve this with just a single-line change rather then the big fix
introduced by HIVE-16886.

What do people think about redoing the fix by using DataNucleus way?

The fix is to do

  query = pm.newQuery(MNotificationNextId.class);
  query.setSerializeRead(true);


- Alex

[jira] [Created] (HIVE-18547) WM: trigger test may fail

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-18547:
---

 Summary: WM: trigger test may fail
 Key: HIVE-18547
 URL: https://issues.apache.org/jira/browse/HIVE-18547
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


https://builds.apache.org/job/PreCommit-HIVE-Build/8818/testReport/org.apache.hive.jdbc/TestTriggersMoveWorkloadManager/testTriggerMoveAndKill/

Looks like the cluster allocation assignment and WM event creation race somehow
{noformat}
'Event: GET Pool: BI Cluster %: 80.00' expected in STDERR capture, but not 
found.

... 

2018-01-24T15:07:31,746  INFO [Workload management master] tez.WorkloadManager: 
Processing changes for pool BI: [BI, query parallelism 1, fraction of the 
cluster 0.80011920929, fraction used by child pools 0.0, active sessions 0, 
initializing sessions 0]
2018-01-24T15:07:31,746  INFO [Workload management master] tez.WorkloadManager: 
Starting 1 queries in pool [BI, query parallelism 1, fraction of the cluster 
0.80011920929, fraction used by child pools 0.0, active sessions 0, 
initializing sessions 0]
2018-01-24T15:07:31,746  INFO [Workload management master] tez.WorkloadManager: 
Received a session from AM pool sessionId=2be29c62-9f2c-40b7-a5eb-6298baf83a34, 
queueName=default, user=hiveptest, doAs=false, isOpen=true, isDefault=true, 
expires in 588529859ms, WM state poolName=null, clusterFraction=0.0, 
queryId=null, killReason=null
2018-01-24T15:07:31,746  INFO [HiveServer2-Background-Pool: Thread-1377] 
tez.WmEvent: Added WMEvent: EventType: GET EventStartTimestamp: 1516835251746 
elapsedTime: 0 wmTezSessionInfo:SessionId: 2be29c62-9f2c-40b7-a5eb-6298baf83a34 
Pool: BI Cluster %: 0.0
2018-01-24T15:07:31,746  INFO [Workload management master] 
tez.GuaranteedTasksAllocator: Updating 2be29c62-9f2c-40b7-a5eb-6298baf83a34 
with 3 guaranteed tasks
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 65342: HIVE-18546

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65342/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-18546
https://issues.apache.org/jira/browse/HIVE-18546


Repository: hive-git


Description
---

HIVE-18546


Diffs
-

  metastore/scripts/upgrade/derby/048-HIVE-14498.derby.sql 
4ffd054530503681de1c9f6d65f8187fc1b7520d 
  metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql 
6a59b0df712c8a9f9be880cec5fd8c8eddda4a7d 
  metastore/scripts/upgrade/derby/hive-txn-schema-3.0.0.derby.sql 
d72b06cb5866edf93dbcbb20268fc899439e5c43 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 
eb4f0124b5a7829e58d5e9a6a604c201ccea998a 
  metastore/scripts/upgrade/mssql/033-HIVE-14498.mssql.sql 
3a47600bb09e2c20cc12f8759e1287001367604e 
  metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 
c45bb3e323c640223b19831abbf4e806c3019f0b 
  metastore/scripts/upgrade/mysql/048-HIVE-14498.mysql.sql 
986eaf5272eab560fa2f862910aaf74c5332c716 
  metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 
01c995d632d94a8f9cc3f46f94c54290abb3da13 
  metastore/scripts/upgrade/mysql/hive-txn-schema-3.0.0.mysql.sql 
497846f994d431d8717aea36d4ad569892e3c8c3 
  metastore/scripts/upgrade/oracle/048-HIVE-14498.oracle.sql 
0b01e89d92f7f48439024aeb326d675d123f0f8c 
  metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 
e1aee6fb6c84999b17f87f80750582fafeae063f 
  metastore/scripts/upgrade/oracle/hive-txn-schema-3.0.0.oracle.sql 
5411bc47103f901623244bc26c0ace87e10ad2e1 
  metastore/scripts/upgrade/postgres/047-HIVE-14498.postgres.sql 
8d4de8870d93bab49c873cab44e6714b93491744 
  metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 
28cb01684a46aaeea40d7cbe1973d7bc20810988 
  metastore/scripts/upgrade/postgres/hive-txn-schema-3.0.0.postgres.sql 
a81d6eec6d6235706f1225d541f8290971cc6215 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
51ef39057434c41fbe760c547e3bf231e65e4cc0 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 
9b0ffe0e91db05ae623531248f12745266789a11 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
f8126f1604c37f601c9d55b99698d2c0fdf71308 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 
aa95d2fcdcd0b3cede35537a1d7d041ee738e4a8 
  standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 
42bc9297e72ac8fd77352cb786cfed3abf5af59b 
  standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 
8b78230a32d4d4339189c1db4b533ed04ec080af 
  
standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp
 6a2ff6c4c681b2dbaf339b214663212a2e6dab22 
  standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h 
df646a7d1771892e4404be5c4fba183c0f914510 
  standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp 
27f8c0f2fcb24a90be8a44d68947589004286c28 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AbortTxnsRequest.java
 398f8d4e93c6077c110e6469bcd3715fdad5a634 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddDynamicPartitions.java
 2102aa5215598edfe5e5c53d541c4fe02ebc7f09 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddForeignKeyRequest.java
 a2225298e72f708e97324048592c37a308e43514 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddNotNullConstraintRequest.java
 ef23d3025aabb2934f93230ea72c4585dda973e4 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsRequest.java
 13a23182488ebda9ab0f7163fd4d6822c04c975f 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsResult.java
 49ce6e1a6cc38994662f56536c6dd6bd55e67d47 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPrimaryKeyRequest.java
 478032a987e7688741fe55b9732f5ec0e8fc209f 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddUniqueConstraintRequest.java
 b58f39f7b045f3dbaf95df9e28190517280bd8c4 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AggrStats.java
 54ef01f3ce4448f0b404772a30a5b0d61641e3c2 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BasicTxnInfo.java
 f695e5d9bd4f96ff5a3d5b055a05ac00574ce01b 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClearFileMetadataRequest.java
 dbda2ab74128a698452f639b6a7b142b47ca351b 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClientCapabilities.java
 0df33f1c8b29f9dfd94815e0d7bcf6a841e54043 
  

[jira] [Created] (HIVE-18546) Remove unnecessary code introduced in HIVE-14498

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-18546:
--

 Summary: Remove unnecessary code introduced in HIVE-14498
 Key: HIVE-18546
 URL: https://issues.apache.org/jira/browse/HIVE-18546
 Project: Hive
  Issue Type: Bug
  Components: Materialized views
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Hive packaging and the standalone metastore

[Alexander Kolbasov]

Alan,

While continuing shipping HMS with Hive makes sense (at least for a while),
what do you think about somehow separating lib/bin directories created in
the distro so Hive and metastore have a separate set of bin/lib dirs?

- Alex

On Wed, Jan 24, 2018 at 12:16 PM, Alan Gates  wrote:

> In HIVE-17983 I have been working on packaing and start/stop scripts for
> the standalone metastore.  One question this brings up is how Hive will be
> released now, with or without the metastore.  I can see two options:
>
> 1) We continue to ship the metastore with Hive.  Not only does this mean
> the metastore code is in the Hive source code release and the metastore
> jars are in the Hive binary distribution, but scripts like metastore.sh are
> still included in Hive's bin directory, so that Hive admins can still do
> 'hive --service metastore' to start the metastore.  I see the following
> advantages of this:
> a) it is completely backwards compatible;
> b) it is what users would expect (I have installed many databases and never
> been asked to first install a separate package for its data catalog or any
> other essential piece);
> c) this will still be the metastore's most frequent use case for at least
> the near future.
>
> The disadvantage is it is error prone when Hive is set up to connect to a
> separate metastore.  An operator could easily start the metastore in the
> Hive package, not realizing Hive is configured to connect to a different
> one.
>
> 2) We remove the metastore from the packaging completely like we do Hadoop
> and require the user to install it separately.  The advantages and
> disadvantages of this exactly mirror those of option 1.
>
> Based on both the 80/20 rule (most metastore users will still be single
> system Hive users) and the law of least astonishment (people expect a
> database to have a data catalog) I vote for option 1.
>
> Anyone strongly feel we should do 2 instead?
>
> Any other options I haven't considered?
>
> Alan.
>

[GitHub] hive pull request #298: HIVE-18478: Avoiding creation of CM recycle file in ...

[maheshk114]

GitHub user maheshk114 opened a pull request:

https://github.com/apache/hive/pull/298

HIVE-18478: Avoiding creation of CM recycle file in case of temp table

In case of drop table, truncate table, load table etc,  the table info is 
deleted which can cause issues during replication. To solve this, the old files 
are stored in the CM directory to be used by replication later. But for 
temporary tables, replication is not done and thus these files creation is not 
required. So extra checks are added to avoid creation of recycle files in case 
of temp tables.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/maheshk114/hive HIVE-18478

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/298.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #298


commit 2a5dd87618c1a323a230eafe1906a7ad8b9e7af7
Author: Mahesh Kumar Behera 
Date:   2018-01-25T16:06:27Z

HIVE-18478: Avoiding creation of CM recycle file in case of temp table




---

[jira] [Created] (HIVE-18545) Add UDF to parse complex types from json

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-18545:
---

 Summary: Add UDF to parse complex types from json
 Key: HIVE-18545
 URL: https://issues.apache.org/jira/browse/HIVE-18545
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
 Attachments: HIVE-18545.01.patch





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65241: HIVE-18496: Create tests to cover add/alter/drop index methods

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65241/#review196257
---


Ship it!




Ship It!

- Peter Vary


On Jan. 24, 2018, 3:41 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65241/
> ---
> 
> (Updated Jan. 24, 2018, 3:41 p.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18496
> https://issues.apache.org/jira/browse/HIVE-18496
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The following methods of IMetaStoreClient are covered by this test.
> - void createIndex(Index, Table)
> - boolean dropIndex(String, String, String, boolean)
> - void alter_index(String, String, String, Index)
> 
> The test covers not just the happy pathes, but the edge cases as well.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddAlterDropIndexes.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65241/diff/3/
> 
> 
> Testing
> ---
> 
> Run the tests
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Re: Review Request 65219: HIVE-18486: Create tests to cover methods for adding Partitions

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65219/#review196256
---


Ship it!




Ship It!

- Peter Vary


On Jan. 25, 2018, 2:40 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65219/
> ---
> 
> (Updated Jan. 25, 2018, 2:40 p.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18486
> https://issues.apache.org/jira/browse/HIVE-18486
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The following methods of IMetaStoreClient are covered by this test.
> - Partition add_partition(Partition)
> - int add_partitions(List)
> - List add_partitions(List, boolean, boolean)
> 
> 
> The test covers not just the happy pathes, but the edge cases as well.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65219/diff/2/
> 
> 
> Testing
> ---
> 
> Run the tests
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Re: Review Request 65284: Create tests to cover listPartition(s) methods

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65284/#review196255
---


Ship it!




Ship It!

- Peter Vary


On Jan. 25, 2018, 1:47 p.m., Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65284/
> ---
> 
> (Updated Jan. 25, 2018, 1:47 p.m.)
> 
> 
> Review request for hive, Marta Kuczora and Peter Vary.
> 
> 
> Bugs: HIVE-18484
> https://issues.apache.org/jira/browse/HIVE-18484
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Create tests to cover listPartition(s) methods
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestListPartitions.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65284/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adam Szita
> 
>

Re: Review Request 65213: HIVE-18479: Create tests to cover methods for dropping Partitions

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65213/#review196254
---


Ship it!




Ship It!

- Peter Vary


On Jan. 24, 2018, 4:37 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65213/
> ---
> 
> (Updated Jan. 24, 2018, 4:37 p.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18479
> https://issues.apache.org/jira/browse/HIVE-18479
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The following methods of IMetaStoreClient are covered by this test.
> - boolean dropPartition(String, String, List, boolean)
> - boolean dropPartition(String, String, List, PartitionDropOptions)
> - boolean dropPartition(String, String, String, boolean)
> 
> The test covers not just the happy pathes, but the edge cases as well.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDropPartitions.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65213/diff/4/
> 
> 
> Testing
> ---
> 
> Run the tests
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

[jira] [Created] (HIVE-18544) Create tests to cover appendPartition methods

[Marta Kuczora (JIRA)]

Marta Kuczora created HIVE-18544:


 Summary: Create tests to cover appendPartition methods
 Key: HIVE-18544
 URL: https://issues.apache.org/jira/browse/HIVE-18544
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Marta Kuczora
Assignee: Marta Kuczora


The following methods of IMetaStoreClient are covered in this Jira:
{code:java}
- Partition appendPartition(String, String, List)
- Partition appendPartition(String, String, String){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65265: HIVE-18509 Create tests for table manipulation related methods (create, alter, drop)

[Marta Kuczora via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65265/#review196253
---


Ship it!




Ship It!

- Marta Kuczora


On Jan. 23, 2018, 4:54 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65265/
> ---
> 
> (Updated Jan. 23, 2018, 4:54 p.m.)
> 
> 
> Review request for hive, Marta Kuczora, Adam Szita, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-18509
> https://issues.apache.org/jira/browse/HIVE-18509
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Added the new tests.
> Modified AbstractMetaStoreService, so when starting mini-metastore it is 
> possible to provide configuration value overrides for non-metastore config 
> values (like fs trash settings)
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesCreateDropAlterTruncate.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/AbstractMetaStoreService.java
>  ed071f8 
> 
> 
> Diff: https://reviews.apache.org/r/65265/diff/4/
> 
> 
> Testing
> ---
> 
> Run the tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 65219: HIVE-18486: Create tests to cover methods for adding Partitions

[Marta Kuczora via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65219/
---

(Updated Jan. 25, 2018, 2:40 p.m.)


Review request for hive, Peter Vary and Adam Szita.


Changes
---

Addressed the review findings.
- Added tests for creating partitions with default attribute values.
- Added tests for creating partitions without columns and with invalid or 
incomplete colum infos
- Added tests for creating partitions without serde info
- Added tests for creating partitions for external table with and without 
location being set
- Made some helper methods static
- Used the List.newArrayList method where it is needed
- Changed the value of the MAX variable to -1
- Removed the creation time checks where it was not necessary
- Fixed some checkstyle issues


Bugs: HIVE-18486
https://issues.apache.org/jira/browse/HIVE-18486


Repository: hive-git


Description
---

The following methods of IMetaStoreClient are covered by this test.
- Partition add_partition(Partition)
- int add_partitions(List)
- List add_partitions(List, boolean, boolean)


The test covers not just the happy pathes, but the edge cases as well.


Diffs (updated)
-

  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/65219/diff/2/

Changes: https://reviews.apache.org/r/65219/diff/1-2/


Testing
---

Run the tests


Thanks,

Marta Kuczora

Re: Review Request 65219: HIVE-18486: Create tests to cover methods for adding Partitions

[Marta Kuczora via Review Board]

> On Jan. 19, 2018, 4:36 p.m., Adam Szita wrote:
> > Looks good! I added small issues to fix, similarly with the dropPartitions 
> > change: some helper methods could be static (the ones that don't depend on 
> > HMS client certainly) and some could use Lists.newArrayList

Thanks a lot Adam for the review.
I made some helper methods static and also used the Lists.newArrayList where is 
was needed.


> On Jan. 19, 2018, 4:36 p.m., Adam Szita wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
> > Lines 68 (patched)
> > 
> >
> > Again this may be -1 instead, when MAX as a value is not used for any 
> > other reason than returning all partitions

No reason, just missed it. Thanks for the hint, fixed it.


> On Jan. 19, 2018, 4:36 p.m., Adam Szita wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
> > Lines 672 (patched)
> > 
> >
> > This is found multiple times in the code, should we create a method for 
> > it?

I refactored a bit handling the creation time, so these duplications are 
eliminated.


- Marta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65219/#review195825
---


On Jan. 18, 2018, 4:58 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65219/
> ---
> 
> (Updated Jan. 18, 2018, 4:58 p.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18486
> https://issues.apache.org/jira/browse/HIVE-18486
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The following methods of IMetaStoreClient are covered by this test.
> - Partition add_partition(Partition)
> - int add_partitions(List)
> - List add_partitions(List, boolean, boolean)
> 
> 
> The test covers not just the happy pathes, but the edge cases as well.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65219/diff/1/
> 
> 
> Testing
> ---
> 
> Run the tests
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

[GitHub] hive pull request #297: HIVE-18543 Add print sessionid in console

[Tartarus0zm]

GitHub user Tartarus0zm opened a pull request:

https://github.com/apache/hive/pull/297

HIVE-18543 Add print sessionid in console



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Tartarus0zm/hive patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/297.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #297


commit 2af02a96de32ede92921a086b04e68e209a27b39
Author: zhangmang 
Date:   2018-01-25T14:23:51Z

Add print sessionid in console




---

Re: Review Request 65219: HIVE-18486: Create tests to cover methods for adding Partitions

[Marta Kuczora via Review Board]

> On Jan. 19, 2018, 1:56 p.m., Peter Vary wrote:
> > Thanks Marta for the patch.
> > Some questions below.
> > Also I think it would be nice to have test for the default values when 
> > adding a partition.
> > 
> > Thanks,
> > Peter

Thanks a lot Peter for the review.
That's a good idea, added tests for creating partitions with default values.


> On Jan. 19, 2018, 1:56 p.m., Peter Vary wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
> > Lines 236 (patched)
> > 
> >
> > There are some interesting validation in the SD. Maybe we can add test 
> > for it here too.

Good point, thanks. Created tests for some use cases.


> On Jan. 19, 2018, 1:56 p.m., Peter Vary wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
> > Lines 282 (patched)
> > 
> >
> > Is it worth to create a test for an External Table too?

Creating partitions for an external table doesn't really differ from creating 
it for a managed table. But I agree it worth to have a test case for that too. 
I added it.


> On Jan. 19, 2018, 1:56 p.m., Peter Vary wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
> > Lines 700 (patched)
> > 
> >
> > Like above - validation inside the SD might be important as well

Created some test cases for sd attribute validations.


> On Jan. 19, 2018, 1:56 p.m., Peter Vary wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
> > Lines 749 (patched)
> > 
> >
> > Maybe for external table too. And maybe check the locations

Added tests for external table with and without setting the location.


> On Jan. 19, 2018, 1:56 p.m., Peter Vary wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
> > Lines 1117 (patched)
> > 
> >
> > I do not understand this check

If the 'metastore.partition.inherit.table.properties' property is set in the 
metastore config, the partition inherits the listed table parameters.
This property is not set in this test, therefore the partition shouldn't 
inherit the table parameters. That is what this check is for.
But it is true that this is not obvious, so I added a comment to this check.


- Marta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65219/#review195811
---


On Jan. 18, 2018, 4:58 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65219/
> ---
> 
> (Updated Jan. 18, 2018, 4:58 p.m.)
> 
> 
> Review request for hive, Peter Vary and Adam Szita.
> 
> 
> Bugs: HIVE-18486
> https://issues.apache.org/jira/browse/HIVE-18486
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The following methods of IMetaStoreClient are covered by this test.
> - Partition add_partition(Partition)
> - int add_partitions(List)
> - List add_partitions(List, boolean, boolean)
> 
> 
> The test covers not just the happy pathes, but the edge cases as well.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65219/diff/1/
> 
> 
> Testing
> ---
> 
> Run the tests
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

[GitHub] hive pull request #295: HIVE-15631 When the Hive client is started, the sess...

[Tartarus0zm]

Github user Tartarus0zm closed the pull request at:

https://github.com/apache/hive/pull/295


---

[jira] [Created] (HIVE-18543) Add print sessionid in console

[tartarus (JIRA)]

tartarus created HIVE-18543:
---

 Summary: Add print sessionid in console
 Key: HIVE-18543
 URL: https://issues.apache.org/jira/browse/HIVE-18543
 Project: Hive
  Issue Type: Improvement
  Components: CLI, Clients
Affects Versions: 2.3.2
 Environment: CentOS6.5

Hive-1.2.1

Hive-2.3.2
Reporter: tartarus
Assignee: tartarus
 Fix For: 3.0.0


Hive client log file already contains sessionid information, but the console 
does not have sessionid information, the user can not be associated with the 
log well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18542) Create tests to cover getTableMeta method

[Adam Szita (JIRA)]

Adam Szita created HIVE-18542:
-

 Summary: Create tests to cover getTableMeta method
 Key: HIVE-18542
 URL: https://issues.apache.org/jira/browse/HIVE-18542
 Project: Hive
  Issue Type: Sub-task
Reporter: Adam Szita
Assignee: Adam Szita






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65284: Create tests to cover listPartition(s) methods

[Marta Kuczora via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65284/#review196246
---


Ship it!




Ship It!

- Marta Kuczora


On Jan. 25, 2018, 1:47 p.m., Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65284/
> ---
> 
> (Updated Jan. 25, 2018, 1:47 p.m.)
> 
> 
> Review request for hive, Marta Kuczora and Peter Vary.
> 
> 
> Bugs: HIVE-18484
> https://issues.apache.org/jira/browse/HIVE-18484
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Create tests to cover listPartition(s) methods
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestListPartitions.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65284/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adam Szita
> 
>

Re: Review Request 65217: Create tests to cover alterPartition and renamePartition methods

[Adam Szita via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65217/
---

(Updated Jan. 25, 2018, 1:52 p.m.)


Review request for hive, Marta Kuczora and Peter Vary.


Changes
---

Checkstyle fixes


Bugs: HIVE-18468
https://issues.apache.org/jira/browse/HIVE-18468


Repository: hive-git


Description
---

Adding IMetastoreClient API tests that cover alterPartition ande 
renamePartition methods


Diffs (updated)
-

  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/65217/diff/4/

Changes: https://reviews.apache.org/r/65217/diff/3-4/


Testing
---


Thanks,

Adam Szita

Re: Review Request 65285: Create tests to cover getPartition(s) methods

[Adam Szita via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65285/
---

(Updated Jan. 25, 2018, 1:51 p.m.)


Review request for hive, Marta Kuczora and Peter Vary.


Bugs: HIVE-18483
https://issues.apache.org/jira/browse/HIVE-18483


Repository: hive-git


Description
---

Create tests to cover getPartition(s) methods


Diffs (updated)
-

  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestGetPartitions.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/65285/diff/3/

Changes: https://reviews.apache.org/r/65285/diff/2-3/


Testing
---


Thanks,

Adam Szita

Re: Review Request 65284: Create tests to cover listPartition(s) methods

[Adam Szita via Review Board]

> On Jan. 25, 2018, 11:37 a.m., Marta Kuczora wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestListPartitions.java
> > Lines 762-765 (patched)
> > 
> >
> > Is this the same use case as in the 
> > testListPartitionsByFilterEmptyFilter test? If it is, I think it is enough 
> > to have only one test for this use case.

Removed redundant case


> On Jan. 25, 2018, 11:37 a.m., Marta Kuczora wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestListPartitions.java
> > Lines 1028-1031 (patched)
> > 
> >
> > What is the expected behaviour in this case? Should the 
> > getNumPartitionsByFilter return 0 or an other value?

Added missing assertion


- Adam


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65284/#review196236
---


On Jan. 25, 2018, 1:47 p.m., Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65284/
> ---
> 
> (Updated Jan. 25, 2018, 1:47 p.m.)
> 
> 
> Review request for hive, Marta Kuczora and Peter Vary.
> 
> 
> Bugs: HIVE-18484
> https://issues.apache.org/jira/browse/HIVE-18484
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Create tests to cover listPartition(s) methods
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestListPartitions.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65284/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adam Szita
> 
>

Re: Review Request 65284: Create tests to cover listPartition(s) methods

[Adam Szita via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65284/
---

(Updated Jan. 25, 2018, 1:47 p.m.)


Review request for hive, Marta Kuczora and Peter Vary.


Changes
---

Addressing Marta's comments


Bugs: HIVE-18484
https://issues.apache.org/jira/browse/HIVE-18484


Repository: hive-git


Description
---

Create tests to cover listPartition(s) methods


Diffs (updated)
-

  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestListPartitions.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/65284/diff/3/

Changes: https://reviews.apache.org/r/65284/diff/2-3/


Testing
---


Thanks,

Adam Szita

[jira] [Created] (HIVE-18541) Secure HS2 web ui ith PAM

[Oleksiy Sayankin (JIRA)]

Oleksiy Sayankin created HIVE-18541:
---

 Summary: Secure HS2 web ui ith PAM
 Key: HIVE-18541
 URL: https://issues.apache.org/jira/browse/HIVE-18541
 Project: Hive
  Issue Type: Sub-task
Reporter: Oleksiy Sayankin
Assignee: Oleksiy Sayankin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65264: HIVE-18481: Create tests for table related methods (get, list, exists)

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65264/
---

(Updated Jan. 25, 2018, 1:01 p.m.)


Review request for hive, Marta Kuczora, Adam Szita, and Vihang Karajgaonkar.


Changes
---

Addessed Sahil's comments


Bugs: HIVE-18481
https://issues.apache.org/jira/browse/HIVE-18481


Repository: hive-git


Description
---

Create IMetaStoreClient tests to cover the table query methods


Diffs (updated)
-

  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesGetExists.java
 PRE-CREATION 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesList.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/65264/diff/4/

Changes: https://reviews.apache.org/r/65264/diff/3-4/


Testing
---

Run the created tests


Thanks,

Peter Vary

[jira] [Created] (HIVE-18540) remove logic for wide terminal to display in-place updates

[Song Jun (JIRA)]

Song Jun created HIVE-18540:
---

 Summary:  remove logic for wide terminal to display in-place 
updates
 Key: HIVE-18540
 URL: https://issues.apache.org/jira/browse/HIVE-18540
 Project: Hive
  Issue Type: Improvement
Affects Versions: 2.3.2
Reporter: Song Jun
Assignee: Song Jun


Hive Shell/beeline run Tez job, when the terminal wide is lower than 95, there 
will be no output to the console, because of the following limit logic in class 
InPlaceUpdate:
{code:java}
public static boolean canRenderInPlace(HiveConf conf) { boolean inPlaceUpdates 
= HiveConf.getBoolVar(conf, HiveConf.ConfVars.TEZ_EXEC_INPLACE_PROGRESS); // we 
need at least 80 chars wide terminal to display in-place updates properly 
return inPlaceUpdates && isUnixTerminal() && TerminalFactory.get().getWidth() 
>= MIN_TERMINAL_WIDTH; }{code}
 

If someone do not know this logic ,and the terminal is not enough wide, he will 
think something wrong and do know how to fix it.

 

and I think it is better to remove the trouble logic.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65264: HIVE-18481: Create tests for table related methods (get, list, exists)

[Peter Vary via Review Board]

> On Jan. 24, 2018, 5:50 p.m., Sahil Takiar wrote:
> >

Thanks for the review Sahil!


> On Jan. 24, 2018, 5:50 p.m., Sahil Takiar wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesGetListExists.java
> > Lines 63 (patched)
> > 
> >
> > A lot of the setup and teardown code for all these classes looks pretty 
> > similar. Is it worth creating an abstract class (something like 
> > `AbstractMetaStoreClientTest`) to encapsulate all the common setup / 
> > teardown logic?
> > 
> > We don't have to do this for all the JIRAs in HIVE-18371, since most of 
> > them are already done. But would be good for future tests.

When we have a junit release with this commit:
https://github.com/junit-team/junit4/commit/1bf8438b65858565dbb64736bfe13aae9cfc1b5a

Then we can change the code to something like this:
```
  @Parameterized.Parameters(name = "{0}")
  public static List getMetaStoreToTest() throws Exception {
return MetaStoreFactoryForTests.getMetaStores();
  }

  public TestTablesList(String name, AbstractMetaStoreService metaStore) throws 
Exception {
this.metaStore = metaStore;
this.metaStore.start();
  }

  @AfterParam
  public static void stopMetaStores() throws Exception {
metaStore.stop();
  }
```
Until then we need a way to stop the metastore services at the end of the runs.
Sidenote: Currently metaStore.stop() does not do anything, since there is no 
clean way to stop a metastore. So really it is only there to remind us, that we 
should do something like this.

Do you think we should remove the extra code, and file a jira to do this, when 
we have either the junit change in a release, or a way to stop a metastore?
Thanks,
Peter


> On Jan. 24, 2018, 5:50 p.m., Sahil Takiar wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesGetListExists.java
> > Lines 218-221 (patched)
> > 
> >
> > It's a bit weird that the embedded metastore and remote metastore throw 
> > different exceptions (speaking broadly here since this applies to a lot of 
> > the tests).
> > 
> > Is there any way to fix that?

Yeah, this "API" does many weird things :)
Since the consensus on the dev list was, that we do not want to change the API, 
I do not see how can we solve this issue:
- Embedded MetaStore should not throw TProtocolException
- Remote MetaStore API is defined by the hive_metastore.thrift definition which 
defines this field as "required"
We can change the IMetaStoreClient implementation, to check this before sending 
it to the MetaStore server, but that solution is even worse :(

So I would skip this for now. What do you think?


> On Jan. 24, 2018, 5:50 p.m., Sahil Takiar wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesGetListExists.java
> > Lines 272 (patched)
> > 
> >
> > since only one exception is thrown, can use `expected` annotation.

I have missed this. Thanks!


> On Jan. 24, 2018, 5:50 p.m., Sahil Takiar wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesGetListExists.java
> > Lines 338 (patched)
> > 
> >
> > should we just create a separate class for the list table APIs? they 
> > use a different set of tables anyway, and it might be good to split up the 
> > different filter tests

Good idea!
Done


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65264/#review196144
---


On Jan. 23, 2018, 11:18 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65264/
> ---
> 
> (Updated Jan. 23, 2018, 11:18 a.m.)
> 
> 
> Review request for hive, Marta Kuczora, Adam Szita, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-18481
> https://issues.apache.org/jira/browse/HIVE-18481
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Create IMetaStoreClient tests to cover the table query methods
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesGetListExists.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65264/diff/3/
> 
> 
> Testing
> ---
> 
> Run the created tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 65284: Create tests to cover listPartition(s) methods

[Marta Kuczora via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65284/#review196236
---



Thanks a lot Adam for the patch. It looks really good to me, I just have few 
questions.


standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestListPartitions.java
Lines 762-765 (patched)


Is this the same use case as in the testListPartitionsByFilterEmptyFilter 
test? If it is, I think it is enough to have only one test for this use case.



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestListPartitions.java
Lines 871-873 (patched)


The same as above. If it is the same use case as in 
testListPartitionSpecsByFilterEmptyFilter, I think it would enought to have 
only the one of them.



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestListPartitions.java
Lines 1028-1031 (patched)


What is the expected behaviour in this case? Should the 
getNumPartitionsByFilter return 0 or an other value?


- Marta Kuczora


On Jan. 24, 2018, 11:22 a.m., Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65284/
> ---
> 
> (Updated Jan. 24, 2018, 11:22 a.m.)
> 
> 
> Review request for hive, Marta Kuczora and Peter Vary.
> 
> 
> Bugs: HIVE-18484
> https://issues.apache.org/jira/browse/HIVE-18484
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Create tests to cover listPartition(s) methods
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestListPartitions.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65284/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adam Szita
> 
>

Re: Review Request 65285: Create tests to cover getPartition(s) methods

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65285/#review196234
---


Ship it!




Ship It!

- Peter Vary


On Jan. 24, 2018, 2:02 p.m., Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65285/
> ---
> 
> (Updated Jan. 24, 2018, 2:02 p.m.)
> 
> 
> Review request for hive, Marta Kuczora and Peter Vary.
> 
> 
> Bugs: HIVE-18483
> https://issues.apache.org/jira/browse/HIVE-18483
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Create tests to cover getPartition(s) methods
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestGetPartitions.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65285/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adam Szita
> 
>

[jira] [Created] (HIVE-18539) optimize bootstrap load failures when dbName is a pattern

[anishek (JIRA)]

anishek created HIVE-18539:
--

 Summary: optimize bootstrap load failures when dbName is a pattern
 Key: HIVE-18539
 URL: https://issues.apache.org/jira/browse/HIVE-18539
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: anishek


currently on failure of boostrap load, the strategy is to clear the replica 
warehouse of data + metadata and restart the whole bootstrap replication load 
process again. if multiple databases are part of the dump then we should have 
the ability to clear data only for the database which has failed and touch 
other successfully loaded databases.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18538) Updating last.repl.id on replica warehouse with db patterns now allowed

[anishek (JIRA)]

anishek created HIVE-18538:
--

 Summary: Updating last.repl.id on replica warehouse with db 
patterns now allowed 
 Key: HIVE-18538
 URL: https://issues.apache.org/jira/browse/HIVE-18538
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: anishek
 Fix For: 3.0.0


Currently for bootstrap load we dynamically generate the task DAG at runtime 
rather than generate the whole task graph at compile / semantic analysis time. 
The number of vertex as part of this DAG is controlled via 
"hive.repl.approx.max.load.tasks".  Till now we only have tested with 
replication with one database and providing the destination database name in 
the repl load command, the ability to update the last.repl.id is dependent on 
this. Given we want to move to a db regex for dumping data we have to rethink 
this feature. 





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65285: Create tests to cover getPartition(s) methods

[Marta Kuczora via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65285/#review196223
---


Ship it!




Ship It!

- Marta Kuczora


On Jan. 24, 2018, 2:02 p.m., Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65285/
> ---
> 
> (Updated Jan. 24, 2018, 2:02 p.m.)
> 
> 
> Review request for hive, Marta Kuczora and Peter Vary.
> 
> 
> Bugs: HIVE-18483
> https://issues.apache.org/jira/browse/HIVE-18483
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Create tests to cover getPartition(s) methods
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestGetPartitions.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65285/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adam Szita
> 
>

Re: Review Request 65217: Create tests to cover alterPartition and renamePartition methods

[Marta Kuczora via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65217/#review196222
---


Ship it!




Ship It!

- Marta Kuczora


On Jan. 23, 2018, 1:38 p.m., Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65217/
> ---
> 
> (Updated Jan. 23, 2018, 1:38 p.m.)
> 
> 
> Review request for hive, Marta Kuczora and Peter Vary.
> 
> 
> Bugs: HIVE-18468
> https://issues.apache.org/jira/browse/HIVE-18468
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Adding IMetastoreClient API tests that cover alterPartition ande 
> renamePartition methods
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65217/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adam Szita
> 
>