[jira] [Created] (HIVE-20929) The method HiveMetaStoreClient.getValidWriteIds(String fullTableName) failed to execute.

2018-11-15 Thread weizheng (JIRA) 
weizheng created HIVE-20929:
---

 Summary: The method HiveMetaStoreClient.getValidWriteIds(String 
fullTableName) failed to execute.
 Key: HIVE-20929
 URL: https://issues.apache.org/jira/browse/HIVE-20929
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Affects Versions: 3.1.0
 Environment: hadoop3.1.0

hive3.1.0
Reporter: weizheng
Assignee: weizheng


When I use the getValidWriteIds(String fullTableName) method of 
HiveMetaStoreClient to get writeIds, I get the errors:

Exception in thread "main" org.apache.thrift.protocol.TProtocolException: 
Required field 'validTxnList' is unset! 
Struct:GetValidWriteIdsRequest(fullTableNames:[default.w2], validTxnList:null)
 at 
org.apache.hadoop.hive.metastore.api.GetValidWriteIdsRequest.validate(GetValidWriteIdsRequest.java:396)
 at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_valid_write_ids_args.validate(ThriftHiveMetastore.java)
 at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_valid_write_ids_args$get_valid_write_ids_argsStandardScheme.write(ThriftHiveMetastore.java)
 at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_valid_write_ids_args$get_valid_write_ids_argsStandardScheme.write(ThriftHiveMetastore.java)
 at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_valid_write_ids_args.write(ThriftHiveMetastore.java)
 at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:71)
 at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
 at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_valid_write_ids(ThriftHiveMetastore.java:5443)
 at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_valid_write_ids(ThriftHiveMetastore.java:5435)
 at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getValidWriteIds(HiveMetaStoreClient.java:2581)
 at com.inspur.mutation.test.TableOperate.getRowsMap(TableOperate.java:521)
 at com.inspur.procuder.GenerateData.main(GenerateData.java:65)

 

So I can only use the following methods instead：

ValidWriteIdList writeIds;
 tablesList.add(AcidUtils.getFullTableName(table.getDbName(), 
table.getTableName()));
 List writeIdList = 
metaStoreClient.getValidWriteIds(tablesList,txns.toString());
 writeIds = TxnUtils.createValidReaderWriteIdList(writeIdList.get(0));



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20928) NPE in StatsUtils for complex type

2018-11-15 Thread Rajesh Balamohan (JIRA) 
Rajesh Balamohan created HIVE-20928:
---

 Summary: NPE in StatsUtils for complex type
 Key: HIVE-20928
 URL: https://issues.apache.org/jira/browse/HIVE-20928
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.3.4
Reporter: Rajesh Balamohan



{noformat}
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.getWritableSize(StatsUtils.java:1147)
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.getSizeOfMap(StatsUtils.java:1108)
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.getSizeOfComplexTypes(StatsUtils.java:978)
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.getAvgColLenOf(StatsUtils.java:916)
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatisticsFromExpression(StatsUtils.java:1374)
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatisticsFromExprMap(StatsUtils.java:1197)
at 
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$GroupByStatsRule.process(StatsRulesProcFactory.java:1009)
at 
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
at 
org.apache.hadoop.hive.ql.lib.LevelOrderWalker.walk(LevelOrderWalker.java:143)
at 
org.apache.hadoop.hive.ql.lib.LevelOrderWalker.startWalking(LevelOrderWalker.java:122)
at 
org.apache.hadoop.hive.ql.optimizer.stats.annotation.AnnotateWithStatistics.transform(AnnotateWithStatistics.java:78)
at 
org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.runStatsAnnotation(SparkCompiler.java:240)
{noformat}

Issue should be there in master as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20927) The beeline has a dead cycle, resulting in high CPU.

2018-11-15 Thread yinghua_zh (JIRA) 
yinghua_zh created HIVE-20927:
-

 Summary: The beeline has a dead cycle, resulting in high CPU.
 Key: HIVE-20927
 URL: https://issues.apache.org/jira/browse/HIVE-20927
 Project: Hive
  Issue Type: Bug
Reporter: yinghua_zh


When beeline executes the command and places it for a period of time, it is 
found that the CPU of the beeline process is very high. It is the code that has 
a dead loop.The stack is as follows:

"Thread-4" #34 daemon prio=5 os_prio=0 tid=0x7fbd6636d800 nid=0x162be 
sleeping[0x7fbd2e042000]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
- locked <0x00076edb39d8> (a java.sql.SQLException)
at java.lang.Throwable.(Throwable.java:265)
at java.lang.Exception.(Exception.java:66)
at java.sql.SQLException.(SQLException.java:119)
at org.apache.hive.jdbc.HiveStatement.checkConnection(HiveStatement.java:377)
at org.apache.hive.jdbc.HiveStatement.getQueryLog(HiveStatement.java:865)
at org.apache.hive.jdbc.HiveStatement.getQueryLog(HiveStatement.java:849)
at 
org.apache.hive.beeline.Commands$LogRunnable.updateQueryLog(Commands.java:1213)
at org.apache.hive.beeline.Commands$LogRunnable.run(Commands.java:1221)
at java.lang.Thread.run(Thread.java:745)

 

The dead cycle code is as is as follows:

@Override public void run() {
 while (hiveStatement.hasMoreLogs()) {
 try {
 updateQueryLog();
 Thread.sleep(queryProgressInterval);
 } catch (SQLException e) {
 commands.error(new SQLWarning(e));
 } catch (InterruptedException e) {
 commands.debug("Getting log thread is interrupted, since query is done!");
 commands.showRemainingLogsIfAny(hiveStatement);
 }
 }
 }
 }

   

 

when the mehtod (checkConnection("getQueryLog");)occurs exception,the condition 
"hiveStatement.hasMoreLogs()" is still true,exist the dead lop,but the thread 
sleep code is not be run,so the CPU is high!

 

 

 

 

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69367: Query based compactor for full CRUD Acid tables

2018-11-15 Thread Eugene Koifman 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69367/#review210589
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 2685 (patched)


"And minor compaction will be disabled." - should make sure Initiator 
doesn't start minor and that Alter Table commands requesting Minor are no-op or 
throw so that these don't get into the compactor queue.  We should also, 
perhaps think about how Initiator triggers Major compactions - are current 
config params adequate?  Should do at least the 2nd part in a follow up jira, 
maybe both.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java
Line 180 (original), 183 (patched)


I guess all this should be no-op for compactor since it only looks at 1 
partition at a time and for acid serde and IF/OF don't change.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java
Lines 197 (patched)


bucketSplitMultiMap?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java
Lines 206 (patched)


the error should include table name if easily available here or if not 
maybe a file path from any of the splits...



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java
Lines 214 (patched)


should we assert that schemaSplitMultiMap has size=1 since that is what we 
expect for compactor?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java
Lines 276 (patched)


Add a comment that this is trully a bucketId (rather than bucket property - 
BucketCodec.java since 3.0) that is derived from file name

WriteId is also from containing file name and for files that have min/max 
wrieid, it's the starting one.  Now that I look at the code in 
TransactionMetadata.findWriteIDForSynthetcRowIDs() - the assert there will 
throw.  It should be removed since where we have to handle files that come from 
compacted dirs so min <> max for all deltas.

maybe these comments should be on OrcSplit where getter methods are defined.



ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java
Lines 68 (patched)


mark these transient for clarity since we don't serialize them



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 245 (patched)


Ideally this should be prevented before it gets into the compction_queue. 
throwing here will cause failed compactions to accumulate in SHOW COMPACTIONS 
and prevent auto-scheduling of new ones.



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 399 (patched)


should this be in a finally{}?  SessionState is threadLocal so it may get 
reused... or do we shutdown the session each time?



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 481 (patched)


current write id should always be the same as original.  Only delete event 
can have these be different but major compaction absorbs delete events.



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 503 (patched)


what's the value of specifying location for tmp table?  I'm surprised it's 
even legal.  Would this be a security hole potentially?



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 510 (patched)


why overwrite?



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 513 (patched)


why do you need partition key/values in the query? we are always reading a 
single partition.  This is achieved by getAcidState() which takes partition dir 
as input (i.e. all the files it returns are within a given partition)



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 542 (patched)


need to think about this.  maybe it's ok...



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 565 (patched)


there should be something in AcidUtils to parse original bucket file name


- Eugene Koifman


On Nov. 15, 2018, 4:59 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply,

Review Request 69367: Query based compactor for full CRUD Acid tables

2018-11-15 Thread Vaibhav Gumashta 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69367/
---

Review request for hive and Eugene Koifman.


Bugs: HIVE-20699
https://issues.apache.org/jira/browse/HIVE-20699


Repository: hive-git


Description
---

https://jira.apache.org/jira/browse/HIVE-20699


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 65264f323f 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
40dd992455 
  pom.xml 26b662e4c3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java 7f8bd229a6 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 4d55592b63 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
92c74e1d06 


Diff: https://reviews.apache.org/r/69367/diff/1/


Testing
---


Thanks,

Vaibhav Gumashta

[jira] [Created] (HIVE-20926) Semi join reduction hint fails when bloom filter entries are high or when there are no stats

2018-11-15 Thread Vineet Garg (JIRA) 
Vineet Garg created HIVE-20926:
--

 Summary: Semi join reduction hint fails when bloom filter entries 
are high or when there are no stats
 Key: HIVE-20926
 URL: https://issues.apache.org/jira/browse/HIVE-20926
 Project: Hive
  Issue Type: Bug
Affects Versions: 4.0.0
Reporter: Vineet Garg
Assignee: Vineet Garg






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69341: HIVE-20891: Call alter_partition in batch when dynamically loading partitions

2018-11-15 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69341/#review210572
---




ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 2350 (patched)


You might consider using a custom thread pool.


- Denys Kuzmenko


On Nov. 15, 2018, 8:52 a.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69341/
> ---
> 
> (Updated Nov. 15, 2018, 8:52 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20891: Call alter_partition in batch when dynamically loading partitions
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
> e8f362357537e73502f743a9df189dec9be2da5d 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> e185bf49d42da9d1497643c20bbd71edaf071bf1 
> 
> 
> Diff: https://reviews.apache.org/r/69341/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 69341: HIVE-20891: Call alter_partition in batch when dynamically loading partitions

2018-11-15 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69341/#review210571
---




ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Line 2310 (original), 2319 (patched)


This method could reuse new partitions method to avoid code duplication.


- Denys Kuzmenko


On Nov. 15, 2018, 8:52 a.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69341/
> ---
> 
> (Updated Nov. 15, 2018, 8:52 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20891: Call alter_partition in batch when dynamically loading partitions
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
> e8f362357537e73502f743a9df189dec9be2da5d 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> e185bf49d42da9d1497643c20bbd71edaf071bf1 
> 
> 
> Diff: https://reviews.apache.org/r/69341/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 69341: HIVE-20891: Call alter_partition in batch when dynamically loading partitions

2018-11-15 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69341/#review210570
---




ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 2350 (patched)


I would extract result to liocal variable for better readability.


- Denys Kuzmenko


On Nov. 15, 2018, 8:52 a.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69341/
> ---
> 
> (Updated Nov. 15, 2018, 8:52 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20891: Call alter_partition in batch when dynamically loading partitions
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
> e8f362357537e73502f743a9df189dec9be2da5d 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> e185bf49d42da9d1497643c20bbd71edaf071bf1 
> 
> 
> Diff: https://reviews.apache.org/r/69341/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 69341: HIVE-20891: Call alter_partition in batch when dynamically loading partitions

2018-11-15 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69341/#review210569
---




ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java
Lines 90 (patched)


3 first method params could be grouped into Table object.


- Denys Kuzmenko


On Nov. 15, 2018, 8:52 a.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69341/
> ---
> 
> (Updated Nov. 15, 2018, 8:52 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20891: Call alter_partition in batch when dynamically loading partitions
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
> e8f362357537e73502f743a9df189dec9be2da5d 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> e185bf49d42da9d1497643c20bbd71edaf071bf1 
> 
> 
> Diff: https://reviews.apache.org/r/69341/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 69341: HIVE-20891: Call alter_partition in batch when dynamically loading partitions

2018-11-15 Thread Denys Kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69341/#review210568
---




ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 2692 (patched)


By default parallelStream uses the ForkJoinPool.commonPool(), a Thread Pool 
shared by the entire application. You might consider creating a custom one.


- Denys Kuzmenko


On Nov. 15, 2018, 8:52 a.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69341/
> ---
> 
> (Updated Nov. 15, 2018, 8:52 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20891: Call alter_partition in batch when dynamically loading partitions
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
> e8f362357537e73502f743a9df189dec9be2da5d 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> e185bf49d42da9d1497643c20bbd71edaf071bf1 
> 
> 
> Diff: https://reviews.apache.org/r/69341/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

[jira] [Created] (HIVE-20925) test

2018-11-15 Thread Sudarshan Bhide (JIRA) 
Sudarshan Bhide created HIVE-20925:
--

 Summary: test
 Key: HIVE-20925
 URL: https://issues.apache.org/jira/browse/HIVE-20925
 Project: Hive
  Issue Type: Bug
Reporter: Sudarshan Bhide


test



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: HIVE-20420 Provide a fallback authorizer when no other authorizer is in use

2018-11-15 Thread Oleksiy S 
thank you.

On Wed, Nov 14, 2018 at 7:44 PM Thejas Nair  wrote:

> This was for CVE-2018-11777.
> You can find more details in description of CVE-2018-11777
>
> On Wed, Nov 14, 2018 at 3:40 AM Oleksiy S 
> wrote:
> >
> > Guys, could you help with this new feature? HIVE-20420
> >
> > I see no docs, no use cases, just nothing. Thanks.
> >
> > --
> > Oleksiy
>


-- 
Oleksiy

[jira] [Created] (HIVE-20924) Property 'hive.driver.parallel.compilation.global.limit' should be immutable at runtime

2018-11-15 Thread Denys Kuzmenko (JIRA) 
Denys Kuzmenko created HIVE-20924:
-

 Summary: Property 'hive.driver.parallel.compilation.global.limit' 
should be immutable at runtime
 Key: HIVE-20924
 URL: https://issues.apache.org/jira/browse/HIVE-20924
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Denys Kuzmenko






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20923) masking_12 qtest expects wrong inputs

2018-11-15 Thread Karen Coppage (JIRA) 
Karen Coppage created HIVE-20923:


 Summary: masking_12 qtest expects wrong inputs
 Key: HIVE-20923
 URL: https://issues.apache.org/jira/browse/HIVE-20923
 Project: Hive
  Issue Type: Bug
  Components: Query Planning, Test, Tests, Views
Reporter: Karen Coppage


It appears this qtest expects the wrong behavior, and should fail but doesn't.

Below Queries 3 and 4 should not include default@src as Input 

Query 1
{noformat}
PREHOOK: query: create table `masking_test_subq_n1` as select cast(key as int) 
as key, value from src
PREHOOK: type: CREATETABLE_AS_SELECT
PREHOOK: Input: default@src
PREHOOK: Output: database:default
PREHOOK: Output: default@masking_test_subq_n1
POSTHOOK: query: create table `masking_test_subq_n1` as select cast(key as int) 
as key, value from src
POSTHOOK: type: CREATETABLE_AS_SELECT
POSTHOOK: Input: default@src
POSTHOOK: Output: database:default
POSTHOOK: Output: default@masking_test_subq_n1
POSTHOOK: Lineage: masking_test_subq_n1.key EXPRESSION 
[(src)src.FieldSchema(name:key, type:string, comment:default), ]
POSTHOOK: Lineage: masking_test_subq_n1.value SIMPLE 
[(src)src.FieldSchema(name:value, type:string, comment:default), ]{noformat}
Query 2
{noformat}
PREHOOK: query: create view `v1_n9` as select * from `masking_test_subq_n1`
PREHOOK: type: CREATEVIEW
PREHOOK: Input: default@masking_test_subq_n1
PREHOOK: Output: database:default
PREHOOK: Output: default@v1_n9
POSTHOOK: query: create view `v1_n9` as select * from `masking_test_subq_n1`
POSTHOOK: type: CREATEVIEW
POSTHOOK: Input: default@masking_test_subq_n1
POSTHOOK: Output: database:default
POSTHOOK: Output: default@v1_n9
POSTHOOK: Lineage: v1_n9.key SIMPLE 
[(masking_test_subq_n1)masking_test_subq_n1.FieldSchema(name:key, type:int, 
comment:null), ]
POSTHOOK: Lineage: v1_n9.value SIMPLE 
[(masking_test_subq_n1)masking_test_subq_n1.FieldSchema(name:value, 
type:string, comment:null), ]
{noformat}
Query 3 (the explain output here describes a TableScan on table src)
{noformat}
PREHOOK: query: explain
select * from `v1_n9`
limit 20
PREHOOK: type: QUERY
PREHOOK: Input: default@masking_test_subq_n1
PREHOOK: Input: default@src
PREHOOK: Input: default@v1_n9
 A masked pattern was here 
POSTHOOK: query: explain
select * from `v1_n9`
limit 20
POSTHOOK: type: QUERY
POSTHOOK: Input: default@masking_test_subq_n1
POSTHOOK: Input: default@src
POSTHOOK: Input: default@v1_n9
{noformat}
Query 4
{noformat}
PREHOOK: query: select * from `v1_n9`
limit 20
PREHOOK: type: QUERY
PREHOOK: Input: default@masking_test_subq_n1
PREHOOK: Input: default@src
PREHOOK: Input: default@v1_n9
 A masked pattern was here 
POSTHOOK: query: select * from `v1_n9`
limit 20
POSTHOOK: type: QUERY
POSTHOOK: Input: default@masking_test_subq_n1
POSTHOOK: Input: default@src
POSTHOOK: Input: default@v1_n9
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69339: Break up UpdateDeleteSemanticAnalyzer

2018-11-15 Thread Miklos Gergely 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69339/
---

(Updated Nov. 15, 2018, 9:43 a.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Fixed some checkstyle issues.


Bugs: HIVE-20919
https://issues.apache.org/jira/browse/HIVE-20919


Repository: hive-git


Description
---

UpdateDeleteSemanticAnalyzer handles update, delete, acid export and merge 
queries by rewriting them to a different form. This is a clear violation of 
SRP, and therefore needs to be refactored. An abstract ancestor needs to take 
the common part, and each of the specific tasks should be handled by a separate 
class.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/AcidExportSemanticAnalyzer.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/RewriteSemanticAnalyzer.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
088b5cf 
  ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java 
8651afd 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableDesc.java 423ca2a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExportWork.java d91569e 


Diff: https://reviews.apache.org/r/69339/diff/2/

Changes: https://reviews.apache.org/r/69339/diff/1-2/


Testing
---

Tested by running all the unit tests, and q tests against these 
functionalities, and they are still fine.
Also tested on local cluster, the commands are still working.


Thanks,

Miklos Gergely

[jira] [Created] (HIVE-20922) HiveServer2 user is switched to another occasionally

2018-11-15 Thread Cao Junliang (JIRA) 
Cao Junliang created HIVE-20922:
---

 Summary: HiveServer2 user is switched to another occasionally
 Key: HIVE-20922
 URL: https://issues.apache.org/jira/browse/HIVE-20922
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.2.1
 Environment: HiveServer2 version: 1.2.1

Hive and yarn were deployed in same machine.
Reporter: Cao Junliang


The HiveServer2（version 1.2.1）authentication configuration is:
authentication type: kerberos
kerboros principal: hive/h...@dc1.fh.com

Other infomation:
client principal: n...@dc1.fh.com
yarn service principal:yarn/_h...@dc1.fh.com

After the hive service was started, application could use jdbc to access hive 
service. But sometime later, clients got the exception. The user of the hive 
service was swiched to another one (yarn).

In my enviroment, Hive and Yarn were deployed in same machine, the user of yarn 
service is yarn/_h...@dc1.fh.com. Bebore the exception accured, the hive 
service was not restarted.

==

Main logs:

2018-11-11 23:59:35 [HiveServer2-Handler-Pool: Thread-115739] INFO 
org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(294) -Client 
protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
2018-11-11 23:59:35 [HiveServer2-Handler-Pool: Thread-115739] WARN 
org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(308) -Error 
opening session: 
org.apache.hive.service.cli.HiveSQLException: Failed to open new session: 
java.lang.RuntimeException: java.lang.RuntimeException: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
 User: yarn/mas...@dc1.fh.com is not allowed to impersonate nbhd
 at 
org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:266)
 at 
org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:202)
 at 
org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:402)
 at 
org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:297)
 at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)
 at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)
 at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
 at 
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692)
 at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
 User: yarn/mas...@dc1.fh.com is not allowed to impersonate nbhd
 at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:83)
 at 
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
 at 
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
 at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
 at com.sun.proxy.$Proxy19.open(Unknown Source)
 at 
org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:258)
 ... 12 more
Caused by: java.lang.RuntimeException: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
 User: yarn/mas...@dc1.fh.com is not allowed to impersonate nbhd
 at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
 at 
org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:137)
 at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
 ... 20 more
Caused by: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
 User: yarn/mas...@dc1.fh.com is not allowed to impersonate nbhd
 at org.apache.hadoop.ipc.Client.call(Client.java:1468)
 at org.apache.hadoop.ipc.Client.call(Client.java:1399)
 at

Review Request 69341: HIVE-20891: Call alter_partition in batch when dynamically loading partitions

2018-11-15 Thread Laszlo Pinter via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69341/
---

Review request for hive.


Repository: hive-git


Description
---

HIVE-20891: Call alter_partition in batch when dynamically loading partitions


Diffs
-

  ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java 
e8f362357537e73502f743a9df189dec9be2da5d 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
e185bf49d42da9d1497643c20bbd71edaf071bf1 


Diff: https://reviews.apache.org/r/69341/diff/1/


Testing
---


Thanks,

Laszlo Pinter