date:20180907

[jira] [Created] (HIVE-20522) HiveFilterSetOpTransposeRule may throw assertion error due to nullability of fields

2018-09-07 Thread Jesus Camacho Rodriguez (JIRA)

Jesus Camacho Rodriguez created HIVE-20522:
--

 Summary: HiveFilterSetOpTransposeRule may throw assertion error 
due to nullability of fields
 Key: HIVE-20522
 URL: https://issues.apache.org/jira/browse/HIVE-20522
 Project: Hive
  Issue Type: Bug
  Components: CBO
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


{noformat}
[ERROR] Failures:
[ERROR]   TestMiniLlapLocalCliDriver.testCliDriver:59 Cannot add expression of 
different type to set:
set type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE 
"ISO-8859-1$en_US$primary" column1, VARCHAR(2147483647) CHARACTER SET 
"UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" column2, VARCHAR(2147483647) 
CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column3) 
NOT NULL
expression type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" 
COLLATE "ISO-8859-1$en_US$primary" NOT NULL column1, VARCHAR(2147483647) 
CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL column2, 
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" 
NOT NULL column3) NOT NULL
set is rel#260:HiveFilter.HIVE.[](input=HepRelVertex#251,condition=<($2, 
_UTF-16LE'100'))
expression is HiveFilter#262
{noformat}

q file contains examples that may to reproduce failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20521) HS2 doAs=true has permission issue with hadoop.tmp.dir

2018-09-07 Thread Thejas M Nair (JIRA)

Thejas M Nair created HIVE-20521:


 Summary: HS2 doAs=true has permission issue with hadoop.tmp.dir
 Key: HIVE-20521
 URL: https://issues.apache.org/jira/browse/HIVE-20521
 Project: Hive
  Issue Type: Improvement
Reporter: Thejas M Nair


This is a result of changes in HIVE-18858.
 As described by [~puneetj] in HIVE-18858 -

{quote}
This seems to have broken working scenarios with Hive MR.  We now see 
hadoop.tmp.dir is always set to /tmp/hadoop-hive (in job.xml). This creates 
problems on a multi-tenant hadoop cluster since ownership of tmp folder is set 
to the user who executes the jobs first and other users fails to write to tmp 
folder.

E.g. User1 run job and /tmp/hadoop-hive is created on worker node with 
ownership to user1 and sibsequently user2 tries to run a job and job fails due 
to no write permission on /tmp/hadoop-hive/

Old behavior allowed multiple tenants to write to their respective tmp folders 
which was secure and contention free. User1 - /tmp/hadoop-user1, User2 - 
/tmp/hadoop-user2.
{quote}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20520) length(CHAR) doesn't consider trailing space

2018-09-07 Thread Jesus Camacho Rodriguez (JIRA)

Jesus Camacho Rodriguez created HIVE-20520:
--

 Summary: length(CHAR) doesn't consider trailing space
 Key: HIVE-20520
 URL: https://issues.apache.org/jira/browse/HIVE-20520
 Project: Hive
  Issue Type: Bug
  Components: UDF
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Reproduce steps:

{code:java}
create table test(a char(2), b varchar(2));
insert into test values('L ', 'L ');
select length(a),length(b) from test;
+--+--+
| _c0  | _c1  |
+--+--+
| 1| 2|
+--+--+
1 row selected (0.185 seconds)
{code}

Here char with trailing spaces are trimmed, whereas leading spaces are not 
trimmed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20519) Remove 30m min value for hive.spark.session.timeout

2018-09-07 Thread Sahil Takiar (JIRA)

Sahil Takiar created HIVE-20519:
---

 Summary: Remove 30m min value for hive.spark.session.timeout
 Key: HIVE-20519
 URL: https://issues.apache.org/jira/browse/HIVE-20519
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Reporter: Sahil Takiar
Assignee: Sahil Takiar


In HIVE-14162 we added the config \{{hive.spark.session.timeout}} which 
provided a way to time out Spark sessions that are active for a long period of 
time. The config has a lower bound of 30m which we should remove. It should be 
possible for users to configure this value so the HoS session is closed as soon 
as the query is complete.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68648: HIVE-20510

2018-09-07 Thread Deepak Jaiswal


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68648/
---

(Updated Sept. 7, 2018, 7:57 p.m.)


Review request for hive, Gopal V and Matt McCline.


Changes
---

Missed non-vectorized case and some result updates.


Bugs: HIVE-20510
https://issues.apache.org/jira/browse/HIVE-20510


Repository: hive-git


Description
---

Vectorization : Support loading bucketed tables using sorted dynamic partition 
optimizer.
Added a new VectorExpression BucketNumberExpression to evaluate _bucket_number.
Made the loops as tight as possible.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 8bf0a9c77d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java a2a9c8421e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
57f7c0108e 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/BucketNumExpression.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
 5ab59c9c61 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java
 51010aac85 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBucketNumber.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/dynpart_sort_opt_vectorization.q 
435cdaddd0 
  ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out 
22f0a31eb3 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out 
21fc2c545a 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out 
a0a5e0cf32 


Diff: https://reviews.apache.org/r/68648/diff/3/

Changes: https://reviews.apache.org/r/68648/diff/2-3/


Testing
---


Thanks,

Deepak Jaiswal

Fwd: Question on partitioning materialized views

2018-09-07 Thread Karthik Palaniappan

Hi All,

Created a materialized view with partitions and
tblproperties('transactional'='true') in Hive Version 4.0 running on Hadoop
3.1. View gets created properly and I can see one file in each partition.
When we tried to query the view though no rows are getting returned.

In describe formatted, we see that numFiles and numRows are 0.
Tried gathering statistics, MSCK repair table etc and manually dropped and
re-added partition. Still the materialized view returns 0 rows.

Not quite sure if I am missing something or if this is a bug. Have attached
the output of describe and hive-server2 logs that got generated when the
materialized view was created.

Log shows below error message while gathering statistics:

MetaException(message:Cannot change stats state for a transactional table
without providing the transactional write state for verification (new write
ID 0, valid write IDs default.test_fact_mv:0:9223372036854775807::;

Can you please give me some pointers to get the above materialized view
working?

Thanks,
Karthik
0: jdbc:hive2://myserver:2181/> describe formatted test_fact_mv;
+--+++
| col_name | data_type  
|  comment   |
+--+++
| # col_name   | data_type  
| comment|
| dim1 | string 
||
| dim2 | string 
||
| dim3 | string 
||
| dim4 | string 
||
| dim5 | string 
||
| dim6 | string 
||
| dim7 | string 
||
| dim8 | string 
||
| dim9 | string 
||
| dim10| string 
||
| dim11| string 
||
| dim12| string 
||
| dim13| string 
||
| dim14| string 
||
| dim15| string 
||
| dim16| string 
||
| dim17| string 
||
| dim18| string 
||
| dim19| string 
||
| amt1 | decimal(20,3)  
||
| amt2 | decimal(20,3)  
||
| amt3 | decimal(20,3)  
||
| dim20| string 
||
|

Re: Review Request 68633: [HIVE-20481] Add record key column

2018-09-07 Thread Vineet Garg


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68633/#review208459
---


Ship it!




Ship It!

- Vineet Garg


On Sept. 5, 2018, 9:15 p.m., Slim Bouguerra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68633/
> ---
> 
> (Updated Sept. 5, 2018, 9:15 p.m.)
> 
> 
> Review request for hive, Gopal V and Vineet Garg.
> 
> 
> Bugs: [HIVE-20481]
> https://issues.apache.org/jira/browse/[HIVE-20481]
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This Patch adds the kafka record key as an extra metadata column.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest-druid/src/main/java/org/apache/hive/kafka/SingleNodeKafkaCluster.java
>  c9339b565e 
>   kafka-handler/src/java/org/apache/hadoop/hive/kafka/GenericKafkaSerDe.java 
> e7ea53f4bc 
>   kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaJsonSerDe.java 
> f383190083 
>   
> kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaPullerRecordReader.java
>  908ee5e29f 
>   
> kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaRecordWritable.java 
> c6924ea480 
>   kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaScanTrimmer.java 
> 76415151ec 
>   
> kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaStorageHandler.java 
> 5847df5e7e 
>   
> kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaStorageHandlerInfo.java
>  PRE-CREATION 
>   
> kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaStreamingUtils.java 
> d2d0ebc192 
>   
> kafka-handler/src/test/org/apache/hadoop/hive/kafka/KafkaPullerInputSplitTest.java
>  be26986818 
>   
> kafka-handler/src/test/org/apache/hadoop/hive/kafka/KafkaRecordIteratorTest.java
>  5de51cd00a 
>   
> kafka-handler/src/test/org/apache/hadoop/hive/kafka/KafkaRecordWritableTest.java
>  8f9df548f7 
>   
> kafka-handler/src/test/org/apache/hadoop/hive/kafka/KafkaScanTrimmerTest.java 
> 289dafde36 
>   
> kafka-handler/src/test/org/apache/hadoop/hive/kafka/KafkaStreamingUtilsTest.java
>  8d68ec27c8 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/StorageHandlerInfo.java 
> dbc44a6621 
>   ql/src/test/queries/clientpositive/kafka_storage_handler.q 8daa3e3bc0 
>   ql/src/test/results/clientpositive/druid/kafka_storage_handler.q.out 
> 3dec33d790 
>   testutils/ptest2/conf/deployed/master-mr2.properties 90a654cf8a 
> 
> 
> Diff: https://reviews.apache.org/r/68633/diff/1/
> 
> 
> Testing
> ---
> 
> Via Qfiles
> 
> 
> Thanks,
> 
> Slim Bouguerra
> 
>

Re: Review Request 68633: [HIVE-20481] Add record key column

2018-09-07 Thread Slim Bouguerra



> On Sept. 7, 2018, 6:17 p.m., Vineet Garg wrote:
> > kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaStorageHandler.java
> > Line 71 (original), 77 (patched)
> > 
> >
> > earlier both topic and broker string was being checked for null but now 
> > only for emptyness. Is it guranteed for them to be not null?

The default of the get is empty string thus nulls will be converted to empty.


- Slim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68633/#review208451
---


On Sept. 5, 2018, 9:15 p.m., Slim Bouguerra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68633/
> ---
> 
> (Updated Sept. 5, 2018, 9:15 p.m.)
> 
> 
> Review request for hive, Gopal V and Vineet Garg.
> 
> 
> Bugs: [HIVE-20481]
> https://issues.apache.org/jira/browse/[HIVE-20481]
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This Patch adds the kafka record key as an extra metadata column.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest-druid/src/main/java/org/apache/hive/kafka/SingleNodeKafkaCluster.java
>  c9339b565e 
>   kafka-handler/src/java/org/apache/hadoop/hive/kafka/GenericKafkaSerDe.java 
> e7ea53f4bc 
>   kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaJsonSerDe.java 
> f383190083 
>   
> kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaPullerRecordReader.java
>  908ee5e29f 
>   
> kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaRecordWritable.java 
> c6924ea480 
>   kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaScanTrimmer.java 
> 76415151ec 
>   
> kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaStorageHandler.java 
> 5847df5e7e 
>   
> kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaStorageHandlerInfo.java
>  PRE-CREATION 
>   
> kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaStreamingUtils.java 
> d2d0ebc192 
>   
> kafka-handler/src/test/org/apache/hadoop/hive/kafka/KafkaPullerInputSplitTest.java
>  be26986818 
>   
> kafka-handler/src/test/org/apache/hadoop/hive/kafka/KafkaRecordIteratorTest.java
>  5de51cd00a 
>   
> kafka-handler/src/test/org/apache/hadoop/hive/kafka/KafkaRecordWritableTest.java
>  8f9df548f7 
>   
> kafka-handler/src/test/org/apache/hadoop/hive/kafka/KafkaScanTrimmerTest.java 
> 289dafde36 
>   
> kafka-handler/src/test/org/apache/hadoop/hive/kafka/KafkaStreamingUtilsTest.java
>  8d68ec27c8 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/StorageHandlerInfo.java 
> dbc44a6621 
>   ql/src/test/queries/clientpositive/kafka_storage_handler.q 8daa3e3bc0 
>   ql/src/test/results/clientpositive/druid/kafka_storage_handler.q.out 
> 3dec33d790 
>   testutils/ptest2/conf/deployed/master-mr2.properties 90a654cf8a 
> 
> 
> Diff: https://reviews.apache.org/r/68633/diff/1/
> 
> 
> Testing
> ---
> 
> Via Qfiles
> 
> 
> Thanks,
> 
> Slim Bouguerra
> 
>

Re: Review Request 68648: HIVE-20510

2018-09-07 Thread Deepak Jaiswal


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68648/
---

(Updated Sept. 7, 2018, 6:42 p.m.)


Review request for hive, Gopal V and Matt McCline.


Changes
---

Addressed concerns from Matt's review.
Replaced the constant string _bucket_number with a UDF GenericUDFBucketNumber() 
to make sure _bucket_number could be uaed as a legitimate string in queries.


Bugs: HIVE-20510
https://issues.apache.org/jira/browse/HIVE-20510


Repository: hive-git


Description
---

Vectorization : Support loading bucketed tables using sorted dynamic partition 
optimizer.
Added a new VectorExpression BucketNumberExpression to evaluate _bucket_number.
Made the loops as tight as possible.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 8bf0a9c77d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
57f7c0108e 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/BucketNumExpression.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
 5ab59c9c61 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java
 51010aac85 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBucketNumber.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/dynpart_sort_opt_vectorization.q 
435cdaddd0 
  ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out 
22f0a31eb3 


Diff: https://reviews.apache.org/r/68648/diff/2/

Changes: https://reviews.apache.org/r/68648/diff/1-2/


Testing
---


Thanks,

Deepak Jaiswal

Re: Review Request 68633: [HIVE-20481] Add record key column

2018-09-07 Thread Vineet Garg


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68633/#review208451
---




kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaStorageHandler.java
Line 71 (original), 77 (patched)


earlier both topic and broker string was being checked for null but now 
only for emptyness. Is it guranteed for them to be not null?


- Vineet Garg


On Sept. 5, 2018, 9:15 p.m., Slim Bouguerra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68633/
> ---
> 
> (Updated Sept. 5, 2018, 9:15 p.m.)
> 
> 
> Review request for hive, Gopal V and Vineet Garg.
> 
> 
> Bugs: [HIVE-20481]
> https://issues.apache.org/jira/browse/[HIVE-20481]
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This Patch adds the kafka record key as an extra metadata column.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest-druid/src/main/java/org/apache/hive/kafka/SingleNodeKafkaCluster.java
>  c9339b565e 
>   kafka-handler/src/java/org/apache/hadoop/hive/kafka/GenericKafkaSerDe.java 
> e7ea53f4bc 
>   kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaJsonSerDe.java 
> f383190083 
>   
> kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaPullerRecordReader.java
>  908ee5e29f 
>   
> kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaRecordWritable.java 
> c6924ea480 
>   kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaScanTrimmer.java 
> 76415151ec 
>   
> kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaStorageHandler.java 
> 5847df5e7e 
>   
> kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaStorageHandlerInfo.java
>  PRE-CREATION 
>   
> kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaStreamingUtils.java 
> d2d0ebc192 
>   
> kafka-handler/src/test/org/apache/hadoop/hive/kafka/KafkaPullerInputSplitTest.java
>  be26986818 
>   
> kafka-handler/src/test/org/apache/hadoop/hive/kafka/KafkaRecordIteratorTest.java
>  5de51cd00a 
>   
> kafka-handler/src/test/org/apache/hadoop/hive/kafka/KafkaRecordWritableTest.java
>  8f9df548f7 
>   
> kafka-handler/src/test/org/apache/hadoop/hive/kafka/KafkaScanTrimmerTest.java 
> 289dafde36 
>   
> kafka-handler/src/test/org/apache/hadoop/hive/kafka/KafkaStreamingUtilsTest.java
>  8d68ec27c8 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/StorageHandlerInfo.java 
> dbc44a6621 
>   ql/src/test/queries/clientpositive/kafka_storage_handler.q 8daa3e3bc0 
>   ql/src/test/results/clientpositive/druid/kafka_storage_handler.q.out 
> 3dec33d790 
>   testutils/ptest2/conf/deployed/master-mr2.properties 90a654cf8a 
> 
> 
> Diff: https://reviews.apache.org/r/68633/diff/1/
> 
> 
> Testing
> ---
> 
> Via Qfiles
> 
> 
> Thanks,
> 
> Slim Bouguerra
> 
>

[jira] [Created] (HIVE-20518) TxnHandler checkLock direct sql fail with ORA-01795 , if the table has more than 1000 partitions

2018-09-07 Thread Ashutosh Chauhan (JIRA)

Ashutosh Chauhan created HIVE-20518:
---

 Summary: TxnHandler checkLock direct sql fail with ORA-01795 , if 
the table has more than 1000 partitions
 Key: HIVE-20518
 URL: https://issues.apache.org/jira/browse/HIVE-20518
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 2.1.0
Reporter: Rajkumar Singh
Assignee: Rajkumar Singh


with Oracle as Metastore, txnhandler checkLock fail with 
"checkLockWithRetry(181398,34773) : ORA-01795: maximum number of expressions in 
a list is 1000" if the write table has more than 1000 partitions.

complete stacktrace

{code}

txn.TxnHandler (TxnHandler.java:checkRetryable(2099)) - Non-retryable error in 
checkLockWithRetry(181398,34773) : ORA-01795: maximum number of expressions in 
a list is 1000

 (SQLState=42000, ErrorCode=1795)

2018-06-25 15:09:35,999 ERROR [pool-7-thread-197]: metastore.RetryingHMSHandler 
(RetryingHMSHandler.java:invokeInternal(203)) - MetaException(message:Unable to 
update transaction database java.sql.SQLSyntaxErrorException: ORA-01795: 
maximum number of expressions in a list is 1000

 

    at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:447)

    at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)

    at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:951)

    at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:513)

    at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)

    at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)

    at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:195)

    at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:876)

    at 
oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1175)

    at 
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1296)

    at 
oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1498)

    at 
oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:406)

    at com.jolbox.bonecp.StatementHandle.executeQuery(StatementHandle.java:464)

    at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(TxnHandler.java:2649)

    at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLockWithRetry(TxnHandler.java:1126)

    at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:895)

    at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:6123)

    at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)

    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke(Method.java:498)

    at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)

    at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)

    at com.sun.proxy.$Proxy11.lock(Unknown Source)

    at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:12012)

    at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:11996)

    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)

    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)

    at 
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:551)

    at 
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:546)

    at java.security.AccessController.doPrivileged(Native Method)

    at javax.security.auth.Subject.doAs(Subject.java:422)

    at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)

    at 
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:546)

    at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)

    at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

    at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

    at java.lang.Thread.run(Thread.java:748)

)

    at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLockWithRetry(TxnHandler.java:1131)

    at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:895)

    at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:6123)

    at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)

    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke(Method.java:498)

    at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)

Re: Question on partitioning materialized views

2018-09-07 Thread Jesus Camacho Rodriguez

Hi Karthik,

Are you querying the MV directly? Can you repro with a simple example and share 
the list of SQL statements that you are executing to get there?

Some examples of partitioned MVs creation/usage:
https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/materialized_view_partitioned.q
https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/materialized_view_partitioned_2.q
https://github.com/apache/hive/blob/master/ql/src/test/queries/clientpositive/materialized_view_partitioned_3.q

Thanks,
Jesús


From: Karthik Palaniappan 
Reply-To: "dev@hive.apache.org" 
Date: Friday, September 7, 2018 at 8:11 AM
To: "dev@hive.apache.org" 
Subject: Question on partitioning materialized views

Hi All,

Created a materialized view with partitions and 
tblproperties('transactional'='true') in Hive Version 4.0 running on Hadoop 
3.1. View gets created properly and I can see one file in each partition. When 
we tried to query the view though no rows are getting returned.

In describe formatted, we see that numFiles and numRows are 0.
Tried gathering statistics, MSCK repair table etc and manually dropped and 
re-added partition. Still the materialized view returns 0 rows.

Not quite sure if I am missing something or if this is a bug. Have attached the 
output of describe and hive-server2 logs that got generated when the 
materialized view was created.

Log shows below error message while gathering statistics:

MetaException(message:Cannot change stats state for a transactional table 
without providing the transactional write state for verification (new write ID 
0, valid write IDs default.test_fact_mv:0:9223372036854775807::;

Can you please give me some pointers to get the above materialized view working?

Thanks,
Karthik

Re: Review Request 61663: WebUI query plan graphs

2018-09-07 Thread Karen Coppage


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61663/
---

(Updated Sept. 7, 2018, 3:24 p.m.)


Review request for hive, Peter Vary and Xuefu Zhang.


Changes
---

Added tests and checks, disabled zoom


Bugs: HIVE-17300
https://issues.apache.org/jira/browse/HIVE-17300


Repository: hive-git


Description
---

I’m working on a feature of the Hive WebUI Query Plan tab that would provide 
the option to display the query plan as a nice graph (scroll down for 
screenshots). If you click on one of the graph’s stages, the plan for that 
stage appears as text below.
Stages are color-coded if they have a status (Success, Error, Running), and the 
rest are grayed out. Coloring is based on status already available in the 
WebUI, under the Stages tab.
There is an additional option to display stats for MapReduce tasks. This 
includes the job’s ID, tracking URL (where the logs are found), and mapper and 
reducer numbers/progress, among other info.
The library I’m using for the graph is called vis.js (http://visjs.org/). It 
has an Apache license, and the only necessary file to be included from this 
library is about 700 KB.
I tried to keep server-side changes minimal, and graph generation is taken care 
of by the client. Plans with more than a given number of stages (default: 25) 
won't be displayed in order to preserve resources.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/LogUtils.java 5068eb5be7 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 40ea3ac0c5 
  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/session/TestQueryDisplay.java
 95b46a8149 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java dad2035362 
  ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java ac45ec46de 
  ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java 9a77c2969e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 
eb6cbf71e2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java a71faf8576 
  service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon f04d655440 
  service/src/resources/hive-webapps/static/css/query-plan-graph.css 
PRE-CREATION 
  service/src/resources/hive-webapps/static/js/query-plan-graph.js PRE-CREATION 
  service/src/resources/hive-webapps/static/js/vis.min.js PRE-CREATION 


Diff: https://reviews.apache.org/r/61663/diff/5/

Changes: https://reviews.apache.org/r/61663/diff/4-5/


Testing
---


File Attachments (updated)


HIVE-17300.7.patch
  
https://reviews.apache.org/media/uploaded/files/2018/09/07/e8ada965-ea33-48a2-a0b1-e56e8185c4fa__HIVE-17300.7.patch


Thanks,

Karen Coppage

Question on partitioning materialized views

2018-09-07 Thread Karthik Palaniappan

Hi All,

Created a materialized view with partitions and
tblproperties('transactional'='true') in Hive Version 4.0 running on Hadoop
3.1. View gets created properly and I can see one file in each partition.
When we tried to query the view though no rows are getting returned.

In describe formatted, we see that numFiles and numRows are 0.
Tried gathering statistics, MSCK repair table etc and manually dropped and
re-added partition. Still the materialized view returns 0 rows.

Not quite sure if I am missing something or if this is a bug. Have attached
the output of describe and hive-server2 logs that got generated when the
materialized view was created.

Log shows below error message while gathering statistics:

MetaException(message:Cannot change stats state for a transactional table
without providing the transactional write state for verification (new write
ID 0, valid write IDs default.test_fact_mv:0:9223372036854775807::;

Can you please give me some pointers to get the above materialized view
working?

Thanks,
Karthik
0: jdbc:hive2://myserver:2181/> describe formatted test_fact_mv;
+--+++
| col_name | data_type  
|  comment   |
+--+++
| # col_name   | data_type  
| comment|
| dim1 | string 
||
| dim2 | string 
||
| dim3 | string 
||
| dim4 | string 
||
| dim5 | string 
||
| dim6 | string 
||
| dim7 | string 
||
| dim8 | string 
||
| dim9 | string 
||
| dim10| string 
||
| dim11| string 
||
| dim12| string 
||
| dim13| string 
||
| dim14| string 
||
| dim15| string 
||
| dim16| string 
||
| dim17| string 
||
| dim18| string 
||
| dim19| string 
||
| amt1 | decimal(20,3)  
||
| amt2 | decimal(20,3)  
||
| amt3 | decimal(20,3)  
||
| dim20| string 
||
|

Re: Review Request 68656: HIVE-20505: upgrade org.openjdk.jmh:jmh-core to 1.21

2018-09-07 Thread Adam Szita via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68656/#review208448
---


Ship it!




Ship It!

- Adam Szita


On Sept. 6, 2018, 10:07 a.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68656/
> ---
> 
> (Updated Sept. 6, 2018, 10:07 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20505: upgrade org.openjdk.jmh:jmh-core to 1.21
> 
> 
> Diffs
> -
> 
>   itests/hive-jmh/pom.xml 0abefdf791a04593c547119256a755adcd78bda5 
> 
> 
> Diff: https://reviews.apache.org/r/68656/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

[jira] [Created] (HIVE-20517) Creation of staging directory and Move operation is taking time in S3

2018-09-07 Thread mahesh kumar behera (JIRA)

mahesh kumar behera created HIVE-20517:
--

 Summary: Creation of staging directory and Move operation is 
taking time in S3
 Key: HIVE-20517
 URL: https://issues.apache.org/jira/browse/HIVE-20517
 Project: Hive
  Issue Type: Bug
  Components: repl
Affects Versions: 4.0.0
Reporter: mahesh kumar behera
Assignee: mahesh kumar behera
 Fix For: 4.0.0


Operations like insert and add partition creates a staging directory to 
generate the files and then move the files created to actual location. In 
replication flow, the files are first copied to the staging directory and then 
moved (rename) to the actual table location. In case of S3, move is not an 
atomic operation. It internally does a copy and delete. So it can not guarantee 
the consistency required. So it is better to copy the files directly to the 
actual location. This will help in avoiding the staging directory creation 
(which takes 1-2 seconds in s3) and move (which takes time proportional to file 
size).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20516) alter table drop partition should be compatible with old metastore, as partition pruner does

2018-09-07 Thread jinzheng (JIRA)

jinzheng created HIVE-20516:
---

 Summary: alter table drop partition should be compatible with old 
metastore, as partition pruner does
 Key: HIVE-20516
 URL: https://issues.apache.org/jira/browse/HIVE-20516
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 3.0.0
 Environment: all
Reporter: jinzheng
Assignee: jinzheng
 Fix For: 3.1.0


    After the change of  HIVE-4914, we always push the partition expression to 
metastore, to avoid filtering partition by partition names.

    And HIVE-4914 added some protection in Partition pruner, in case that 
metastore may not have api of get_partitions_by_expr.

    Therefore, we should also add similar protection to another calling point, 
when dealing with "alter table drop partition".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20522) HiveFilterSetOpTransposeRule may throw assertion error due to nullability of fields

[jira] [Created] (HIVE-20521) HS2 doAs=true has permission issue with hadoop.tmp.dir

[jira] [Created] (HIVE-20520) length(CHAR) doesn't consider trailing space

[jira] [Created] (HIVE-20519) Remove 30m min value for hive.spark.session.timeout

Re: Review Request 68648: HIVE-20510

Fwd: Question on partitioning materialized views

Re: Review Request 68633: [HIVE-20481] Add record key column

Re: Review Request 68633: [HIVE-20481] Add record key column

Re: Review Request 68648: HIVE-20510

Re: Review Request 68633: [HIVE-20481] Add record key column

[jira] [Created] (HIVE-20518) TxnHandler checkLock direct sql fail with ORA-01795 , if the table has more than 1000 partitions

Re: Question on partitioning materialized views

Re: Review Request 61663: WebUI query plan graphs

Question on partitioning materialized views

Re: Review Request 68656: HIVE-20505: upgrade org.openjdk.jmh:jmh-core to 1.21

[jira] [Created] (HIVE-20517) Creation of staging directory and Move operation is taking time in S3

[jira] [Created] (HIVE-20516) alter table drop partition should be compatible with old metastore, as partition pruner does

17 matches

Site Navigation

Mail list logo

Footer information