Re: Review Request 65634: HIVE-18264: CachedStore: Store cached partitions/col stats within the table cache

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65634/#review198507
---



Changes are pretty big, I didn't go through all of them 0 some comments below.


standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
Lines 62 (patched)


Please add Javadoc comment, explaining what this function does.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
Lines 63 (patched)


It would be cleaner and easier to read to rewrite this as

```
  public static String buildKey(List partVals) {
if (partVals == null || partVals.isEmpty()) {
  return "";
}
return String.join(delimit, partVals);
  }
```



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
Lines 70 (patched)


1) Please add Javadoc comment, explaining what this function does.
2) Is overloading really useful here?



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
Lines 71 (patched)


why not just 

`return buildKey(partVals) + delimit + colName`

can colName be empty here or not?



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
Line 62 (original), 75 (patched)


This method is never used



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 128 (patched)


1) Please add units in the name and use constant for the default value.
2) Please document what is `cacheRefreshPeriod`.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 223 (original), 137 (patched)


Why do you need an empty public constructor?



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 226 (original), 140 (patched)


Please document this method - in particular how does it gets cache 
implementaiton from config.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 236 (original), 150 (patched)


Please used internal formatting for LOG:

LOG.debug("CachedStore is not enabled; using {}", clazzName)



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 237 (original), 151 (patched)


This return is not needed.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 259 (original), 172 (patched)


Can this be an else part of the prior if?



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 262 (original), 175 (patched)


This doesn't look correct:

1) initBlackListWhiteList() will not update any existing whitelist or 
blacklist, only add one if it wasn't there.
2) initBlackListWhiteList() is calling 
`Collections.reverse(blacklistPatterns)` which doesn't make sense when 
configuration is set to a new value.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 176 (patched)


Looks like every time someone calls setConf() a new thread is started - 
isn't it a threda leak?
In general it isn't a good practice to add such side-effects for config 
changes like setConf - it is better to explicitly call a method which will do 
whatever is needed after conf update.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 179 (patched)


Please document this method.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 180 (patched)


This seem to repeat the code from setConf. is there any way to avoid code 
copy?

The only difference seems to be the call to startCacheUpdateService() which 
shows that it isn't a good idea to have it there.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 266 (original), 202 (patched)

Re: Review Request 65745: HIVE-18743: CREATE TABLE on S3 data can be extremely slow. DO_NOT_UPDATE_STATS workaround is buggy.

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65745/#review198506
---



ping

- Alexander Kolbasov


On Feb. 22, 2018, 3:44 a.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65745/
> ---
> 
> (Updated Feb. 22, 2018, 3:44 a.m.)
> 
> 
> Review request for hive, Andrew Sherman, Janaki Lahorani, Zoltan Haindrich, 
> Sahil Takiar, Thejas Nair, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-18743
> https://issues.apache.org/jira/browse/HIVE-18743
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18743: CREATE TABLE on S3 data can be extremely slow. 
> DO_NOT_UPDATE_STATS workaround is buggy.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
>  89354a2d34249903a9ff13c4ed913a68de93057e 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  47de215a23afb0d2589f129c75000e36c334888c 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  50f873a013a9aa3cea0a2af8146484b9387c08f2 
> 
> 
> Diff: https://reviews.apache.org/r/65745/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

[jira] [Created] (HIVE-18849) Java 9 Building "mvn clean package -Pdist -rf :hive-common" get Caused by: java.lang.NullPointerException

[JIRA]

Мирон created HIVE-18849:


 Summary: Java 9 Building "mvn clean package -Pdist -rf 
:hive-common" get Caused by: java.lang.NullPointerException
 Key: HIVE-18849
 URL: https://issues.apache.org/jira/browse/HIVE-18849
 Project: Hive
  Issue Type: Bug
Reporter: Мирон
 Fix For: 2.3.2
 Attachments: Issue_Build_Complete_Log.txt, Issue_Stack_Trace.txt

Please see attached stack trace, both, brief and complete capture, both using 
-X verbose output maven flag.

Irrespective of the true cause, it would be very nice if this message of build 
tool ( maven )

--

Caused by: java.lang.NullPointerException
    at com.sun.tools.javac.main.JavaCompiler.readSourceFile 
(JavaCompiler.java:825)
    at 
com.sun.tools.javac.processing.JavacProcessingEnvironment$ImplicitCompleter.complete
 (JavacProcessingEnvironment.java:1510)
    at com.sun.tools.javac.code.Symbol.complete (Symbol.java:633)

--

carried actual file name, that was attempted reading and failed with exception.

 

Git repository cloned from [https://github.com/apache/hive.git] yesterday - 
today overnight.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18848) Improve readability of filter conditions in explain plan when CBO is run

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-18848:
--

 Summary: Improve readability of filter conditions in explain plan 
when CBO is run
 Key: HIVE-18848
 URL: https://issues.apache.org/jira/browse/HIVE-18848
 Project: Hive
  Issue Type: Improvement
  Components: CBO
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


CBO might return comparison operands in any non-deterministic order. Try to 
show {{  }} when possible, i.e., {{c < 10}} rather 
than {{10 > c}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18847) CachedStore: Investigate TestCachedStore#testTableColStatsOps

[Vaibhav Gumashta (JIRA)]

Vaibhav Gumashta created HIVE-18847:
---

 Summary: CachedStore: Investigate 
TestCachedStore#testTableColStatsOps 
 Key: HIVE-18847
 URL: https://issues.apache.org/jira/browse/HIVE-18847
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Vaibhav Gumashta


Currently commented out due to ObjectStore.updateTableColumnStatistics call 
unable to persist stats to derby. Needs investigation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18846) Query results cache: Allow queries to refer to the pending results of a query that has not finished yet

[Jason Dere (JIRA)]

Jason Dere created HIVE-18846:
-

 Summary: Query results cache: Allow queries to refer to the 
pending results of a query that has not finished yet
 Key: HIVE-18846
 URL: https://issues.apache.org/jira/browse/HIVE-18846
 Project: Hive
  Issue Type: Sub-task
  Components: Query Planning
Reporter: Jason Dere
Assignee: Jason Dere


Currently, a query's results can only be looked up in the cache if the query 
has completely finished execution. Allow new queries to use the results cache 
to find queries that are still executing so they can re-use the results when 
the query has finished.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18845) SHOW COMAPCTIONS should show host name

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-18845:
-

 Summary: SHOW COMAPCTIONS should show host name
 Key: HIVE-18845
 URL: https://issues.apache.org/jira/browse/HIVE-18845
 Project: Hive
  Issue Type: Improvement
  Components: Transactions
Affects Versions: 1.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


once the job starts, the WorkerId includes the hostname submitting the job
but before that there is no way to tell which of the Metastores in HA set up 
has picked up a given item to compact.  Should make it obvious to know which 
log to look at.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18844) Compaction Initiator/Worker/Cleaner should log CompactionInfo.id consistently

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-18844:
-

 Summary: Compaction Initiator/Worker/Cleaner should log 
CompactionInfo.id consistently
 Key: HIVE-18844
 URL: https://issues.apache.org/jira/browse/HIVE-18844
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Reporter: Eugene Koifman


easier to debug/read logs
Should also include this in exceptions consistently



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18843) Improve information shown by SHOW MATERIALIZED VIEWS statement

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-18843:
--

 Summary: Improve information shown by SHOW MATERIALIZED VIEWS 
statement
 Key: HIVE-18843
 URL: https://issues.apache.org/jira/browse/HIVE-18843
 Project: Hive
  Issue Type: Improvement
  Components: Materialized views
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Currently only the materialized view name is shown. We should show some other 
information such as whether rewriting is enabled for it or if it is up-to-date.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18842) CLUSTER BY/DISTRIBUTE BY/SORT BY support for materialized views

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-18842:
--

 Summary: CLUSTER BY/DISTRIBUTE BY/SORT BY support for materialized 
views
 Key: HIVE-18842
 URL: https://issues.apache.org/jira/browse/HIVE-18842
 Project: Hive
  Issue Type: Improvement
  Components: Materialized views
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


We should support defining a {{CLUSTER BY/DISTRIBUTE BY/SORT BY}} specification 
for materialized views. 

The syntax should be extended as follows:

{code:sql}
CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
  [COMMENT materialized_view_comment]
  [CLUSTER BY (col_name, ...) | ( [DISTRIBUTE BY (col_name, ...)] [SORT BY 
(col_name, ...)] ) ] -- NEW!
  [
   [ROW FORMAT row_format] 
   [STORED AS file_format]
 | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
  ]
  [LOCATION hdfs_path]
  [TBLPROPERTIES (property_name=property_value, ...)]
  AS select_statement;
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18841) Support authorization of UDF usage in hive

[Thejas M Nair (JIRA)]

Thejas M Nair created HIVE-18841:


 Summary: Support authorization of UDF usage in hive
 Key: HIVE-18841
 URL: https://issues.apache.org/jira/browse/HIVE-18841
 Project: Hive
  Issue Type: New Feature
Reporter: Thejas M Nair
Assignee: Thejas M Nair


It should be possible to create authorization policies on UDF usage. 
ie, it should be possible to control who can use certain UDF in their queries.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18840) CachedStore: Prioritize loading of recently accessed tables during prewarm

[Vaibhav Gumashta (JIRA)]

Vaibhav Gumashta created HIVE-18840:
---

 Summary: CachedStore: Prioritize loading of recently accessed 
tables during prewarm
 Key: HIVE-18840
 URL: https://issues.apache.org/jira/browse/HIVE-18840
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Vaibhav Gumashta


On clusters with large metadata, prewarming the cache can take several hours. 
Now that CachedStore does not block on prewarm anymore (after HIVE-18264), we 
should prioritize loading of recently accessed tables during prewarm.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in based tables)

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-18839:
--

 Summary: Implement incremental rebuild for materialized views 
(only insert operations in based tables)
 Key: HIVE-18839
 URL: https://issues.apache.org/jira/browse/HIVE-18839
 Project: Hive
  Issue Type: Improvement
  Components: Materialized views
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Implementation will follow current code path for full rebuild. 

When the MV query plan is retrieved, if the MV contents are outdated because 
there were insert operations in the source tables, we will introduce a filter 
with a condition based on stored value of ValidWriteIdLists. For instance, 
{{WRITE_ID < high_txn_id AND WRITE_ID NOT IN (x, y, ...)}}. Then the rewriting 
will do the rest of the work by creating a partial rewriting, where the 
contents of the MV are read as well as the new contents from the source tables.

This mechanism will not work only for ALTER MV... REBUILD, but also for user 
queries which will be able to benefit from using outdated MVs to compute part 
of the needed results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18838) investigate potential issue in FixedSizedObjectPool

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-18838:
---

 Summary: investigate potential issue in FixedSizedObjectPool
 Key: HIVE-18838
 URL: https://issues.apache.org/jira/browse/HIVE-18838
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


Looks like some bugs are possible with multiple consumers. The memory model 
assumptions in some places may be too optimistic. 
For now the pools will be disabled (see Hive-18837), this JIRA is to reenable 
if needed after a follow up investigation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18837) add a flag and disable some object pools in LLAP until further testing

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-18837:
---

 Summary: add a flag and disable some object pools in LLAP until 
further testing
 Key: HIVE-18837
 URL: https://issues.apache.org/jira/browse/HIVE-18837
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


There appears to be some subtle concurrency issue in FixedSizedObjectPool that 
happens with multiple consumers where some object may be retrieved twice. 
Unfortunately running a load test for hour(s) does not trigger it for me and 
overall it happens extremely rarely on non-specific tests; adding debug info at 
this level is a little bit difficult to determine how it could have happened 
and interlocked operations in the trace may actually eliminate the issue. I 
suspect it has something to do with aggressive assumptions made for locking and 
array elements and the memory model. Maybe that can be simplified without much 
perf loss.
Anyway, for now we will disable the pools where multiple consumers use them.
Need to test perf to see if these two pools even matter; if so, we can simplify 
the model as per above or debug the issue in some way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65766: HIVE-18738 LLAP IO ACID - includes handling is broken

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65766/#review198477
---




ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java
Lines 127 (patched)


will remove on commit, unneeded


- Sergey Shelukhin


On March 1, 2018, 10:35 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65766/
> ---
> 
> (Updated March 1, 2018, 10:35 p.m.)
> 
> 
> Review request for hive, Prasanth_J and Teddy Choi.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 2776fe95f1 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/ColumnVectorBatch.java
>  9262bf0ce3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  bb319f0d4a 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  a69c9a023c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
>  2a2be56cab 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/GenericColumnVectorProducer.java
>  d66e2f2540 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
>  3a7b192a1c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  36810d9480 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ReadPipeline.java 
> 06708d34a7 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  a6d2a0497c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
>  166abf7c70 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExtractRow.java 
> 681d9cad7b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSelectOperator.java 
> 59b3ae9698 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java e9564852bd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  e296351225 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 
> c1e55c7fda 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> 32bdf6e68e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
>  1e7708e9ae 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 50d10e33cd 
>   ql/src/test/queries/clientpositive/llap_acid2.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/llap_acid2.q.out PRE-CREATION 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
>  aa2615060a 
> 
> 
> Diff: https://reviews.apache.org/r/65766/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 65766: HIVE-18738 LLAP IO ACID - includes handling is broken

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65766/
---

(Updated March 1, 2018, 10:35 p.m.)


Review request for hive, Prasanth_J and Teddy Choi.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 2776fe95f1 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/ColumnVectorBatch.java
 9262bf0ce3 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
 bb319f0d4a 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 a69c9a023c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
 2a2be56cab 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/GenericColumnVectorProducer.java
 d66e2f2540 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
 3a7b192a1c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 36810d9480 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ReadPipeline.java 
06708d34a7 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 a6d2a0497c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 166abf7c70 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExtractRow.java 
681d9cad7b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSelectOperator.java 
59b3ae9698 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java e9564852bd 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 e296351225 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 
c1e55c7fda 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
32bdf6e68e 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
 1e7708e9ae 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 50d10e33cd 
  ql/src/test/queries/clientpositive/llap_acid2.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/llap_acid2.q.out PRE-CREATION 
  
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
 aa2615060a 


Diff: https://reviews.apache.org/r/65766/diff/3/

Changes: https://reviews.apache.org/r/65766/diff/2-3/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 65634: HIVE-18264: CachedStore: Store cached partitions/col stats within the table cache

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65634/#review198471
---


Ship it!




Ship It!

- Daniel Dai


On March 1, 2018, 11:09 a.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65634/
> ---
> 
> (Updated March 1, 2018, 11:09 a.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-18264
> https://issues.apache.org/jira/browse/HIVE-18264
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-18264
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  a3725c5395 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 86c9c2b33c 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  ac71d0882f 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  7b44df4128 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  f500d63725 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
>  f0f650ddcf 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  0d132f2074 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  32ea17495f 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  50f873a013 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  75ea8c4a77 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  207d842f94 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  ab6feb6f0b 
>   standalone-metastore/src/test/resources/log4j2.properties 365687e1c9 
> 
> 
> Diff: https://reviews.apache.org/r/65634/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

[jira] [Created] (HIVE-18836) backport HIVE-17600 to hive 1

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-18836:
---

 Summary: backport HIVE-17600 to hive 1
 Key: HIVE-18836
 URL: https://issues.apache.org/jira/browse/HIVE-18836
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18835) JDBC standalone jar download link in ambari

[Miklos Gergely (JIRA)]

Miklos Gergely created HIVE-18835:
-

 Summary: JDBC standalone jar download link in ambari
 Key: HIVE-18835
 URL: https://issues.apache.org/jira/browse/HIVE-18835
 Project: Hive
  Issue Type: New Feature
  Components: Hive
Affects Versions: 3.0.0
Reporter: Miklos Gergely
Assignee: Miklos Gergely
 Attachments: HIVE-18835.patch

Let HS2 offer the file for download, so that Ambari can create link on it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65422: HIVE-17626

[Zoltan Haindrich]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
---

(Updated March 1, 2018, 3:35 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

06


Bugs: HIVE-17626
https://issues.apache.org/jira/browse/HIVE-17626


Repository: hive-git


Description
---

preview


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6 
  data/conf/hive-site.xml b56cbd2469 
  data/conf/llap/hive-site.xml c4c299c5de 
  data/conf/perf-reg/spark/hive-site.xml 5ca660d4e7 
  data/conf/perf-reg/tez/hive-site.xml 62ecb74f3f 
  data/conf/rlist/hive-site.xml 630e481edc 
  data/conf/spark/local/hive-site.xml 8ff6256638 
  data/conf/spark/standalone/hive-site.xml 84851c769c 
  data/conf/spark/yarn-client/hive-site.xml 6c633623ab 
  data/conf/tez/hive-site.xml 236adc7087 
  
druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java
 cbeac2c00a 
  itests/src/test/resources/testconfiguration.properties 2776fe95f1 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 6cd7a136ae 
  itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java 
c01d87bf51 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93 
  ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4 
  ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588 
  ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java 
08b791ad42 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
131127e50d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java 
1e28ca843f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java 
2b345d6ec7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java 
085d6a7d94 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java 
0d31e6e422 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 8dd7cfe58c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
 134fc0ff0b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
 1eb72ce4d9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java
 384bd74686 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java b75850760f 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
b0cf3bd94e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java
 8c1bcb3f62 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
783a672c47 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignatureFactory.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkPartitionPruningSinkDesc.java
 d1c53cf345 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 9a3f81c98f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 733ec79ce1 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java 
714cf3914b 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AppMasterEventDesc.java 7d5be6ba81 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java 
7332693513 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 
5d3fdb8b63 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ce61fc5a2e 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java d59834ce08 
  ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 86cc77d43b 
  ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java 9c651ab3ab 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java 6dcf05af28 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java bd45c752e1 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java 
3837a49934 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LimitDesc.java ce53feae00 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cf4ab606f2 
  

[jira] [Created] (HIVE-18834) Lzo files not getting split in hive jobs on hive2.1.0、hive2.2.0

[Saijin Huang (JIRA)]

Saijin Huang created HIVE-18834:
---

 Summary: Lzo files not getting split in hive jobs on 
hive2.1.0、hive2.2.0
 Key: HIVE-18834
 URL: https://issues.apache.org/jira/browse/HIVE-18834
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.2.0, 2.1.0
Reporter: Saijin Huang
Assignee: Saijin Huang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65634: HIVE-18264: CachedStore: Store cached partitions/col stats within the table cache

[Vaibhav Gumashta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65634/
---

(Updated March 1, 2018, 11:09 a.m.)


Review request for hive, Daniel Dai and Thejas Nair.


Bugs: HIVE-18264
https://issues.apache.org/jira/browse/HIVE-18264


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-18264


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 a3725c5395 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 86c9c2b33c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 ac71d0882f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 7b44df4128 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 f500d63725 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
 f0f650ddcf 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 0d132f2074 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
 32ea17495f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 50f873a013 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 75ea8c4a77 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 207d842f94 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
 ab6feb6f0b 
  standalone-metastore/src/test/resources/log4j2.properties 365687e1c9 


Diff: https://reviews.apache.org/r/65634/diff/4/

Changes: https://reviews.apache.org/r/65634/diff/3-4/


Testing
---


Thanks,

Vaibhav Gumashta

Re: How to Load Data From a CSV to a parquet table

[Jörn Franke]

You have defined a parquet only table. It interprets your CSV file as parquet. 
You can for instance define 2 tables:

* one external for the CSV file
* one table for the parquet file

Afterwards you select from the first table and insert in the second table. 

> On 1. Mar 2018, at 08:31, Anubhav Tarar  wrote:
> 
> Hi i m trying to load data from a csv file into parquet in hive but got
> this exception
> 
> hive> create table if not exists REGION( R_NAME string, R_REGIONKEY string,
> R_COMMENT string ) stored as parquet;
> OK
> Time taken: 0.414 seconds
> hive> load data local inpath
> 'file:///home/anubhav/Downloads/dbgen/region.tbl' into table region;
> Loading data to table default.region
> OK
> Time taken: 1.011 seconds
> hive> select * from region;
> OK
> Failed with exception java.io.IOException:java.lang.RuntimeException:
> hdfs://localhost:54311/user/hive/warehouse/region/region.tbl is not a
> Parquet file. expected magic number at tail [80, 65, 82, 49] but found
> [115, 108, 124, 10]
> Time taken: 0.108 seconds
> 
> can anyone help?hive version is 2.1
> 
> -- 
> Thanks and Regards
> 
> *   Anubhav Tarar *
> 
> 
> * Software Consultant*
>  *Knoldus Software LLP    *
>   LinkedIn  Twitter
> fb 
>  mob : 8588915184