Review Request 53593: HIVE-14089 complex type support in LLAP IO is broken

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53593/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
 a27266f 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 2fa68a9 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 eb8ee6c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
 c9b0a4d 
  orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c347181 
  orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 245e3ce 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java a9dbc3e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ccb39da 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 
4d09dcd 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
bcb54d6 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
 ebbdf8d 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 4405232 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 5cc3663 
  ql/src/test/queries/clientpositive/vector_complex_all.q 91a7368 
  ql/src/test/results/clientpositive/llap/vector_complex_all.q.out 08d49bc 
  ql/src/test/results/clientpositive/vector_complex_all.q.out a54a371 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java 76a1286 
  
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
 907181e 

Diff: https://reviews.apache.org/r/53593/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-15167) undeprecate Deserializer, SerDe etc

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-15167:
---

 Summary: undeprecate Deserializer, SerDe etc
 Key: HIVE-15167
 URL: https://issues.apache.org/jira/browse/HIVE-15167
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


SerDe interfaces were deprecated in HIVE-4007 to suggest that users do not 
implement them. However, this results in deprecation warnings all over the 
codebase where they are actually used.
We should un-deprecate (reprecate? precate?) them. We can add a comment for 
implementers instead (we could add a method with a clearly bogus name like 
useThisAbstractClassInstead, and implement it in the class, so it would be 
noticeable, but that would break compat).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 53591: Bootstrap support for replv2

[Vaibhav Gumashta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53591/
---

Review request for hive, Sushanth Sowmyan and Thejas Nair.


Bugs: HIVE-15151
https://issues.apache.org/jira/browse/HIVE-15151


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-15151


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5ea9751 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 14fd61a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java ffb6ae3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java 167f7a5 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java 
475f2c9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 5d3fa6a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g bf78545 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 4a44173 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
9986fcf 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java a7005f1 
  ql/src/java/org/apache/hadoop/hive/ql/parse/MetaDataExportListener.java 
a17696a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSpec.java 4668271 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
4f0ead0 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ReplCopyWork.java PRE-CREATION 
  
ql/src/test/org/apache/hadoop/hive/ql/parse/TestReplicationSemanticAnalyzer.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/53591/diff/


Testing
---


Thanks,

Vaibhav Gumashta

[jira] [Created] (HIVE-15166) Provide beeline option to set the jline history max size

[Eric Lin (JIRA)]

Eric Lin created HIVE-15166:
---

 Summary: Provide beeline option to set the jline history max size
 Key: HIVE-15166
 URL: https://issues.apache.org/jira/browse/HIVE-15166
 Project: Hive
  Issue Type: Improvement
  Components: Beeline
Affects Versions: 2.1.0
Reporter: Eric Lin


Currently Beeline does not provide an option to limit the max size for beeline 
history file, in the case that each query is very big, it will flood the 
history file and slow down beeline on start up and shutdown.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15165) Flaky test: TestSparkNegativeCliDriver

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15165:
-

 Summary: Flaky test: TestSparkNegativeCliDriver
 Key: HIVE-15165
 URL: https://issues.apache.org/jira/browse/HIVE-15165
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15164) Change default RPC port for llap to be a dynamic port

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15164:
-

 Summary: Change default RPC port for llap to be a dynamic port
 Key: HIVE-15164
 URL: https://issues.apache.org/jira/browse/HIVE-15164
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15163) LLAP Finishable state updates may not go to daemons restarted on the same node

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15163:
-

 Summary: LLAP Finishable state updates may not go to daemons 
restarted on the same node
 Key: HIVE-15163
 URL: https://issues.apache.org/jira/browse/HIVE-15163
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth


Finishable state updates only go out to nodes when the source state changes. 
When a node goes down, and comes back up on the same port - updates which had 
been sent to the old node will not be sent to the new node. This could lead to 
a fragment being in a permanent preemptable state, and lead tp starvation on 
busy clusters.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15162) NPE in ATSHook

[Jason Dere (JIRA)]

Jason Dere created HIVE-15162:
-

 Summary: NPE in ATSHook
 Key: HIVE-15162
 URL: https://issues.apache.org/jira/browse/HIVE-15162
 Project: Hive
  Issue Type: Bug
  Components: Hooks
Reporter: Jason Dere
Assignee: Jason Dere


{noformat}
2016-11-08T14:21:15,025 INFO  [ATS Logger 0]: hooks.ATSHook 
(ATSHook.java:run(156)) - Failed to submit plan to ATS: 
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.hooks.ATSHook$2.run(ATSHook.java:141)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15161) migrate ColumnStats to use jackson

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-15161:
---

 Summary: migrate ColumnStats to use jackson
 Key: HIVE-15161
 URL: https://issues.apache.org/jira/browse/HIVE-15161
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich



* json.org has license issues
* jackson can provide a fully compatible alternative to it
* there are a few flakiness issues caused by the order of the map entries of 
the columns...this cat be addressed, org.json api was unfriendly in this manner 
;)





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15160) Can't group by an unselected column

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-15160:
--

 Summary: Can't group by an unselected column
 Key: HIVE-15160
 URL: https://issues.apache.org/jira/browse/HIVE-15160
 Project: Hive
  Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


If a grouping key hasn't been selected, Hive complains. For comparison, 
Postgres does not.

Example. Notice i_item_id is not selected:

{code}
select  i_item_desc
   ,i_category
   ,i_class
   ,i_current_price
   ,sum(cs_ext_sales_price) as itemrevenue
   ,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
   (partition by i_class) as revenueratio
 from   catalog_sales
 ,item
 ,date_dim
 where cs_item_sk = i_item_sk
   and i_category in ('Jewelry', 'Sports', 'Books')
   and cs_sold_date_sk = d_date_sk
 and d_date between cast('2001-01-12' as date)
and (cast('2001-01-12' as date) + 30 days)
 group by i_item_id
 ,i_item_desc
 ,i_category
 ,i_class
 ,i_current_price
 order by i_category
 ,i_class
 ,i_item_id
 ,i_item_desc
 ,revenueratio
limit 100;
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15159) LLAP ContainerRunner should not reduce the available heap while distributing to individual executors

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15159:
-

 Summary: LLAP ContainerRunner should not reduce the available heap 
while distributing to individual executors
 Key: HIVE-15159
 URL: https://issues.apache.org/jira/browse/HIVE-15159
 Project: Hive
  Issue Type: Task
  Components: llap
Reporter: Siddharth Seth
Assignee: Siddharth Seth
 Attachments: HIVE-15159.01.patch

hive.llap.daemon.memory.per.instance.mb is set to the heapsize of the process 
when llap service driver is used.
ContainerRunner currently cuts this down to 80% of the available. This 
reduction is not necessary if the heapsize is used, instead of a container size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 53570: HIVE-12891 : Hive fails when java.io.tmpdir is set to a relative location

[Chaoyu Tang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53570/#review155315
---


Ship it!




Ship It!

- Chaoyu Tang


On Nov. 8, 2016, 9:22 a.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53570/
> ---
> 
> (Updated Nov. 8, 2016, 9:22 a.m.)
> 
> 
> Review request for hive, Reuben Kuhnert and Sergio Pena.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12891 : Hive fails when java.io.tmpdir is set to a relative location
> This is a continuation of this review request: 
> https://reviews.apache.org/r/42726/
> I had a look at the comment, but I think calling makeAbsolute() directly in 
> SystemVariables is not that nice. The variable value is not necessarily a 
> path, so we would need to test for that first, and I don't think we 
> need/should convert every path of every variable. The current solution only 
> converts the relative path for the java.io.tmpdir and nothing else.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 
> 1d734f9860311cb3832c8356ddd6f8ed4a6465c5 
>   common/src/java/org/apache/hadoop/hive/conf/SystemVariables.java 
> 9f59f11ca6459853b15ca80fa9751db934befc71 
>   
> common/src/java/org/apache/hadoop/hive/conf/valcoersion/JavaIOTmpdirVariableCoercion.java
>  PRE-CREATION 
>   
> common/src/java/org/apache/hadoop/hive/conf/valcoersion/VariableCoercion.java 
> PRE-CREATION 
>   common/src/test/org/apache/hadoop/hive/common/TestFileUtils.java 
> c02217ae48176f5d4869cd21cd1a27678f2ded45 
>   common/src/test/org/apache/hadoop/hive/conf/TestSystemVariables.java 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/53570/diff/
> 
> 
> Testing
> ---
> 
> Tested that if -Djava.io.tmpdir is set to a relative path we can still 
> connect using beeline and run queries.
> Added and ran unit tests.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

[jira] [Created] (HIVE-15158) Partition Table With timestamp type on S3 storage --> Error in getting fields from serde.Invalid Field null

[thauvin damien (JIRA)]

thauvin damien created HIVE-15158:
-

 Summary: Partition Table With timestamp type on S3 storage --> 
Error in getting fields from serde.Invalid Field null
 Key: HIVE-15158
 URL: https://issues.apache.org/jira/browse/HIVE-15158
 Project: Hive
  Issue Type: Bug
  Components: Clients
Affects Versions: 2.1.0
 Environment: JDK 1.8 101 
Reporter: thauvin damien


Hello 

I get the error above when i try to perform  :
hive> DESCRIBE formatted table partition (tsbucket='2016-10-28 16%3A00%3A00');
FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.DDLTask. Error in getting fields from 
serde.Invalid Field null

Here is the description of the issue.
--External table Hive with dynamic partition enable on Aws S3 storage.
--Partition Table with timestamp type .

When i perform "show partition table;" everything is fine :
hive>  show partitions table;
OK
tsbucket=2016-10-01 11%3A00%3A00
tsbucket=2016-10-28 16%3A00%3A00

And when i perform "describe FORMATTED table;" everything is fine

Is this a bug ? 

The stacktrace of hive.log :

2016-11-08T10:30:20,868 ERROR [ac3e0d48-22c5-4d04-a788-aeb004ea94f3 main([])]: 
exec.DDLTask (DDLTask.java:failed(574)) - 
org.apache.hadoop.hive.ql.metadata.HiveException: Error in getting fields from 
serde.Invalid Field null
at 
org.apache.hadoop.hive.ql.metadata.Hive.getFieldsFromDeserializer(Hive.java:3414)
at 
org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:3109)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:408)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1072)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: MetaException(message:Invalid Field null)
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.getFieldsFromDeserializer(MetaStoreUtils.java:1336)
at 
org.apache.hadoop.hive.ql.metadata.Hive.getFieldsFromDeserializer(Hive.java:3409)
... 21 more




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15157) Partition Table With timestamp type on S3 storage --> Error in getting fields from serde.Invalid Field null

[thauvin damien (JIRA)]

thauvin damien created HIVE-15157:
-

 Summary: Partition Table With timestamp type on S3 storage --> 
Error in getting fields from serde.Invalid Field null
 Key: HIVE-15157
 URL: https://issues.apache.org/jira/browse/HIVE-15157
 Project: Hive
  Issue Type: Bug
  Components: Clients
Affects Versions: 2.1.0
 Environment: JDK 1.8 101 
Reporter: thauvin damien


Hello 

I get the error above when i try to perform  :
hive> DESCRIBE formatted table partition (tsbucket='2016-10-28 16%3A00%3A00');
FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.DDLTask. Error in getting fields from 
serde.Invalid Field null

Here is the description of the issue.
--External table Hive with dynamic partition enable on Aws S3 storage.
--Partition Table with timestamp type .

When i perform "show partition table;" everything is fine :
hive>  show partitions table;
OK
tsbucket=2016-10-01 11%3A00%3A00
tsbucket=2016-10-28 16%3A00%3A00

And when i perform "describe FORMATTED table;" everything is fine

Is this a bug ? 

The stacktrace of hive.log :

2016-11-08T10:30:20,868 ERROR [ac3e0d48-22c5-4d04-a788-aeb004ea94f3 main([])]: 
exec.DDLTask (DDLTask.java:failed(574)) - 
org.apache.hadoop.hive.ql.metadata.HiveException: Error in getting fields from 
serde.Invalid Field null
at 
org.apache.hadoop.hive.ql.metadata.Hive.getFieldsFromDeserializer(Hive.java:3414)
at 
org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:3109)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:408)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1072)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: MetaException(message:Invalid Field null)
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.getFieldsFromDeserializer(MetaStoreUtils.java:1336)
at 
org.apache.hadoop.hive.ql.metadata.Hive.getFieldsFromDeserializer(Hive.java:3409)
... 21 more




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52800: HIVE-13966: DbNotificationListener: can loose DDL operation notifications

[Chaoyu Tang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52800/#review155256
---



The patch looks good, however, I have a couple of questions.


common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 764)


What kind of events need to be invoked in a separate JDO transactions? or 
did you mean the events that do not require a JDO transaction?



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 768)


I wonder why we need a separate transactional event listener in Hive. Can 
it be deferred to listener implementation? For example, a listener which needs 
be transactional first checks if it is under a transaction context, joins (or 
enlists) itself to the transaction if it is. This way might also be applicable 
to the cases where multiple backends invovle (e.g. distributed transaction). It 
is just my thought, and may not necessarily be right or feasible.



metastore/src/java/org/apache/hadoop/hive/metastore/AlterHandler.java (line 86)


I wonder if the API changes will cause some backward compatibility issue, 
if the HiveMetaStore is currently the only consumer of the AlterHandler, it 
should be fine.


- Chaoyu Tang


On Nov. 7, 2016, 11:25 p.m., Mohit Sabharwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52800/
> ---
> 
> (Updated Nov. 7, 2016, 11:25 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-13966
> https://issues.apache.org/jira/browse/HIVE-13966
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Metadata event and associated notification should be committed in the same
> JDO transaction.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 80cd5ada060331797a603848e268c7d2a78a679c 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  PRE-CREATION 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
>  81ce67bdc8fdaf11ff4fec3f255ed0021a4752c7 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java
>  af16f75e63c372c37bfd73567b777bba53f94db3 
>   metastore/src/java/org/apache/hadoop/hive/metastore/AlterHandler.java 
> dedd4497adfcc9d57090a943f6bb4f35ea87fa61 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
> 40b337a9e40ea04a37f108146853d2d1f42dcd29 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> 60e462fd06a3f84d5b87cd335afb49768cb27562 
> 
> Diff: https://reviews.apache.org/r/52800/diff/
> 
> 
> Testing
> ---
> 
> Enhanced TestDbNotificationListener
> 
> 
> Thanks,
> 
> Mohit Sabharwal
> 
>

Re: HIVE-1555 [JDBCStorageHandler]

[Dmitry Zagorulkin]

I have solved the first problem.
Looking forward for your clarification about other problems.

Thanks in advance.

> On 8 Nov 2016, at 13:08, Dmitry Zagorulkin  wrote:
> 
> Hello everyone!
> 
> I have started work on HIVE-1555 feature.
> 
> There are few questions:
> 
> 1)  Current module structure looks like:
> 
> ├── jdbc-handler.iml
> ├── pom.xml
> └── src
>├── java
>│   └── org
>│   └── apache
>│   └── hive
>│   └── storagehandler
>│   ├── JDBCStorageHandler.java
>│   ├── db
>│   │   ├── HiveJDBCTypeBridge.java
>│   │   ├── HiveJDBCVendorBridge.java
>│   │   └── mysql
>│   │   ├── HiveMySqlTypeBridge.java
>│   │   └── HiveMySqlVendorBridge.java
>│   └── serde
>│   └── JDBCSerde.java
>├── main
>│   └── resources
>└── test
>└── java
> 
> pom.xml fragment:
> 
>  
>${basedir}/src/java
>${basedir}/src/test
>
>
>org.apache.maven.plugins
>maven-jar-plugin
>
>
>
>test-jar
>
>
>
>
>
>
> 
> So this module builds fine, but i could not see jdbc-handler-XXX-.jar inside 
> lib directory.
> What should i add in assembly build.xml?
> 
> 2) What is the main difference between HiveStoragePredicateHandler and 
> HiveStorageHandler? I’ve found that we need to implement 
> HiveStoragePredicateHandler if storage handler needs support decomposition of 
> predicates being pushed down into table scans. Is it true that RDBMS does not 
> have this feature and i need to implement HiveStorageHandler?
> 
> 3) I’ve found many useful classes like 
> DBInputFormat,DBOutputFormat,{Oracle|Mysql*.Reader} which exists in 
> org.apache.hadoop.mapreduce.lib.db but org.apache.hadoop packages are 
> excluded from build. Should i use this classes in my code? Should i need to 
> exclude filtering for org.apache.hadoop group?
> 
> 
> Thanks,
> 
> Dmitry Zagorulkin
> 
> 
> 
> 
> 

HIVE-1555 [JDBCStorageHandler]

[Dmitry Zagorulkin]

Hello everyone!

I have started work on HIVE-1555 feature.

There are few questions:

1)  Current module structure looks like:

├── jdbc-handler.iml
├── pom.xml
└── src
├── java
│   └── org
│   └── apache
│   └── hive
│   └── storagehandler
│   ├── JDBCStorageHandler.java
│   ├── db
│   │   ├── HiveJDBCTypeBridge.java
│   │   ├── HiveJDBCVendorBridge.java
│   │   └── mysql
│   │   ├── HiveMySqlTypeBridge.java
│   │   └── HiveMySqlVendorBridge.java
│   └── serde
│   └── JDBCSerde.java
├── main
│   └── resources
└── test
└── java

pom.xml fragment:

  
${basedir}/src/java
${basedir}/src/test


org.apache.maven.plugins
maven-jar-plugin



test-jar







So this module builds fine, but i could not see jdbc-handler-XXX-.jar inside 
lib directory.
What should i add in assembly build.xml?

2) What is the main difference between HiveStoragePredicateHandler and 
HiveStorageHandler? I’ve found that we need to implement 
HiveStoragePredicateHandler if storage handler needs support decomposition of 
predicates being pushed down into table scans. Is it true that RDBMS does not 
have this feature and i need to implement HiveStorageHandler?

3) I’ve found many useful classes like 
DBInputFormat,DBOutputFormat,{Oracle|Mysql*.Reader} which exists in 
org.apache.hadoop.mapreduce.lib.db but org.apache.hadoop packages are excluded 
from build. Should i use this classes in my code? Should i need to exclude 
filtering for org.apache.hadoop group?


Thanks,

Dmitry Zagorulkin

[jira] [Created] (HIVE-15156) Support Nested Column Field Pruning for Parquet Vectorized Reader

[Ferdinand Xu (JIRA)]

Ferdinand Xu created HIVE-15156:
---

 Summary: Support Nested Column Field Pruning for Parquet 
Vectorized Reader
 Key: HIVE-15156
 URL: https://issues.apache.org/jira/browse/HIVE-15156
 Project: Hive
  Issue Type: Sub-task
Reporter: Ferdinand Xu


As in HIVE-15055, we need support nested column fields pruning for vectorized 
reader as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15155) Change Hive version shortname to 2.2.0

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-15155:
--

 Summary: Change Hive version shortname to 2.2.0
 Key: HIVE-15155
 URL: https://issues.apache.org/jira/browse/HIVE-15155
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.1.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
Priority: Minor


Pointing to 2.1.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15154) Fix rest of q test file changes in branch-2.1

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-15154:
--

 Summary: Fix rest of q test file changes in branch-2.1
 Key: HIVE-15154
 URL: https://issues.apache.org/jira/browse/HIVE-15154
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Affects Versions: 2.1.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15153) Loading data from files having invalid extensions or no extension

[Pallavi Singh (JIRA)]

Pallavi Singh created HIVE-15153:


 Summary: Loading data from files having invalid extensions or no 
extension
 Key: HIVE-15153
 URL: https://issues.apache.org/jira/browse/HIVE-15153
 Project: Hive
  Issue Type: Bug
Reporter: Pallavi Singh


When we execute the following queries:

LOAD DATA INPATH 'hdfs://localhost:54310/user/hive/warehouse/file1.csv.csv' 
INTO TABLE emp;

LOAD DATA INPATH 
'hdfs://localhost:54310/user/hive/warehouse/file1.csv.csv.csv.csv' INTO TABLE 
emp;

LOAD DATA INPATH 'hdfs://localhost:54310/user/hive/warehouse/file1.txttt' INTO 
TABLE emp;

LOAD DATA INPATH 'hdfs://localhost:54310/user/hive/warehouse/file1' INTO TABLE 
emp;

LOAD DATA INPATH 'hdfs://localhost:54310/user/hive/warehouse/file1.txt.bat.csv' 
INTO TABLE emp;

We should get Input File Error, but we are able to load data successfully into 
hive table



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Work on HIVE-6009

[naveen mahadevuni]

Hi,

I'm starting as a Hive contributor. I would like to work on HIVE-6009
. I found some related
JIRAs HIVE-345 , HIVE-2867
 and HIVE-6040
. Is anyone else working
on similar changes.

Thanks,
Naveen

[jira] [Created] (HIVE-15152) beeline.cmd does not work when JAVA_HOME contains spaces

[Gary Gregory (JIRA)]

Gary Gregory created HIVE-15152:
---

 Summary: beeline.cmd does not work when JAVA_HOME contains spaces
 Key: HIVE-15152
 URL: https://issues.apache.org/jira/browse/HIVE-15152
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.1.0
Reporter: Gary Gregory


beeline.cmd does not work when JAVA_HOME contains spaces.

When you install Java on Windows by default, Java goes in {{C:\Program Files}} 
which contains a space.

Some references like {{%FOO%\path\to\here}} should be double quoted like 
{{"%FOO%\path\to\here"}}. Even with that, I could not get it to all work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 53570: HIVE-12891 : Hive fails when java.io.tmpdir is set to a relative location

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53570/
---

Review request for hive, Reuben Kuhnert and Sergio Pena.


Repository: hive-git


Description
---

HIVE-12891 : Hive fails when java.io.tmpdir is set to a relative location
This is a continuation of this review request: 
https://reviews.apache.org/r/42726/
I had a look at the comment, but I think calling makeAbsolute() directly in 
SystemVariables is not that nice. The variable value is not necessarily a path, 
so we would need to test for that first, and I don't think we need/should 
convert every path of every variable. The current solution only converts the 
relative path for the java.io.tmpdir and nothing else.


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java 
1d734f9860311cb3832c8356ddd6f8ed4a6465c5 
  common/src/java/org/apache/hadoop/hive/conf/SystemVariables.java 
9f59f11ca6459853b15ca80fa9751db934befc71 
  
common/src/java/org/apache/hadoop/hive/conf/valcoersion/JavaIOTmpdirVariableCoercion.java
 PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/conf/valcoersion/VariableCoercion.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/TestFileUtils.java 
c02217ae48176f5d4869cd21cd1a27678f2ded45 
  common/src/test/org/apache/hadoop/hive/conf/TestSystemVariables.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/53570/diff/


Testing
---

Tested that if -Djava.io.tmpdir is set to a relative path we can still connect 
using beeline and run queries.
Added and ran unit tests.


Thanks,

Barna Zsombor Klara

[jira] [Created] (HIVE-15151) Bootstrap support for replv2

[Sushanth Sowmyan (JIRA)]

Sushanth Sowmyan created HIVE-15151:
---

 Summary: Bootstrap support for replv2
 Key: HIVE-15151
 URL: https://issues.apache.org/jira/browse/HIVE-15151
 Project: Hive
  Issue Type: Sub-task
  Components: repl
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan


We need to support the ability to bootstrap an initial state, dumping out 
currently existing dbs/tables, etc, so that incremental replication can take 
over from that point. To this end, we should implement commands such as REPL 
DUMP, REPL LOAD, REPL STATUS, as described over at 
https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)