[jira] [Commented] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14141865#comment-14141865
]
Satish Mittal commented on HIVE-7892:
-
[~leftylev] The following wiki briefly mentions various available Hive Serdes:
* [Developer Guide -- Hive SerDe |
https://cwiki.apache.org/confluence/display/Hive/DeveloperGuide#DeveloperGuide-HiveSerDe
]
We can probably mention against ThriftSerde that Thrift complex types get
mapped to Hive complex types (List-Array, Map-, Struct - Struct
respectively). With HIVE-7892, Thrift Set type gets mapped to Hive Array type.
User can setup a Thrift based table using ThriftDeserializer class and query
various Thrift fields through Hive query.
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Fix For: 0.14.0
Attachments: HIVE-7892.1.patch, HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[
https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14133653#comment-14133653
]
Satish Mittal commented on HIVE-6109:
-
Karthik, refer to the release notes. You need to set it as a job conf property.
E.g.
{noformat}
job.getConfiguration().set(hcat.dynamic.partitioning.custom.pattern,
${year}/${month}/${day}/${hour}/${minute});
{noformat}
Support customized location for EXTERNAL tables created by Dynamic
Partitioning
---
Key: HIVE-6109
URL: https://issues.apache.org/jira/browse/HIVE-6109
Project: Hive
Issue Type: Improvement
Components: HCatalog
Reporter: Satish Mittal
Assignee: Satish Mittal
Fix For: 0.13.0
Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt,
HIVE-6109.3.patch.txt, HIVE-6109.pdf
Currently when dynamic partitions are created by HCatalog, the underlying
directories for the partitions are created in a fixed 'Hive-style' format,
i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of
external table, user should be able to control the format of directories
created for dynamic partitions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8040) Commit for HIVE-7925 breaks hadoop-1 build
[
https://issues.apache.org/jira/browse/HIVE-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131330#comment-14131330
]
Satish Mittal commented on HIVE-8040:
-
Applied 2nd patch and ran 'mvn clean install -DskipTests -Phadoop-1'. Now it
failed at:
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile (default-compile)
on project hive-exec: Compilation failure: Compilation failure:
[ERROR]
/home/satish/work/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionEdge.java:[29,27]
cannot find symbol
[ERROR] symbol : class DataInputByteBuffer
[ERROR] location: package org.apache.hadoop.io
[ERROR]
/home/satish/work/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionEdge.java:[73,4]
cannot find symbol
[ERROR] symbol : class DataInputByteBuffer
[ERROR] location: class org.apache.hadoop.hive.ql.exec.tez.CustomPartitionEdge
[ERROR]
/home/satish/work/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/CustomPartitionEdge.java:[73,35]
cannot find symbol
[ERROR] symbol : class DataInputByteBuffer
[ERROR] location: class org.apache.hadoop.hive.ql.exec.tez.CustomPartitionEdge
{noformat}
Commit for HIVE-7925 breaks hadoop-1 build
--
Key: HIVE-8040
URL: https://issues.apache.org/jira/browse/HIVE-8040
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Affects Versions: 0.14.0
Reporter: Xuefu Zhang
Attachments: HIVE-8040.1.patch.txt, HIVE-8040.2.patch.txt
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
on project hive-metastore: Compilation failure
[ERROR]
/home/xzhang/apache/hive7/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:[45,37]
package org.apache.commons.math3.stat does not exist
[ERROR] - [Help 1]
{code}
Missing pom file changes maybe?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Status: In Progress (was: Patch Available)
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.patch.1.txt, HIVE-7892.patch.2.txt,
HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Attachment: (was: HIVE-7892.patch.1.txt)
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Attachment: (was: HIVE-7892.patch.2.txt)
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Attachment: HIVE-7892.1.patch
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.1.patch, HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Status: Patch Available (was: In Progress)
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.1.patch, HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-8053) Support custom path pattern when dynamic partitions are added in Hive
Satish Mittal created HIVE-8053: --- Summary: Support custom path pattern when dynamic partitions are added in Hive Key: HIVE-8053 URL: https://issues.apache.org/jira/browse/HIVE-8053 Project: Hive Issue Type: Improvement Reporter: Satish Mittal Assignee: Satish Mittal Currently when dynamic partitions get added in Hive, the path of dynamic partition gets created in native 'Hive style', i.e.key1=value/key2=value2/ Typical scenarios include: 1) insert - select, 2) export-import However if the table is external, user should be able to control the format of path created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129843#comment-14129843 ] Satish Mittal commented on HIVE-6109: - Hi karthik, Currently this setting allows customizing path pattern when dynamic partitions are added through HCatalog APIs (HCatOutputFormat in case of MR job). It's not available when dynamic partitions get added through Hive query. I have created HIVE-8053 to track it. Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Assignee: Satish Mittal Fix For: 0.13.0 Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt, HIVE-6109.3.patch.txt, HIVE-6109.pdf Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8041) Hadoop-2 build is broken with JDK6
[
https://issues.apache.org/jira/browse/HIVE-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130340#comment-14130340
]
Satish Mittal commented on HIVE-8041:
-
I am seeing this with 1.6.0_26 Oracle java on ubuntu.
Hadoop-2 build is broken with JDK6
--
Key: HIVE-8041
URL: https://issues.apache.org/jira/browse/HIVE-8041
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Affects Versions: 0.14.0
Reporter: Xuefu Zhang
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
on project hive-exec: Compilation failure
[ERROR]
/home/xzhang/apache/hive7/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIf.java:[81,1]
illegal start of expression
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8040) Commit for HIVE-7925 breaks hadoop-1 build
[
https://issues.apache.org/jira/browse/HIVE-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130363#comment-14130363
]
Satish Mittal commented on HIVE-8040:
-
Also the default profile for pre-commit builds is hadoop-2. Building both
profiles per commit might be costly too. Should there be some build to test
hadoop-1 profile periodically?
Commit for HIVE-7925 breaks hadoop-1 build
--
Key: HIVE-8040
URL: https://issues.apache.org/jira/browse/HIVE-8040
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Affects Versions: 0.14.0
Reporter: Xuefu Zhang
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
on project hive-metastore: Compilation failure
[ERROR]
/home/xzhang/apache/hive7/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:[45,37]
package org.apache.commons.math3.stat does not exist
[ERROR] - [Help 1]
{code}
Missing pom file changes maybe?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Attachment: HIVE-7892.patch.1.txt
Attaching updated patch. The test convert_enum_to_string.q works with existing
MegaStruct thrift table, which contains set columns with older description.
Fixed the description.
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.patch.1.txt, HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 25473: Thrift Set type not working with Hive
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25473/ --- (Updated Sept. 10, 2014, 7:03 a.m.) Review request for hive, Amareshwari Sriramadasu, Ashutosh Chauhan, and Navis Ryu. Changes --- The test convert_enum_to_string.q works with existing MegaStruct thrift table, which contains set columns with older description. Fixed the columns description in the updated patch. Bugs: HIVE-7892 https://issues.apache.org/jira/browse/HIVE-7892 Repository: hive-git Description --- Thrift supports List, Map and Struct complex types, which get mapped to Array, Map and Struct complex types in Hive respectively. However thrift Set type doesn't get mapped to any Hive type, and hence doesn't work with ThriftDeserializer serde. Diffs (updated) - ql/src/test/results/beelinepositive/convert_enum_to_string.q.out 24acdcd ql/src/test/results/clientpositive/convert_enum_to_string.q.out a1ef04f serde/if/test/complex.thrift 308b64c serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/SetIntString.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java 9a226b3 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardListObjectInspector.java 6eb8803 serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestThriftObjectInspectors.java 5f692fb Diff: https://reviews.apache.org/r/25473/diff/ Testing --- 1) Added Unit test along with the fix. 2) Manually tested by creating a table with ThriftDeserializer serde and having thrift set columns: a) described the table b) issued query to select the set column Thanks, Satish Mittal
Re: Timeline for release of Hive 0.14
Hi, Can you please include HIVE-7892 (Thrift Set type not working with Hive) as well? It is under code review. Regards, Satish On Tue, Sep 9, 2014 at 2:10 PM, Suma Shivaprasad sumasai.shivapra...@gmail.com wrote: Please include https://issues.apache.org/jira/browse/HIVE-7694 as well. It is currently under review by Amareshwari and should be done in the next couple of days. Thanks Suma On Mon, Sep 8, 2014 at 5:44 PM, Alan Gates ga...@hortonworks.com wrote: I'll review that. I just need the time to test it against mysql, oracle, and hopefully sqlserver. But I think we can do this post branch if we need to, as it's a bug fix rather than a feature. Alan. Damien Carol dca...@blitzbs.com September 8, 2014 at 3:19 Same request for https://issues.apache.org/jira/browse/HIVE-7689 I already provided a patch, re-based it many times and I'm waiting for a review. Regards, Le 08/09/2014 12:08, amareshwarisr . a écrit : amareshwarisr . amareshw...@gmail.com September 8, 2014 at 3:08 Would like to include https://issues.apache.org/jira/browse/HIVE-2390 and https://issues.apache.org/jira/browse/HIVE-7936 . I can review and merge them. Thanks Amareshwari Vikram Dixit vik...@hortonworks.com September 5, 2014 at 17:53 Hi Folks, I am going to start consolidating the items mentioned in this list and create a wiki page to track it. I will wait till the end of next week to create the branch taking into account Ashutosh's request. Thanks Vikram. On Fri, Sep 5, 2014 at 5:39 PM, Ashutosh Chauhan hashut...@apache.org hashut...@apache.org Ashutosh Chauhan hashut...@apache.org September 5, 2014 at 17:39 Vikram, Some of us are working on stabilizing cbo branch and trying to get it merged into trunk. We feel we are close. May I request to defer cutting the branch for few more days? Folks interested in this can track our progress here : https://issues.apache.org/jira/browse/HIVE-7946 Thanks, Ashutosh On Fri, Aug 22, 2014 at 4:09 PM, Lars Francke lars.fran...@gmail.com lars.fran...@gmail.com Lars Francke lars.fran...@gmail.com August 22, 2014 at 16:09 Thank you for volunteering to do the release. I think a 0.14 release is a good idea. I have a couple of issues I'd like to get in too: * Either HIVE-7107[0] (Fix an issue in the HiveServer1 JDBC driver) or HIVE-6977[1] (Delete HiveServer1). The former needs a review the latter a patch * HIVE-6123[2] Checkstyle in Maven needs a review HIVE-7622[3] HIVE-7543[4] are waiting for any reviews or comments on my previous thread[5]. I'd still appreciate any helpers for reviews or even just comments. I'd feel very sad if I had done all that work for nothing. Hoping this thread gives me a wider audience. Both patches fix up issues that should have been caught in earlier reviews as they are almost all Checkstyle or other style violations but they make for huge patches. I could also create hundreds of small issues or stop doing these things entirely [0] https://issues.apache.org/jira/browse/HIVE-7107 https://issues.apache.org/jira/browse/HIVE-7107 [1] https://issues.apache.org/jira/browse/HIVE-6977 https://issues.apache.org/jira/browse/HIVE-6977 [2] https://issues.apache.org/jira/browse/HIVE-6123 https://issues.apache.org/jira/browse/HIVE-6123 [3] https://issues.apache.org/jira/browse/HIVE-7622 https://issues.apache.org/jira/browse/HIVE-7622 [4] https://issues.apache.org/jira/browse/HIVE-7543 https://issues.apache.org/jira/browse/HIVE-7543 On Fri, Aug 22, 2014 at 11:01 PM, John Pullokkaran -- Sent with Postbox http://www.getpostbox.com CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- _ The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Status: In Progress (was: Patch Available)
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.patch.1.txt, HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Status: Patch Available (was: In Progress)
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.patch.1.txt, HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Status: In Progress (was: Patch Available)
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.patch.1.txt, HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Attachment: HIVE-7892.patch.2.txt
Attaching patch again to trigger build.
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.patch.1.txt, HIVE-7892.patch.2.txt,
HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Status: Patch Available (was: In Progress)
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.patch.1.txt, HIVE-7892.patch.2.txt,
HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Attachment: HIVE-7892.patch.txt
Attaching patch that resolves the issue.
The approach taken here is to essentially map thrift Set type to hive Array
type (thrift List type already maps to hive Array). Since both List and Set are
essentially collections, we can simply leverage the existing Array type,
instead of exposing a new complex type at hive level.
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Review Request 25473: Thrift Set type not working with Hive
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25473/ --- Review request for hive, Amareshwari Sriramadasu, Ashutosh Chauhan, and Navis Ryu. Bugs: HIVE-7892 https://issues.apache.org/jira/browse/HIVE-7892 Repository: hive-git Description --- Thrift supports List, Map and Struct complex types, which get mapped to Array, Map and Struct complex types in Hive respectively. However thrift Set type doesn't get mapped to any Hive type, and hence doesn't work with ThriftDeserializer serde. Diffs - serde/if/test/complex.thrift 308b64c serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/SetIntString.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java 9a226b3 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardListObjectInspector.java 6eb8803 serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestThriftObjectInspectors.java 5f692fb Diff: https://reviews.apache.org/r/25473/diff/ Testing --- 1) Added Unit test along with the fix. 2) Manually tested by creating a table with ThriftDeserializer serde and having thrift set columns: a) described the table b) issued query to select the set column Thanks, Satish Mittal
[jira] [Commented] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126866#comment-14126866
]
Satish Mittal commented on HIVE-7892:
-
Review: https://reviews.apache.org/r/25473/
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Status: Patch Available (was: Open)
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Support for Thrift Set type in Hive
Hi,
I am wondering if anyone has been able to get Thrift Set type working in
Hive.
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. (Though there is
a difference in the support for Map: Thrift map allows a key of complex
type whereas Hive map doesn't.)
However thrift Set type doesn't seem to be working. Here is an example
thrift struct:
struct setrow {
1: required seti32 ids,
2: required string name,
}
I created a Hive table using ThriftDeserializer as the serde class and
serialization.class is set to above thrift java class. However a describe
on this table shows type of column 'ids' as struct. Also issuing a select
query on set column throws SemanticException:
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
Any pointers/thoughts?
Thanks,
Satish
--
_
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
[jira] [Created] (HIVE-7892) Thrift Set type not working with Hive
Satish Mittal created HIVE-7892:
---
Summary: Thrift Set type not working with Hive
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Thrift supports List, Map and Struct complex types, which get mapped to Array,
Map and Struct complex types in Hive respectively. However thrift Set type
doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH SERDEPROPERTIES
('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestring from deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959769#comment-13959769 ] Satish Mittal commented on HIVE-6109: - [~leftylev], Very nicely documented, with detailed examples that explain the feature! A minor comment: the number of partitions is not consistent in the 1st example. The schema has 6 partitions (year, month, day, hour, minute, country), but examples have 4/5 partitions. A new user might get confused. Would be good to be consistent throughout. Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Assignee: Satish Mittal Fix For: 0.13.0 Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt, HIVE-6109.3.patch.txt, HIVE-6109.pdf Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[
https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1392#comment-1392
]
Satish Mittal commented on HIVE-6109:
-
You could use the following details:
A new Job conf property hcat.dynamic.partitioning.custom.pattern is
introduced that can be configured to provide custom path pattern in case of
dynamic partitioning. E.g. suppose a table user_logs is partitioned by (year,
month, day, hour, minute, country). If user wants data for dynamic partitions
to get generated in the following location format:
hdfs://hcat/data/user_logs/2013/12/06/10/US, then this property can be set to:
${year}/${month}/${day}/${hour}/${minute}/${country}.
Support customized location for EXTERNAL tables created by Dynamic
Partitioning
---
Key: HIVE-6109
URL: https://issues.apache.org/jira/browse/HIVE-6109
Project: Hive
Issue Type: Improvement
Components: HCatalog
Reporter: Satish Mittal
Assignee: Satish Mittal
Fix For: 0.13.0
Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt,
HIVE-6109.3.patch.txt, HIVE-6109.pdf
Currently when dynamic partitions are created by HCatalog, the underlying
directories for the partitions are created in a fixed 'Hive-style' format,
i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of
external table, user should be able to control the format of directories
created for dynamic partitions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13925683#comment-13925683 ] Satish Mittal commented on HIVE-6109: - [~leftylev], can you please update the HCatalog wikidoc section on 'external tables' in 'dynamic partitioning' with release notes of this ticket? Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Assignee: Satish Mittal Fix For: 0.13.0 Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt, HIVE-6109.3.patch.txt, HIVE-6109.pdf Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13900531#comment-13900531 ] Satish Mittal commented on HIVE-6109: - Oh OK. I missed it. You can include the license header. Thanks [~susanths]! Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Assignee: Satish Mittal Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt, HIVE-6109.3.patch.txt, HIVE-6109.pdf Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[
https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-6109:
Release Note: A new Job conf property
hcat.dynamic.partitioning.custom.pattern is introduced that can be configured
to provide custom path pattern in case of dynamic partitioning. E.g.
${year}/${month}/${day}/${hour}/${minute}/${country}
Support customized location for EXTERNAL tables created by Dynamic
Partitioning
---
Key: HIVE-6109
URL: https://issues.apache.org/jira/browse/HIVE-6109
Project: Hive
Issue Type: Improvement
Components: HCatalog
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt,
HIVE-6109.3.patch.txt, HIVE-6109.pdf
Currently when dynamic partitions are created by HCatalog, the underlying
directories for the partitions are created in a fixed 'Hive-style' format,
i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of
external table, user should be able to control the format of directories
created for dynamic partitions.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Re: Review Request 16951: HIVE-6109: Support customized location for EXTERNAL tables created by Dynamic Partitioning
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16951/ --- (Updated Feb. 6, 2014, 7:19 p.m.) Review request for hive and Sushanth Sowmyan. Changes --- Uploading patch that incorporates review comments. Repository: hive-git Description --- - Attaching the patch that implements the functionality to support custom location for external tables in dynamic partitioning. Diffs (updated) - hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatConstants.java 2ee50b3 hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java a5ae1be hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java 288b7a3 hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatFileUtil.java PRE-CREATION hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatOutputFormat.java 78e77e8 hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/OutputJobInfo.java b63bdc2 hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/HCatMapReduceTest.java 77bdb9d hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatDynamicPartitioned.java d8b69c2 hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatExternalDynamicPartitioned.java 36c7945 Diff: https://reviews.apache.org/r/16951/diff/ Testing --- - Added unit test. - Tested the functionality through a sample MR program that uses HCatOutputFormat interface configured with the new custom dynamic location. Thanks, Satish Mittal
[jira] [Updated] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Mittal updated HIVE-6109: Attachment: HIVE-6109.3.patch.txt Uploading patch that incorporates review comments. Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Assignee: Satish Mittal Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt, HIVE-6109.3.patch.txt, HIVE-6109.pdf Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13893691#comment-13893691 ] Satish Mittal commented on HIVE-6109: - Copy-pasting my response to review comments from review-board: 1)Sushanth: _DYN is already defined in FosterStorageHandler, needs to have one place where it's defined. I'm okay with it being defined here if the FosterStorageHandler constant is removed and references to that are changed to reference this. Satish Mittal: Will refactor it now. I didn't want to touch unrelated code in the first cut. 2) Sushanth: whitespace errors - git refers to a bunch of these through the patch when we try to apply, please correct for final patch upload. Satish Mittal: Will do. 3) Sushanth: A bit about code readability - if we add a special case, then it makes sense to add the special case as an else, rather than as an if - that way, the default behaviour is visible first, and then the special case - please swap this around so that this is a if (!customDynamicLocationUsed) structure. Satish Mittal: Fine, will do that. 4) Sushanth: This is now significant amount of code repetition from line 720-741 above, please see if we can refactor this into a separate method. Satish Mittal: I will see if it can be easily refactored into a private method. 5) Sushanth: This becomes the primary API point with this change, wherein, a user that is using HCatOutputFormat will generate an OutputJobInfo, and then call setCustomDynamicLocation on it. This is fine for M/R users of HCat, but is something that will wind up having to be implemented for each M/R user. It might have been better to define a constant in HCatConstants, say hcat.dynamic.partitioning.custom.pattern, and to use that as a JobInfo parameter. That makes it easier for other tools to integrate with this feature. For example, with your patch, we still do not support the ability for the HCatStorer from pig to be able to write to custom dynamic partitions, while we do want to keep feature parity where possible between HCatOutputFormat and HCatStorer. In fact, as a design goal for HCat, we're trying to move away from letting(requiring) users explicitly muck around with OutputJobInfo and InputJobInfo, and stick to static calls to HCatInputFormat/HCatOutputFormat. I would like to see this call be something the HCatOutputFormat automatically calls if a jobConf parameter(as above) is set. That way, we can solve pig compatibility as well easily. Satish Mittal: Thanks for this feedback Sushanth. I had realized that with the current patch, HCatStorer is still not covered (only M/R jobs are covered), and was thinking of exposing equivalent APIs in HCatStorer to achieve it. However your suggestion sounds cleaner to me. Will do that! Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Assignee: Satish Mittal Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt, HIVE-6109.3.patch.txt, HIVE-6109.pdf Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[
https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13894263#comment-13894263
]
Satish Mittal commented on HIVE-6109:
-
The latest patch HIVE-6109.3.patch.txt introduces a new Job conf property
hcat.dynamic.partitioning.custom.pattern that can be configured to provide
custom path pattern in case of dynamic partitioning. E.g.
${year}/${month}/${day}/${hour}/${minute}/${country} in the above example.
Support customized location for EXTERNAL tables created by Dynamic
Partitioning
---
Key: HIVE-6109
URL: https://issues.apache.org/jira/browse/HIVE-6109
Project: Hive
Issue Type: Improvement
Components: HCatalog
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt,
HIVE-6109.3.patch.txt, HIVE-6109.pdf
Currently when dynamic partitions are created by HCatalog, the underlying
directories for the partitions are created in a fixed 'Hive-style' format,
i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of
external table, user should be able to control the format of directories
created for dynamic partitions.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Re: Review Request 16951: HIVE-6109: Support customized location for EXTERNAL tables created by Dynamic Partitioning
On Jan. 22, 2014, 10:25 p.m., Sushanth Sowmyan wrote: Hi, Sorry for the delay, I thought I'd published this review over the weekend, but reviewboard was unresponsive, and it looks like it didn't save. Thanks Sushanth for the review. On Jan. 22, 2014, 10:25 p.m., Sushanth Sowmyan wrote: hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java, line 73 https://reviews.apache.org/r/16951/diff/1/?file=424591#file424591line73 _DYN is already defined in FosterStorageHandler, needs to have one place where it's defined. I'm okay with it being defined here if the FosterStorageHandler constant is removed and references to that are changed to reference this. Will refactor it now. I didn't want to touch unrelated code in the first cut. On Jan. 22, 2014, 10:25 p.m., Sushanth Sowmyan wrote: hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java, line 272 https://reviews.apache.org/r/16951/diff/1/?file=424591#file424591line272 whitespace errors - git refers to a bunch of these through the patch when we try to apply, please correct for final patch upload. Will do. On Jan. 22, 2014, 10:25 p.m., Sushanth Sowmyan wrote: hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java, line 721 https://reviews.apache.org/r/16951/diff/1/?file=424591#file424591line721 A bit about code readability - if we add a special case, then it makes sense to add the special case as an else, rather than as an if - that way, the default behaviour is visible first, and then the special case - please swap this around so that this is a if (!customDynamicLocationUsed) structure. Fine, will do that. On Jan. 22, 2014, 10:25 p.m., Sushanth Sowmyan wrote: hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java, line 765 https://reviews.apache.org/r/16951/diff/1/?file=424591#file424591line765 This is now significant amount of code repetition from line 720-741 above, please see if we can refactor this into a separate method. I will see if it can be easily refactored into a private method. On Jan. 22, 2014, 10:25 p.m., Sushanth Sowmyan wrote: hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/OutputJobInfo.java, line 178 https://reviews.apache.org/r/16951/diff/1/?file=424594#file424594line178 This becomes the primary API point with this change, wherein, a user that is using HCatOutputFormat will generate an OutputJobInfo, and then call setCustomDynamicLocation on it. This is fine for M/R users of HCat, but is something that will wind up having to be implemented for each M/R user. It might have been better to define a constant in HCatConstants, say hcat.dynamic.partitioning.custom.pattern, and to use that as a JobInfo parameter. That makes it easier for other tools to integrate with this feature. For example, with your patch, we still do not support the ability for the HCatStorer from pig to be able to write to custom dynamic partitions, while we do want to keep feature parity where possible between HCatOutputFormat and HCatStorer. In fact, as a design goal for HCat, we're trying to move away from letting(requiring) users explicitly muck around with OutputJobInfo and InputJobInfo, and stick to static calls to HCatInputFormat/HCatOutputFormat. I would like to see this call be something the HCatOutputFormat automatically calls if a jobConf parameter(as above) is set. That way, we can solve pig compatibility as well easily. Thanks for this feedback Sushanth. I had realized that with the current patch, HCatStorer is still not covered (only M/R jobs are covered), and was thinking of exposing equivalent APIs in HCatStorer to achieve it. However your suggestion sounds cleaner to me. Will do that! - Satish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16951/#review32561 --- On Jan. 16, 2014, 12:09 p.m., Satish Mittal wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16951/ --- (Updated Jan. 16, 2014, 12:09 p.m.) Review request for hive and Sushanth Sowmyan. Repository: hive-git Description --- - Attaching the patch that implements the functionality to support custom location for external tables in dynamic partitioning. Diffs - hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java a5ae1be hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13878320#comment-13878320 ] Satish Mittal commented on HIVE-6109: - Hi [~sushanth], did you get a chance to look at the changes? Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Assignee: Satish Mittal Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt, HIVE-6109.pdf Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13874577#comment-13874577 ] Satish Mittal commented on HIVE-6109: - Created review request for the patch: https://reviews.apache.org/r/16951 Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Assignee: Satish Mittal Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt, HIVE-6109.pdf Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Review Request 16951: HIVE-6109: Support customized location for EXTERNAL tables created by Dynamic Partitioning
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16951/ --- Review request for hive and Sushanth Sowmyan. Repository: hive-git Description --- - Attaching the patch that implements the functionality to support custom location for external tables in dynamic partitioning. Diffs - hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java a5ae1be hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java 288b7a3 hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatFileUtil.java PRE-CREATION hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/OutputJobInfo.java b63bdc2 hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/HCatMapReduceTest.java 77bdb9d hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatDynamicPartitioned.java d8b69c2 hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatExternalDynamicPartitioned.java 36c7945 Diff: https://reviews.apache.org/r/16951/diff/ Testing --- - Added unit test. - Tested the functionality through a sample MR program that uses HCatOutputFormat interface configured with the new custom dynamic location. Thanks, Satish Mittal
Re: how to use phabricator with maven
Thanks Navis. -Satish On Thu, Jan 16, 2014 at 10:22 AM, Navis류승우 navis@nexr.com wrote: We don't use phabricator anymore except some patches on it which are made long ago. Use apache review board, instead. - Navis 2014/1/16 Satish Mittal satish.mit...@inmobi.com Hi All, The following phabricator link https://cwiki.apache.org/confluence/display/Hive/PhabricatorCodeReviewdescribes the review process with ant. However is there any way to raise a review request with mvn based setup? Thanks, Satish -- _ The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt. -- _ The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
how to use phabricator with maven
Hi All, The following phabricator link https://cwiki.apache.org/confluence/display/Hive/PhabricatorCodeReviewdescribes the review process with ant. However is there any way to raise a review request with mvn based setup? Thanks, Satish -- _ The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870929#comment-13870929 ] Satish Mittal commented on HIVE-6109: - This test has passed on my setup, with the following command: - cd itests - mvn test -Phadoop-1 Not sure if it is a random test failure (the test is not related to the patch). Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt, HIVE-6109.pdf Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Mittal updated HIVE-6109: Attachment: HIVE-6109.1.patch.txt Attaching the patch that implements the functionality to support custom location for external tables in dynamic partitioning. Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Attachments: HIVE-6109.1.patch.txt Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Mittal updated HIVE-6109: Status: Patch Available (was: Open) Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Attachments: HIVE-6109.1.patch.txt Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Mittal updated HIVE-6109: Attachment: HIVE-6109.2.patch.txt Updated patch. Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Mittal updated HIVE-6109: Attachment: HIVE-6109.pdf Attaching a document that describes the approach taken by the patch in designing/implementing the functionality. Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt, HIVE-6109.pdf Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13868665#comment-13868665 ] Satish Mittal commented on HIVE-6109: - Working on this issue. Can someone please assign it to me? Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
Satish Mittal created HIVE-6109: --- Summary: Support customized location for EXTERNAL tables created by Dynamic Partitioning Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13856862#comment-13856862 ] Satish Mittal commented on HIVE-6109: - As an example, suppose a table user_logs is partitioned by (year, month, day, hour, minute, country) and stored at location hdfs://hcat/data/user_logs. Currently dynamic partitions for user_logs would get created at HDFS locations in the fixed format hdfs://hcat/data/user_logs/year=2013/month=12/hour=06/minute=10/country=US. However in our use-case this breaks other data pipeline applications which expect the data to arrive in a different format: hdfs://hcat/data/user_logs/2013/12/06/10/US. There could be similar other use-cases possible. Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5011) Dynamic partitioning in HCatalog broken on external tables
[
https://issues.apache.org/jira/browse/HIVE-5011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13856864#comment-13856864
]
Satish Mittal commented on HIVE-5011:
-
Created [HIVE-6109|https://issues.apache.org/jira/browse/HIVE-6109] to track
the original issue.
Dynamic partitioning in HCatalog broken on external tables
--
Key: HIVE-5011
URL: https://issues.apache.org/jira/browse/HIVE-5011
Project: Hive
Issue Type: Bug
Components: HCatalog
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan
Priority: Critical
Fix For: 0.12.0
Attachments: HIVE-5011.patch
Dynamic partitioning with HCatalog has been broken as a result of
HCATALOG-500 trying to support user-set paths for external tables.
The goal there was to be able to support other custom destinations apart from
the normal hive-style partitions. However, it is not currently possible for
users to set paths for dynamic ptn writes, since we don't support any way for
users to specify patterns(like, say $\{rootdir\}/$v1.$v2/) into which
writes happen, only locations, and the values for dyn. partitions are not
known ahead of time. Also, specifying a custom path messes with the way
dynamic ptn. code tries to determine what was written to where from the
output committer, which means that even if we supported patterned-writes
instead of location-writes, we still have to do some more deep diving into
the output committer code to support it.
Thus, my current proposal is that we honour writes to user-specified paths
for external tables *ONLY* for static partition writes - i.e., if we can
determine that the write is a dyn. ptn. write, we will ignore the user
specification. (Note that this does not mean we ignore the table's external
location - we honour that - we just don't honour any HCatStorer/etc provided
additional location - we stick to what metadata tells us the root location is.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5621) Target tar does not exist in the project hcatalog.
[
https://issues.apache.org/jira/browse/HIVE-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848253#comment-13848253
]
Satish Mittal commented on HIVE-5621:
-
I got the same error while giving target tar. On specifying the target as
package, the build proceeds ahead but fails with following error:
{code}
satish@dev1$ /usr/bin/ant -Dhcatalog.version=0.5.0
-Dforrest.home=/home/satish/work/forrest/apache-forrest-0.9 package
[artifact:dependencies] Unable to resolve artifact: Missing:
[artifact:dependencies] --
[artifact:dependencies] 1) org.apache.hcatalog:hcatalog-core:jar:0.5.0
[artifact:dependencies]
[artifact:dependencies]
[artifact:dependencies] Path to dependency:
[artifact:dependencies] 1)
org.apache.hcatalog:hcatalog-pig-adapter:jar:0.5.0-incubating
[artifact:dependencies] 2) org.apache.hcatalog:hcatalog-core:jar:0.5.0
[artifact:dependencies] --
{code}
This issue got resolved by specifying project.version as 0.5.0-incubating
instead of 0.5.0 in build command line. The reason for error is that build
installs hcatalog-core artifact as hcatalog-core-0.5.0-incubating.jar. However
because of wrong project.version, the dependent projects were instead searching
for hcatalog-core-0.5.0.jar.
Build succeeded with:
{code}
satish@dev1$ /usr/bin/ant -Dhcatalog.version=0.5.0-incubating
-Dforrest.home=/home/satish/work/forrest/apache-forrest-0.9 package
{code}
Probably the build instruction at the
[link|http://hive.apache.org/docs/hcat_r0.5.0/install.html] needs to be fixed.
Target tar does not exist in the project hcatalog.
--
Key: HIVE-5621
URL: https://issues.apache.org/jira/browse/HIVE-5621
Project: Hive
Issue Type: Bug
Reporter: tony
Buildfile: /home/murkuser/hcatalog-src-0.5.0-incubating/build.xml
BUILD FAILED
Target tar does not exist in the project hcatalog.
Total time: 0 seconds
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
