[jira] [Commented] (HIVE-15157) Partition Table With timestamp type on S3 storage --> Error in getting fields from serde.Invalid Field null
[ https://issues.apache.org/jira/browse/HIVE-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443743#comment-16443743 ] thauvin damien commented on HIVE-15157: --- Thank's [~jcamachorodriguez] and [~prasanth_j] Is there a chance that this patch will be released in version 2.1.0 and 2.3.3? > Partition Table With timestamp type on S3 storage --> Error in getting fields > from serde.Invalid Field null > --- > > Key: HIVE-15157 > URL: https://issues.apache.org/jira/browse/HIVE-15157 > Project: Hive > Issue Type: Bug > Components: Clients >Affects Versions: 2.1.0 > Environment: JDK 1.8 101 >Reporter: thauvin damien >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Labels: timestamp > Attachments: HIVE-15157.01.patch, HIVE-15157.02.patch > > > Hello > I get the error above when i try to perform : > hive> DESCRIBE formatted table partition (tsbucket='2016-10-28 16%3A00%3A00'); > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask. Error in getting fields from > serde.Invalid Field null > Here is the description of the issue. > --External table Hive with dynamic partition enable on Aws S3 storage. > --Partition Table with timestamp type . > When i perform "show partition table;" everything is fine : > hive> show partitions table; > OK > tsbucket=2016-10-01 11%3A00%3A00 > tsbucket=2016-10-28 16%3A00%3A00 > And when i perform "describe FORMATTED table;" everything is fine > Is this a bug ? > The stacktrace of hive.log : > 2016-11-08T10:30:20,868 ERROR [ac3e0d48-22c5-4d04-a788-aeb004ea94f3 > main([])]: exec.DDLTask (DDLTask.java:failed(574)) - > org.apache.hadoop.hive.ql.metadata.HiveException: Error in getting fields > from serde.Invalid Field null > at > org.apache.hadoop.hive.ql.metadata.Hive.getFieldsFromDeserializer(Hive.java:3414) > at > org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:3109) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:408) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1072) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > Caused by: MetaException(message:Invalid Field null) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.getFieldsFromDeserializer(MetaStoreUtils.java:1336) > at > org.apache.hadoop.hive.ql.metadata.Hive.getFieldsFromDeserializer(Hive.java:3409) > ... 21 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-6589) Automatically add partitions for external tables
[
https://issues.apache.org/jira/browse/HIVE-6589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056006#comment-16056006
]
thauvin damien commented on HIVE-6589:
--
Yes this issue can help everyone in many workoad .
> Automatically add partitions for external tables
>
>
> Key: HIVE-6589
> URL: https://issues.apache.org/jira/browse/HIVE-6589
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 0.14.0
>Reporter: Ken Dallmeyer
>Assignee: Dharmendra Pratap Singh
>
> I have a data stream being loaded into Hadoop via Flume. It loads into a date
> partition folder in HDFS. The path looks like this:
> {code}/flume/my_data//MM/DD/HH
> /flume/my_data/2014/03/02/01
> /flume/my_data/2014/03/02/02
> /flume/my_data/2014/03/02/03{code}
> On top of it I create an EXTERNAL hive table to do querying. As of now, I
> have to manually add partitions. What I want is for EXTERNAL tables, Hive
> should "discover" those partitions. Additionally I would like to specify a
> partition pattern so that when I query Hive will know to use the partition
> pattern to find the HDFS folder.
> So something like this:
> {code}CREATE EXTERNAL TABLE my_data (
> col1 STRING,
> col2 INT
> )
> PARTITIONED BY (
> dt STRING,
> hour STRING
> )
> LOCATION
> '/flume/mydata'
> TBLPROPERTIES (
> 'hive.partition.spec' = 'dt=$Y-$M-$D, hour=$H',
> 'hive.partition.spec.location' = '$Y/$M/$D/$H',
> );
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-15157) Partition Table With timestamp type on S3 storage --> Error in getting fields from serde.Invalid Field null
[ https://issues.apache.org/jira/browse/HIVE-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thauvin damien updated HIVE-15157: -- Priority: Major (was: Critical) > Partition Table With timestamp type on S3 storage --> Error in getting fields > from serde.Invalid Field null > --- > > Key: HIVE-15157 > URL: https://issues.apache.org/jira/browse/HIVE-15157 > Project: Hive > Issue Type: Bug > Components: Clients >Affects Versions: 2.1.0 > Environment: JDK 1.8 101 >Reporter: thauvin damien > > Hello > I get the error above when i try to perform : > hive> DESCRIBE formatted table partition (tsbucket='2016-10-28 16%3A00%3A00'); > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask. Error in getting fields from > serde.Invalid Field null > Here is the description of the issue. > --External table Hive with dynamic partition enable on Aws S3 storage. > --Partition Table with timestamp type . > When i perform "show partition table;" everything is fine : > hive> show partitions table; > OK > tsbucket=2016-10-01 11%3A00%3A00 > tsbucket=2016-10-28 16%3A00%3A00 > And when i perform "describe FORMATTED table;" everything is fine > Is this a bug ? > The stacktrace of hive.log : > 2016-11-08T10:30:20,868 ERROR [ac3e0d48-22c5-4d04-a788-aeb004ea94f3 > main([])]: exec.DDLTask (DDLTask.java:failed(574)) - > org.apache.hadoop.hive.ql.metadata.HiveException: Error in getting fields > from serde.Invalid Field null > at > org.apache.hadoop.hive.ql.metadata.Hive.getFieldsFromDeserializer(Hive.java:3414) > at > org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:3109) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:408) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1072) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > Caused by: MetaException(message:Invalid Field null) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.getFieldsFromDeserializer(MetaStoreUtils.java:1336) > at > org.apache.hadoop.hive.ql.metadata.Hive.getFieldsFromDeserializer(Hive.java:3409) > ... 21 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15157) Partition Table With timestamp type on S3 storage --> Error in getting fields from serde.Invalid Field null
[
https://issues.apache.org/jira/browse/HIVE-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696054#comment-15696054
]
thauvin damien commented on HIVE-15157:
---
Hello You can reproduce the problem with the following commands:
CREATE EXTERNAL TABLE if not exists tablesource (tsbucket TIMESTAMP,geo_country
STRING,aid BIGINT) stored as orc LOCATION 's3://XXX/xxx'
TBLPROPERTIES("orc.compress"="ZLIB");
INSERT INTO TABLE tablesource VALUES ('2016-11-02 17:00:00','France',7453);
INSERT INTO TABLE tablesource VALUES ('2016-11-02 18:00:00','Canada',57008);
INSERT INTO TABLE tablesource VALUES ('2016-11-02 17:00:00','Morocco',58097);
CREATE EXTERNAL TABLE if not exists tabledestination (geo_country STRING,aid
BIGINT) partitioned by (tsbucket TIMESTAMP) stored as orc LOCATION
's3://XXX/xxx' TBLPROPERTIES("orc.compress"="ZLIB");
INSERT INTO TABLE tabledestination PARTITION (tsbucket) SELECT
geo_country,aid,tsbucket FROM tablesource;
DESCRIBE formatted tabledestination partition (tsbucket='2016-11-02 17:00:00');
Thanks for you help
> Partition Table With timestamp type on S3 storage --> Error in getting fields
> from serde.Invalid Field null
> ---
>
> Key: HIVE-15157
> URL: https://issues.apache.org/jira/browse/HIVE-15157
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 2.1.0
> Environment: JDK 1.8 101
>Reporter: thauvin damien
>Priority: Critical
>
> Hello
> I get the error above when i try to perform :
> hive> DESCRIBE formatted table partition (tsbucket='2016-10-28 16%3A00%3A00');
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. Error in getting fields from
> serde.Invalid Field null
> Here is the description of the issue.
> --External table Hive with dynamic partition enable on Aws S3 storage.
> --Partition Table with timestamp type .
> When i perform "show partition table;" everything is fine :
> hive> show partitions table;
> OK
> tsbucket=2016-10-01 11%3A00%3A00
> tsbucket=2016-10-28 16%3A00%3A00
> And when i perform "describe FORMATTED table;" everything is fine
> Is this a bug ?
> The stacktrace of hive.log :
> 2016-11-08T10:30:20,868 ERROR [ac3e0d48-22c5-4d04-a788-aeb004ea94f3
> main([])]: exec.DDLTask (DDLTask.java:failed(574)) -
> org.apache.hadoop.hive.ql.metadata.HiveException: Error in getting fields
> from serde.Invalid Field null
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getFieldsFromDeserializer(Hive.java:3414)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:3109)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:408)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1072)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: MetaException(message:Invalid Field null)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getFieldsFromDeserializer(MetaStoreUtils.java:1336)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getFieldsFromDeserializer(Hive.java:3409)
> ... 21 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15157) Partition Table With timestamp type on S3 storage --> Error in getting fields from serde.Invalid Field null
[ https://issues.apache.org/jira/browse/HIVE-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thauvin damien updated HIVE-15157: -- Priority: Critical (was: Major) > Partition Table With timestamp type on S3 storage --> Error in getting fields > from serde.Invalid Field null > --- > > Key: HIVE-15157 > URL: https://issues.apache.org/jira/browse/HIVE-15157 > Project: Hive > Issue Type: Bug > Components: Clients >Affects Versions: 2.1.0 > Environment: JDK 1.8 101 >Reporter: thauvin damien >Priority: Critical > > Hello > I get the error above when i try to perform : > hive> DESCRIBE formatted table partition (tsbucket='2016-10-28 16%3A00%3A00'); > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask. Error in getting fields from > serde.Invalid Field null > Here is the description of the issue. > --External table Hive with dynamic partition enable on Aws S3 storage. > --Partition Table with timestamp type . > When i perform "show partition table;" everything is fine : > hive> show partitions table; > OK > tsbucket=2016-10-01 11%3A00%3A00 > tsbucket=2016-10-28 16%3A00%3A00 > And when i perform "describe FORMATTED table;" everything is fine > Is this a bug ? > The stacktrace of hive.log : > 2016-11-08T10:30:20,868 ERROR [ac3e0d48-22c5-4d04-a788-aeb004ea94f3 > main([])]: exec.DDLTask (DDLTask.java:failed(574)) - > org.apache.hadoop.hive.ql.metadata.HiveException: Error in getting fields > from serde.Invalid Field null > at > org.apache.hadoop.hive.ql.metadata.Hive.getFieldsFromDeserializer(Hive.java:3414) > at > org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:3109) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:408) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1072) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > Caused by: MetaException(message:Invalid Field null) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.getFieldsFromDeserializer(MetaStoreUtils.java:1336) > at > org.apache.hadoop.hive.ql.metadata.Hive.getFieldsFromDeserializer(Hive.java:3409) > ... 21 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-15158) Partition Table With timestamp type on S3 storage --> Error in getting fields from serde.Invalid Field null
[ https://issues.apache.org/jira/browse/HIVE-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thauvin damien resolved HIVE-15158. --- Resolution: Duplicate duplicate with this jira HIVE-15157 > Partition Table With timestamp type on S3 storage --> Error in getting fields > from serde.Invalid Field null > --- > > Key: HIVE-15158 > URL: https://issues.apache.org/jira/browse/HIVE-15158 > Project: Hive > Issue Type: Bug > Components: Clients >Affects Versions: 2.1.0 > Environment: JDK 1.8 101 >Reporter: thauvin damien > > Hello > I get the error above when i try to perform : > hive> DESCRIBE formatted table partition (tsbucket='2016-10-28 16%3A00%3A00'); > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask. Error in getting fields from > serde.Invalid Field null > Here is the description of the issue. > --External table Hive with dynamic partition enable on Aws S3 storage. > --Partition Table with timestamp type . > When i perform "show partition table;" everything is fine : > hive> show partitions table; > OK > tsbucket=2016-10-01 11%3A00%3A00 > tsbucket=2016-10-28 16%3A00%3A00 > And when i perform "describe FORMATTED table;" everything is fine > Is this a bug ? > The stacktrace of hive.log : > 2016-11-08T10:30:20,868 ERROR [ac3e0d48-22c5-4d04-a788-aeb004ea94f3 > main([])]: exec.DDLTask (DDLTask.java:failed(574)) - > org.apache.hadoop.hive.ql.metadata.HiveException: Error in getting fields > from serde.Invalid Field null > at > org.apache.hadoop.hive.ql.metadata.Hive.getFieldsFromDeserializer(Hive.java:3414) > at > org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:3109) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:408) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1072) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > Caused by: MetaException(message:Invalid Field null) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.getFieldsFromDeserializer(MetaStoreUtils.java:1336) > at > org.apache.hadoop.hive.ql.metadata.Hive.getFieldsFromDeserializer(Hive.java:3409) > ... 21 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
