[jira] [Commented] (HIVE-6784) parquet-hive should allow column type change
[ https://issues.apache.org/jira/browse/HIVE-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14094118#comment-14094118 ] Tongjie Chen commented on HIVE-6784: if you backport this jira, it will be OK. https://issues.apache.org/jira/browse/HIVE-6785 > parquet-hive should allow column type change > > > Key: HIVE-6784 > URL: https://issues.apache.org/jira/browse/HIVE-6784 > Project: Hive > Issue Type: Bug > Components: File Formats, Serializers/Deserializers >Affects Versions: 0.13.0 >Reporter: Tongjie Chen > Fix For: 0.14.0 > > Attachments: HIVE-6784.1.patch.txt, HIVE-6784.2.patch.txt > > > see also in the following parquet issue: > https://github.com/Parquet/parquet-mr/issues/323 > Currently, if we change parquet format hive table using "alter table > parquet_table change c1 c1 bigint " ( assuming original type of c1 is int), > it will result in exception thrown from SerDe: > "org.apache.hadoop.io.IntWritable cannot be cast to > org.apache.hadoop.io.LongWritable" in query runtime. > This is different behavior from hive (using other file format), where it will > try to perform cast (null value in case of incompatible type). > Parquet Hive's RecordReader returns an ArrayWritable (based on schema stored > in footers of parquet files); ParquetHiveSerDe also creates an corresponding > ArrayWritableObjectInspector (but using column type info from metastore). > Whenever there is column type change, the objector inspector will throw > exception, since WritableLongObjectInspector cannot inspect an IntWritable > etc... > Conversion has to happen somewhere if we want to allow type change. SerDe's > deserialize method seems a natural place for it. > Currently, serialize method calls createStruct (then createPrimitive) for > every record, but it creates a new object regardless, which seems expensive. > I think that could be optimized a bit by just returning the object passed if > already of the right type. deserialize also reuse this method, if there is a > type change, there will be new object to be created, which I think is > inevitable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6784) parquet-hive should allow column type change
[
https://issues.apache.org/jira/browse/HIVE-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14094075#comment-14094075
]
Zhichun Wu commented on HIVE-6784:
--
@ [~tongjie] ,
bq. The exception raised from changing type actually only happens to
non-partitioned tables. For partitioned tables, if there is type change in
table level, there will be an ObjectInspectorConverter (in parquet's case —
StructConverter) to convert type between partition and table.
I've test changing type with parquet partitioned table in hive 0.13 and found
that this issue also existed in partitioned tables. After changing type, when
I query old partition data, it would throw following exception:
{code}
Cannot inspect java.util.ArrayList
at
org.apache.hadoop.hive.ql.io.parquet.serde.ArrayWritableObjectInspector.getStructFieldData(ArrayWritableObjectInspector.java:139)
at
org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldData(UnionStructObjectInspector.java:135)
at
org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:349)
at
org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:193)
at
org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:179)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:545)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
{code}
> parquet-hive should allow column type change
>
>
> Key: HIVE-6784
> URL: https://issues.apache.org/jira/browse/HIVE-6784
> Project: Hive
> Issue Type: Bug
> Components: File Formats, Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Tongjie Chen
> Fix For: 0.14.0
>
> Attachments: HIVE-6784.1.patch.txt, HIVE-6784.2.patch.txt
>
>
> see also in the following parquet issue:
> https://github.com/Parquet/parquet-mr/issues/323
> Currently, if we change parquet format hive table using "alter table
> parquet_table change c1 c1 bigint " ( assuming original type of c1 is int),
> it will result in exception thrown from SerDe:
> "org.apache.hadoop.io.IntWritable cannot be cast to
> org.apache.hadoop.io.LongWritable" in query runtime.
> This is different behavior from hive (using other file format), where it will
> try to perform cast (null value in case of incompatible type).
> Parquet Hive's RecordReader returns an ArrayWritable (based on schema stored
> in footers of parquet files); ParquetHiveSerDe also creates an corresponding
> ArrayWritableObjectInspector (but using column type info from metastore).
> Whenever there is column type change, the objector inspector will throw
> exception, since WritableLongObjectInspector cannot inspect an IntWritable
> etc...
> Conversion has to happen somewhere if we want to allow type change. SerDe's
> deserialize method seems a natural place for it.
> Currently, serialize method calls createStruct (then createPrimitive) for
> every record, but it creates a new object regardless, which seems expensive.
> I think that could be optimized a bit by just returning the object passed if
> already of the right type. deserialize also reuse this method, if there is a
> type change, there will be new object to be created, which I think is
> inevitable.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6784) parquet-hive should allow column type change
[ https://issues.apache.org/jira/browse/HIVE-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967292#comment-13967292 ] Tongjie Chen commented on HIVE-6784: OK, I will cancel this patch. The exception raised from changing type actually only happens to non-partitioned tables. For partitioned tables, if there is type change in table level, there will be an ObjectInspectorConverter (in parquet's case --- StructConverter) to convert type between partition and table. For non-partitioned tables, the ObjectInspectorConverter is always IdentityConverter, which passes the deserialized object as it is, causing type mismatch between object and ObjectInspector. For now, we can live with the workaround (insert overwrite table) given that it is only affecting non-partitioned tables and relatively rare. > parquet-hive should allow column type change > > > Key: HIVE-6784 > URL: https://issues.apache.org/jira/browse/HIVE-6784 > Project: Hive > Issue Type: Bug > Components: File Formats, Serializers/Deserializers >Affects Versions: 0.13.0 >Reporter: Tongjie Chen > Fix For: 0.14.0 > > Attachments: HIVE-6784.1.patch.txt, HIVE-6784.2.patch.txt > > > see also in the following parquet issue: > https://github.com/Parquet/parquet-mr/issues/323 > Currently, if we change parquet format hive table using "alter table > parquet_table change c1 c1 bigint " ( assuming original type of c1 is int), > it will result in exception thrown from SerDe: > "org.apache.hadoop.io.IntWritable cannot be cast to > org.apache.hadoop.io.LongWritable" in query runtime. > This is different behavior from hive (using other file format), where it will > try to perform cast (null value in case of incompatible type). > Parquet Hive's RecordReader returns an ArrayWritable (based on schema stored > in footers of parquet files); ParquetHiveSerDe also creates an corresponding > ArrayWritableObjectInspector (but using column type info from metastore). > Whenever there is column type change, the objector inspector will throw > exception, since WritableLongObjectInspector cannot inspect an IntWritable > etc... > Conversion has to happen somewhere if we want to allow type change. SerDe's > deserialize method seems a natural place for it. > Currently, serialize method calls createStruct (then createPrimitive) for > every record, but it creates a new object regardless, which seems expensive. > I think that could be optimized a bit by just returning the object passed if > already of the right type. deserialize also reuse this method, if there is a > type change, there will be new object to be created, which I think is > inevitable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6784) parquet-hive should allow column type change
[
https://issues.apache.org/jira/browse/HIVE-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13966341#comment-13966341
]
Justin Coffey commented on HIVE-6784:
-
You've cited a "lazy" serde. Parquet is not "lazy". It is similar to ORC.
Have a look ORC's deserialize() method
(org.apache.hadoop.hive.ql.io.orc.OrcSerde):
{code}
@Override
public Object deserialize(Writable writable) throws SerDeException {
return writable;
}
{code}
A quick look through ORC code indicates to me that they don't do any reparsing
(though I might have missed something).
Looking through other serde's not a single one (that I checked) reparses
values. Value parsing is handled in ObjectInspectors (poke around
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils).
In my opinion, the *substantial* performance penalty that you are introducing
with this patch is going to be a much bigger negative to adopting parquet than
obliging people to rebuild their data set in the rare event that you have to
change a type.
And if you do need to change a type, insert overwrite table is a good work
around.
-1
> parquet-hive should allow column type change
>
>
> Key: HIVE-6784
> URL: https://issues.apache.org/jira/browse/HIVE-6784
> Project: Hive
> Issue Type: Bug
> Components: File Formats, Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Tongjie Chen
> Fix For: 0.14.0
>
> Attachments: HIVE-6784.1.patch.txt, HIVE-6784.2.patch.txt
>
>
> see also in the following parquet issue:
> https://github.com/Parquet/parquet-mr/issues/323
> Currently, if we change parquet format hive table using "alter table
> parquet_table change c1 c1 bigint " ( assuming original type of c1 is int),
> it will result in exception thrown from SerDe:
> "org.apache.hadoop.io.IntWritable cannot be cast to
> org.apache.hadoop.io.LongWritable" in query runtime.
> This is different behavior from hive (using other file format), where it will
> try to perform cast (null value in case of incompatible type).
> Parquet Hive's RecordReader returns an ArrayWritable (based on schema stored
> in footers of parquet files); ParquetHiveSerDe also creates an corresponding
> ArrayWritableObjectInspector (but using column type info from metastore).
> Whenever there is column type change, the objector inspector will throw
> exception, since WritableLongObjectInspector cannot inspect an IntWritable
> etc...
> Conversion has to happen somewhere if we want to allow type change. SerDe's
> deserialize method seems a natural place for it.
> Currently, serialize method calls createStruct (then createPrimitive) for
> every record, but it creates a new object regardless, which seems expensive.
> I think that could be optimized a bit by just returning the object passed if
> already of the right type. deserialize also reuse this method, if there is a
> type change, there will be new object to be created, which I think is
> inevitable.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6784) parquet-hive should allow column type change
[
https://issues.apache.org/jira/browse/HIVE-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13965702#comment-13965702
]
Tongjie Chen commented on HIVE-6784:
Not allowing changing types of columns would be negative for adopting parquet
--- at least this is a different behavior from using other file format.
Regarding performance penalty,
1) if you look at current implementation of LazyInteger, LayzLong etc (from
LazySimpleSerDe, in package org.apache.hadoop.hive.serde2.lazy), they try to
parseInt, parseLong for every column (all initially represented as string, the
parsing overhead occurs even if the type is expected). This is how hive
achieves change types of columns ("schema on read"); in another word, the
similar performance penalty is already there for other SerDe in order to
achieve "schema on read".
/**
*Parses the string argument as if it was an int value and returns the
* result. Throws NumberFormatException if the string does not represent an
* int quantity.
...
public static int parseInt(byte[] bytes, int start, int length, int radix) {
/**
* Parses the string argument as if it was a long value and returns the
* result. Throws NumberFormatException if the string does not represent a
* long quantity.
public static long parseLong(byte[] bytes, int start, int length, int radix)
{
2) In he patch for this jira, the extra overhead is to list the top level
element of ArrayWritable to inspect whether there is a type change or not. A
new converted object is created ONLY IF the type is changed.
I agree that there would be some overhead to list the element of
ArrayWritable, but it is a tradeoff.
3) The patch in this jira would actually help performance when serializing (in
time of writing) ArrayWritable. The old approach is to create a new object for
every single writable element; with this patch, it only creates new a new
object when there is type change.
> parquet-hive should allow column type change
>
>
> Key: HIVE-6784
> URL: https://issues.apache.org/jira/browse/HIVE-6784
> Project: Hive
> Issue Type: Bug
> Components: File Formats, Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Tongjie Chen
> Fix For: 0.14.0
>
> Attachments: HIVE-6784.1.patch.txt, HIVE-6784.2.patch.txt
>
>
> see also in the following parquet issue:
> https://github.com/Parquet/parquet-mr/issues/323
> Currently, if we change parquet format hive table using "alter table
> parquet_table change c1 c1 bigint " ( assuming original type of c1 is int),
> it will result in exception thrown from SerDe:
> "org.apache.hadoop.io.IntWritable cannot be cast to
> org.apache.hadoop.io.LongWritable" in query runtime.
> This is different behavior from hive (using other file format), where it will
> try to perform cast (null value in case of incompatible type).
> Parquet Hive's RecordReader returns an ArrayWritable (based on schema stored
> in footers of parquet files); ParquetHiveSerDe also creates an corresponding
> ArrayWritableObjectInspector (but using column type info from metastore).
> Whenever there is column type change, the objector inspector will throw
> exception, since WritableLongObjectInspector cannot inspect an IntWritable
> etc...
> Conversion has to happen somewhere if we want to allow type change. SerDe's
> deserialize method seems a natural place for it.
> Currently, serialize method calls createStruct (then createPrimitive) for
> every record, but it creates a new object regardless, which seems expensive.
> I think that could be optimized a bit by just returning the object passed if
> already of the right type. deserialize also reuse this method, if there is a
> type change, there will be new object to be created, which I think is
> inevitable.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6784) parquet-hive should allow column type change
[ https://issues.apache.org/jira/browse/HIVE-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13965279#comment-13965279 ] Justin Coffey commented on HIVE-6784: - -1 on this patch. Looping on the arraywriteable in deserialize() will cause a performance penalty at read time, and running parseXxxx(obj.toString) in the event of a type mismatch is also painful. Changing types of columns is a rare event, we shouldn't write code that will cause performance penalties to handle it. Users should recreate the table with the new type and load it from the old table casting and converting as appropriate in their query. > parquet-hive should allow column type change > > > Key: HIVE-6784 > URL: https://issues.apache.org/jira/browse/HIVE-6784 > Project: Hive > Issue Type: Bug > Components: File Formats, Serializers/Deserializers >Affects Versions: 0.13.0 >Reporter: Tongjie Chen > Fix For: 0.14.0 > > Attachments: HIVE-6784.1.patch.txt, HIVE-6784.2.patch.txt > > > see also in the following parquet issue: > https://github.com/Parquet/parquet-mr/issues/323 > Currently, if we change parquet format hive table using "alter table > parquet_table change c1 c1 bigint " ( assuming original type of c1 is int), > it will result in exception thrown from SerDe: > "org.apache.hadoop.io.IntWritable cannot be cast to > org.apache.hadoop.io.LongWritable" in query runtime. > This is different behavior from hive (using other file format), where it will > try to perform cast (null value in case of incompatible type). > Parquet Hive's RecordReader returns an ArrayWritable (based on schema stored > in footers of parquet files); ParquetHiveSerDe also creates an corresponding > ArrayWritableObjectInspector (but using column type info from metastore). > Whenever there is column type change, the objector inspector will throw > exception, since WritableLongObjectInspector cannot inspect an IntWritable > etc... > Conversion has to happen somewhere if we want to allow type change. SerDe's > deserialize method seems a natural place for it. > Currently, serialize method calls createStruct (then createPrimitive) for > every record, but it creates a new object regardless, which seems expensive. > I think that could be optimized a bit by just returning the object passed if > already of the right type. deserialize also reuse this method, if there is a > type change, there will be new object to be created, which I think is > inevitable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6784) parquet-hive should allow column type change
[
https://issues.apache.org/jira/browse/HIVE-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13961358#comment-13961358
]
Hive QA commented on HIVE-6784:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12638877/HIVE-6784.2.patch.txt
{color:green}SUCCESS:{color} +1 5550 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2147/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2147/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12638877
> parquet-hive should allow column type change
>
>
> Key: HIVE-6784
> URL: https://issues.apache.org/jira/browse/HIVE-6784
> Project: Hive
> Issue Type: Bug
> Components: File Formats, Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Tongjie Chen
> Fix For: 0.14.0
>
> Attachments: HIVE-6784.1.patch.txt, HIVE-6784.2.patch.txt
>
>
> see also in the following parquet issue:
> https://github.com/Parquet/parquet-mr/issues/323
> Currently, if we change parquet format hive table using "alter table
> parquet_table change c1 c1 bigint " ( assuming original type of c1 is int),
> it will result in exception thrown from SerDe:
> "org.apache.hadoop.io.IntWritable cannot be cast to
> org.apache.hadoop.io.LongWritable" in query runtime.
> This is different behavior from hive (using other file format), where it will
> try to perform cast (null value in case of incompatible type).
> Parquet Hive's RecordReader returns an ArrayWritable (based on schema stored
> in footers of parquet files); ParquetHiveSerDe also creates an corresponding
> ArrayWritableObjectInspector (but using column type info from metastore).
> Whenever there is column type change, the objector inspector will throw
> exception, since WritableLongObjectInspector cannot inspect an IntWritable
> etc...
> Conversion has to happen somewhere if we want to allow type change. SerDe's
> deserialize method seems a natural place for it.
> Currently, serialize method calls createStruct (then createPrimitive) for
> every record, but it creates a new object regardless, which seems expensive.
> I think that could be optimized a bit by just returning the object passed if
> already of the right type. deserialize also reuse this method, if there is a
> type change, there will be new object to be created, which I think is
> inevitable.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6784) parquet-hive should allow column type change
[ https://issues.apache.org/jira/browse/HIVE-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13960421#comment-13960421 ] Tongjie Chen commented on HIVE-6784: https://reviews.apache.org/r/20048/ > parquet-hive should allow column type change > > > Key: HIVE-6784 > URL: https://issues.apache.org/jira/browse/HIVE-6784 > Project: Hive > Issue Type: Bug > Components: File Formats, Serializers/Deserializers >Affects Versions: 0.13.0 >Reporter: Tongjie Chen > Attachments: HIVE-6784.1.patch.txt > > > see also in the following parquet issue: > https://github.com/Parquet/parquet-mr/issues/323 > Currently, if we change parquet format hive table using "alter table > parquet_table change c1 c1 bigint " ( assuming original type of c1 is int), > it will result in exception thrown from SerDe: > "org.apache.hadoop.io.IntWritable cannot be cast to > org.apache.hadoop.io.LongWritable" in query runtime. > This is different behavior from hive (using other file format), where it will > try to perform cast (null value in case of incompatible type). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6784) parquet-hive should allow column type change
[ https://issues.apache.org/jira/browse/HIVE-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954290#comment-13954290 ] Brock Noland commented on HIVE-6784: FYI [~szehon] > parquet-hive should allow column type change > > > Key: HIVE-6784 > URL: https://issues.apache.org/jira/browse/HIVE-6784 > Project: Hive > Issue Type: Bug > Components: File Formats, Serializers/Deserializers >Affects Versions: 0.13.0 >Reporter: Tongjie Chen > > see also in the following parquet issue: > https://github.com/Parquet/parquet-mr/issues/323 > Currently, if we change parquet format hive table using "alter table > parquet_table change c1 c1 bigint " ( assuming original type of c1 is int), > it will result in exception thrown from SerDe: > "org.apache.hadoop.io.IntWritable cannot be cast to > org.apache.hadoop.io.LongWritable" in query runtime. > This is different behavior from hive (using other file format), where it will > try to perform cast (null value in case of incompatible type). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6784) parquet-hive should allow column type change
[ https://issues.apache.org/jira/browse/HIVE-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954289#comment-13954289 ] Brock Noland commented on HIVE-6784: FYI [~jcoffey] [~xuefuz] > parquet-hive should allow column type change > > > Key: HIVE-6784 > URL: https://issues.apache.org/jira/browse/HIVE-6784 > Project: Hive > Issue Type: Bug > Components: File Formats, Serializers/Deserializers >Affects Versions: 0.13.0 >Reporter: Tongjie Chen > > see also in the following parquet issue: > https://github.com/Parquet/parquet-mr/issues/323 > Currently, if we change parquet format hive table using "alter table > parquet_table change c1 c1 bigint " ( assuming original type of c1 is int), > it will result in exception thrown from SerDe: > "org.apache.hadoop.io.IntWritable cannot be cast to > org.apache.hadoop.io.LongWritable" in query runtime. > This is different behavior from hive (using other file format), where it will > try to perform cast (null value in case of incompatible type). -- This message was sent by Atlassian JIRA (v6.2#6252)
