[jira] [Updated] (HIVE-15739) Incorrect exception message in PartExprEvalUtils
[
https://issues.apache.org/jira/browse/HIVE-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Wagner updated HIVE-15739:
---
Affects Version/s: 2.2.0
> Incorrect exception message in PartExprEvalUtils
>
>
> Key: HIVE-15739
> URL: https://issues.apache.org/jira/browse/HIVE-15739
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Mark Wagner
>Assignee: Mark Wagner
>Priority: Minor
> Attachments: HIVE-15739.1.patch.txt
>
>
> The check is on partSpec, not partProps:
> {noformat}
> if (partSpec.size() != partKeyTypes.length) {
> throw new HiveException("Internal error : Partition Spec size, " +
> partProps.size() +
> " doesn't match partition key definition size, " +
> partKeyTypes.length);
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15739) Incorrect exception message in PartExprEvalUtils
[
https://issues.apache.org/jira/browse/HIVE-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840500#comment-15840500
]
Mark Wagner commented on HIVE-15739:
Attached patch for master.
> Incorrect exception message in PartExprEvalUtils
>
>
> Key: HIVE-15739
> URL: https://issues.apache.org/jira/browse/HIVE-15739
> Project: Hive
> Issue Type: Bug
>Reporter: Mark Wagner
>Assignee: Mark Wagner
>Priority: Minor
> Attachments: HIVE-15739.1.patch.txt
>
>
> The check is on partSpec, not partProps:
> {noformat}
> if (partSpec.size() != partKeyTypes.length) {
> throw new HiveException("Internal error : Partition Spec size, " +
> partProps.size() +
> " doesn't match partition key definition size, " +
> partKeyTypes.length);
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15739) Incorrect exception message in PartExprEvalUtils
[
https://issues.apache.org/jira/browse/HIVE-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Wagner updated HIVE-15739:
---
Attachment: HIVE-15739.1.patch.txt
> Incorrect exception message in PartExprEvalUtils
>
>
> Key: HIVE-15739
> URL: https://issues.apache.org/jira/browse/HIVE-15739
> Project: Hive
> Issue Type: Bug
>Reporter: Mark Wagner
>Assignee: Mark Wagner
>Priority: Minor
> Attachments: HIVE-15739.1.patch.txt
>
>
> The check is on partSpec, not partProps:
> {noformat}
> if (partSpec.size() != partKeyTypes.length) {
> throw new HiveException("Internal error : Partition Spec size, " +
> partProps.size() +
> " doesn't match partition key definition size, " +
> partKeyTypes.length);
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15739) Incorrect exception message in PartExprEvalUtils
[
https://issues.apache.org/jira/browse/HIVE-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Wagner updated HIVE-15739:
---
Status: Patch Available (was: Open)
> Incorrect exception message in PartExprEvalUtils
>
>
> Key: HIVE-15739
> URL: https://issues.apache.org/jira/browse/HIVE-15739
> Project: Hive
> Issue Type: Bug
>Reporter: Mark Wagner
>Assignee: Mark Wagner
>Priority: Minor
> Attachments: HIVE-15739.1.patch.txt
>
>
> The check is on partSpec, not partProps:
> {noformat}
> if (partSpec.size() != partKeyTypes.length) {
> throw new HiveException("Internal error : Partition Spec size, " +
> partProps.size() +
> " doesn't match partition key definition size, " +
> partKeyTypes.length);
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14509) AvroSerde mutates tinyint and smallint columns when specifying native columns
[
https://issues.apache.org/jira/browse/HIVE-14509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Wagner updated HIVE-14509:
---
Attachment: avro-tinyint.demo.patch
Here's a patch for a q test which demonstrates the problem. The flow goes like
this:
* AvroSerde is initialized with the right columns
* Avro doesn't have a tinyint type, so it is stored as int. Internally, the
AvroSerde translates the column types to an Avro schema, then generates an OI
from that. This OI then has a IntegerColumnInspector in it instead of a
TinyintColumnInspector.
* Table.getColsInternal gets to this section:
{code}
try {
// Do the lightweight check for general case.
if (hasMetastoreBasedSchema(SessionState.getSessionConf(),
serializationLib)) {
return tTable.getSd().getCols();
} else if (forMs && !shouldStoreFieldsInMetastore(
SessionState.getSessionConf(), serializationLib,
tTable.getParameters())) {
return Hive.getFieldsFromDeserializerForMsStorage(this,
getDeserializer());
} else {
return MetaStoreUtils.getFieldsFromDeserializer(getTableName(),
getDeserializer());
}
{code}
which dutifully sets the columns according to the OI returned by the Serde.
> AvroSerde mutates tinyint and smallint columns when specifying native columns
> -
>
> Key: HIVE-14509
> URL: https://issues.apache.org/jira/browse/HIVE-14509
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 2.2.0
>Reporter: Mark Wagner
> Attachments: avro-tinyint.demo.patch
>
>
> tinyint and smallint go in, int comes out:
> {noformat}
> string1 string
> int1int
> tinyint1int
> smallint1 int
> bigint1 bigint
> boolean1boolean
> float1 float
> double1 double
> list1 array
> map1map
> struct1 struct
>
> enum1 string
> nullableint int
> bytes1 binary
> fixed1 binary
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-14406) ORC should be supported in mixed file format tables
[ https://issues.apache.org/jira/browse/HIVE-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Wagner reassigned HIVE-14406: -- Assignee: Mark Wagner > ORC should be supported in mixed file format tables > --- > > Key: HIVE-14406 > URL: https://issues.apache.org/jira/browse/HIVE-14406 > Project: Hive > Issue Type: Bug > Components: ORC >Reporter: Mark Wagner >Assignee: Mark Wagner > > Hive supports tables with partition-wise file formats and serdes (see > partition_wise_fileformat*.q tests for example usage). The ORC file > format/serde combination is explicitly prevented from being used in mixed > format tables. This was added in HIVE-12728. To have parity with the other > formats, ORC should be supported in mixed format tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10187) Avro backed tables don't handle cyclical or recursive records
[ https://issues.apache.org/jira/browse/HIVE-10187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142238#comment-15142238 ] Mark Wagner commented on HIVE-10187: Those tests failures are unrelated and all the other precommit builds have been failing on the same tests for as long back as I can see. > Avro backed tables don't handle cyclical or recursive records > - > > Key: HIVE-10187 > URL: https://issues.apache.org/jira/browse/HIVE-10187 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 1.2.0 >Reporter: Mark Wagner >Assignee: Mark Wagner > Attachments: HIVE-10187.1.patch, HIVE-10187.2.patch, > HIVE-10187.3.patch, HIVE-10187.4.patch, HIVE-10187.5.patch, > HIVE-10187.demo.patch > > > [HIVE-7653] changed the Avro SerDe to make it generate TypeInfos even for > recursive/cyclical schemas. However, any attempt to serialize data which > exploits that ability results in silently dropped fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10187) Avro backed tables don't handle cyclical or recursive records
[ https://issues.apache.org/jira/browse/HIVE-10187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Wagner updated HIVE-10187: --- Attachment: HIVE-10187.5.patch > Avro backed tables don't handle cyclical or recursive records > - > > Key: HIVE-10187 > URL: https://issues.apache.org/jira/browse/HIVE-10187 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 1.2.0 >Reporter: Mark Wagner >Assignee: Mark Wagner > Attachments: HIVE-10187.1.patch, HIVE-10187.2.patch, > HIVE-10187.3.patch, HIVE-10187.4.patch, HIVE-10187.5.patch, > HIVE-10187.demo.patch > > > [HIVE-7653] changed the Avro SerDe to make it generate TypeInfos even for > recursive/cyclical schemas. However, any attempt to serialize data which > exploits that ability results in silently dropped fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10187) Avro backed tables don't handle cyclical or recursive records
[ https://issues.apache.org/jira/browse/HIVE-10187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138322#comment-15138322 ] Mark Wagner commented on HIVE-10187: New patch with Anthony's comments addressed. > Avro backed tables don't handle cyclical or recursive records > - > > Key: HIVE-10187 > URL: https://issues.apache.org/jira/browse/HIVE-10187 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 1.2.0 >Reporter: Mark Wagner >Assignee: Mark Wagner > Attachments: HIVE-10187.1.patch, HIVE-10187.2.patch, > HIVE-10187.3.patch, HIVE-10187.4.patch, HIVE-10187.5.patch, > HIVE-10187.demo.patch > > > [HIVE-7653] changed the Avro SerDe to make it generate TypeInfos even for > recursive/cyclical schemas. However, any attempt to serialize data which > exploits that ability results in silently dropped fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10187) Avro backed tables don't handle cyclical or recursive records
[ https://issues.apache.org/jira/browse/HIVE-10187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Wagner updated HIVE-10187: --- Attachment: HIVE-10187.4.patch I've rebased on master. > Avro backed tables don't handle cyclical or recursive records > - > > Key: HIVE-10187 > URL: https://issues.apache.org/jira/browse/HIVE-10187 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 1.2.0 >Reporter: Mark Wagner >Assignee: Mark Wagner > Attachments: HIVE-10187.1.patch, HIVE-10187.2.patch, > HIVE-10187.3.patch, HIVE-10187.4.patch, HIVE-10187.demo.patch > > > [HIVE-7653] changed the Avro SerDe to make it generate TypeInfos even for > recursive/cyclical schemas. However, any attempt to serialize data which > exploits that ability results in silently dropped fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10187) Avro backed tables don't handle cyclical or recursive records
[ https://issues.apache.org/jira/browse/HIVE-10187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Wagner updated HIVE-10187: --- Attachment: HIVE-10187.3.patch Rebased and re-uploading in an attempt to get the QA tests to run. Avro backed tables don't handle cyclical or recursive records - Key: HIVE-10187 URL: https://issues.apache.org/jira/browse/HIVE-10187 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 1.2.0 Reporter: Mark Wagner Assignee: Mark Wagner Attachments: HIVE-10187.1.patch, HIVE-10187.2.patch, HIVE-10187.3.patch, HIVE-10187.demo.patch [HIVE-7653] changed the Avro SerDe to make it generate TypeInfos even for recursive/cyclical schemas. However, any attempt to serialize data which exploits that ability results in silently dropped fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10187) Avro backed tables don't handle cyclical or recursive records
[ https://issues.apache.org/jira/browse/HIVE-10187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482195#comment-14482195 ] Mark Wagner commented on HIVE-10187: The test failure seems to be unrelated. Avro backed tables don't handle cyclical or recursive records - Key: HIVE-10187 URL: https://issues.apache.org/jira/browse/HIVE-10187 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 1.2.0 Reporter: Mark Wagner Assignee: Mark Wagner Attachments: HIVE-10187.1.patch, HIVE-10187.demo.patch [HIVE-7653] changed the Avro SerDe to make it generate TypeInfos even for recursive/cyclical schemas. However, any attempt to serialize data which exploits that ability results in silently dropped fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-10187) Avro backed tables don't handle cyclical or recursive records
[ https://issues.apache.org/jira/browse/HIVE-10187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Wagner reassigned HIVE-10187: -- Assignee: Mark Wagner Avro backed tables don't handle cyclical or recursive records - Key: HIVE-10187 URL: https://issues.apache.org/jira/browse/HIVE-10187 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 1.2.0 Reporter: Mark Wagner Assignee: Mark Wagner Attachments: HIVE-10187.demo.patch [HIVE-7653] changed the Avro SerDe to make it generate TypeInfos even for recursive/cyclical schemas. However, any attempt to serialize data which exploits that ability results in silently dropped fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10187) Avro backed tables don't handle cyclical or recursive records
[ https://issues.apache.org/jira/browse/HIVE-10187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Wagner updated HIVE-10187: --- Attachment: HIVE-10187.demo.patch Here's a patch which extends the testcase added in HIVE-7653 to illustrate the issue. The problem here is that the generated TypeInfo for these data sets replaces the recursive fields with void, so any serializer will faithfully serialize nulls instead of any relevant data. However, making the TypeInfo properly recursive is likely to not work with other components of Hive. At the very least, the 'describe' statement will cause an infinite loop. I believe these issues could be fixed, but it would require making cyclical/recursive datasets a first class citizen in Hive's data model. Avro backed tables don't handle cyclical or recursive records - Key: HIVE-10187 URL: https://issues.apache.org/jira/browse/HIVE-10187 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 1.2.0 Reporter: Mark Wagner Attachments: HIVE-10187.demo.patch [HIVE-7653] changed the Avro SerDe to make it generate TypeInfos even for recursive/cyclical schemas. However, any attempt to serialize data which exploits that ability results in silently dropped fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9851) org.apache.hadoop.hive.serde2.avro.AvroSerializer should use org.apache.avro.generic.GenericData.Array when serializing a list
[ https://issues.apache.org/jira/browse/HIVE-9851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351478#comment-14351478 ] Mark Wagner commented on HIVE-9851: --- Sorry, I just realized that this is limited to the serialization side, so maybe my previous comment isn't as applicable. org.apache.hadoop.hive.serde2.avro.AvroSerializer should use org.apache.avro.generic.GenericData.Array when serializing a list -- Key: HIVE-9851 URL: https://issues.apache.org/jira/browse/HIVE-9851 Project: Hive Issue Type: Bug Components: Hive, Serializers/Deserializers Reporter: Ratandeep Ratti Attachments: HIVE-9851.patch Currently AvroSerializer uses java.util.ArrayList for serializing a list in Hive. This causes problems when we need to convert the avro object into some other representation say a tuple in Pig. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9851) org.apache.hadoop.hive.serde2.avro.AvroSerializer should use org.apache.avro.generic.GenericData.Array when serializing a list
[ https://issues.apache.org/jira/browse/HIVE-9851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351477#comment-14351477 ] Mark Wagner commented on HIVE-9851: --- [~rdsr], I have a rebased and updated version of HIVE-4734 which I was preparing to post. If this is going to also be an issue for Records, Byte arrays, fixeds, enums, etc. then we may just want to wait for that, which will pass the full Avro record all the way through the serializer. I should have some time to finish testing and post that next week. org.apache.hadoop.hive.serde2.avro.AvroSerializer should use org.apache.avro.generic.GenericData.Array when serializing a list -- Key: HIVE-9851 URL: https://issues.apache.org/jira/browse/HIVE-9851 Project: Hive Issue Type: Bug Components: Hive, Serializers/Deserializers Reporter: Ratandeep Ratti Attachments: HIVE-9851.patch Currently AvroSerializer uses java.util.ArrayList for serializing a list in Hive. This causes problems when we need to convert the avro object into some other representation say a tuple in Pig. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
