[
https://issues.apache.org/jira/browse/HIVE-29798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-29798:
--------------------------------
Description:
This was discovered while testing HIVE-28822, repro attached:
{code}
Caused by: java.lang.NumberFormatException: For input string: "NION"
at
java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
at java.base/java.lang.Long.parseLong(Long.java:709)
at java.base/java.lang.Long.parseLong(Long.java:832)
at
org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight.parse(AcidUtils.java:1081)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:355)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:175)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:170)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:2008)
at
org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:76)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:469)
... 28 more
{code}
this happens after "alter table tbl convert to acid" as well as altering table
properties to transactional=true
2 scenarios I can imagine:
1a) this is a bug, fix in writer path: alter table should take care of
flattening the union subdirs and converting the file structure to an ACID
compliant layout
1b) this is a bug, fix on reader path: considering [a unit
test|https://github.com/apache/hive/blob/9173850b9d8d51b253a0763115a6e5720b040640/ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java#L418-L437],
having the union subdirs in acid tables are totally fine, so no need to flatten
2. this is by design, and tables containing union subdirs must be flattened
BEFORE converting them to ACID
I'm about to handle this as 1b) as it makes to most sense to me
was:
This was discovered while testing HIVE-28822, repro attached:
{code}
Caused by: java.lang.NumberFormatException: For input string: "NION"
at
java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
at java.base/java.lang.Long.parseLong(Long.java:709)
at java.base/java.lang.Long.parseLong(Long.java:832)
at
org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight.parse(AcidUtils.java:1081)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:355)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:175)
at
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:170)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:2008)
at
org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:76)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:469)
... 28 more
{code}
this happens after "alter table tbl convert to acid" as well as altering table
properties to transactional=true
2 scenarios I can imagine:
1. this is a bug: alter table should take care of flattening the union subdirs
and converting the file structure to an ACID compliant layout
2. this is by design, and tables containing union subdirs must be flattened
BEFORE converting them to ACID
I'm about to handle this as 1), because the alter command should at least throw
an exception if the conversion would not lead to a layout which can be used
afterwards
> NumberFormatException after converting table with UNION subdirs to ACID
> -----------------------------------------------------------------------
>
> Key: HIVE-29798
> URL: https://issues.apache.org/jira/browse/HIVE-29798
> Project: Hive
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Attachments: TestUnionAllToAcidConversionReadFails.java
>
>
> This was discovered while testing HIVE-28822, repro attached:
> {code}
> Caused by: java.lang.NumberFormatException: For input string: "NION"
> at
> java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
> at java.base/java.lang.Long.parseLong(Long.java:709)
> at java.base/java.lang.Long.parseLong(Long.java:832)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight.parse(AcidUtils.java:1081)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:355)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:175)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:170)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:2008)
> at
> org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:76)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:469)
> ... 28 more
> {code}
> this happens after "alter table tbl convert to acid" as well as altering
> table properties to transactional=true
> 2 scenarios I can imagine:
> 1a) this is a bug, fix in writer path: alter table should take care of
> flattening the union subdirs and converting the file structure to an ACID
> compliant layout
> 1b) this is a bug, fix on reader path: considering [a unit
> test|https://github.com/apache/hive/blob/9173850b9d8d51b253a0763115a6e5720b040640/ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java#L418-L437],
> having the union subdirs in acid tables are totally fine, so no need to
> flatten
> 2. this is by design, and tables containing union subdirs must be flattened
> BEFORE converting them to ACID
> I'm about to handle this as 1b) as it makes to most sense to me
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
