[
https://issues.apache.org/jira/browse/HIVE-29798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-29798:
--------------------------------
Summary: Fix union directory and file layout to ACID conversion issues
(was: NumberFormatException while reading a table having UNION subdirs after
ACID conversion)
> Fix union directory and file layout to ACID conversion issues
> -------------------------------------------------------------
>
> Key: HIVE-29798
> URL: https://issues.apache.org/jira/browse/HIVE-29798
> Project: Hive
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Labels: pull-request-available
> Attachments: TestUnionAllToAcidConversionReadFails.java
>
>
> This was discovered while testing HIVE-28822, repro attached:
> {code}
> Caused by: java.lang.NumberFormatException: For input string: "NION"
> at
> java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
> at java.base/java.lang.Long.parseLong(Long.java:709)
> at java.base/java.lang.Long.parseLong(Long.java:832)
> at
> org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight.parse(AcidUtils.java:1081)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:355)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:175)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.<init>(VectorizedOrcAcidRowBatchReader.java:170)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:2008)
> at
> org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:76)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:469)
> ... 28 more
> {code}
> this happens after "alter table tbl convert to acid" as well as altering
> table properties to transactional=true
> 2 scenarios I can imagine:
> 1a) this is a bug, fix in writer path: alter table should take care of
> flattening the union subdirs and converting the file structure to an ACID
> compliant layout
> 1b) this is a bug, fix on reader path: considering [a unit
> test|https://github.com/apache/hive/blob/9173850b9d8d51b253a0763115a6e5720b040640/ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java#L418-L437],
> having the union subdirs in acid tables is totally fine, so no need to
> flatten
> 2. this is by design, and tables containing union subdirs must be flattened
> BEFORE converting them to ACID
> I'm about to handle this as 1b) as it makes to most sense to me
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
