Jonathan Vexler created HUDI-9683:
-------------------------------------
Summary: Fix spark rename columns for
HoodieInternalRowUtils.genUnsafeRowWriter
Key: HUDI-9683
URL: https://issues.apache.org/jira/browse/HUDI-9683
Project: Apache Hudi
Issue Type: Bug
Components: spark
Reporter: Jonathan Vexler
Assignee: Jonathan Vexler
Fix For: 1.1.0
{code:java}
val (fieldWriter, prevFieldPos): (RowFieldUpdater, Int) =
prevStructType.getFieldIndex(newField.name) match {
case Some(prevFieldPos) =>
val prevField = prevStructType(prevFieldPos)
(newWriterRenaming(prevField.dataType, newField.dataType,
renamedColumnsMap, fieldNamesStack), prevFieldPos)
case None =>
val newFieldQualifiedName = createFullName(fieldNamesStack)
val prevFieldName: String = lookupRenamedField(newFieldQualifiedName,
renamedColumnsMap)
// Handle rename
prevStructType.getFieldIndex(prevFieldName) match {
case Some(prevFieldPos) =>
val prevField = prevStructType.fields(prevFieldPos)
(newWriterRenaming(prevField.dataType, newField.dataType,
renamedColumnsMap, fieldNamesStack), prevFieldPos)
case None =>
val updater: RowFieldUpdater = (fieldUpdater, ordinal, _) =>
fieldUpdater.setNullAt(ordinal)
(updater, -1)
}
} {code}
We only look up the rename columns if the column is not found. If a new column
has been renamed to the name of an old column then we will not respect the
rename
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
