hudi-agent commented on code in PR #19283:
URL: https://github.com/apache/hudi/pull/19283#discussion_r3575960044
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -144,10 +144,9 @@ class
SparkFileFormatInternalRowReaderContext(baseFileReader: SparkColumnarFileR
// Parquet stores VECTOR as FIXED_LEN_BYTE_ARRAY, so the reader needs
BinaryType
// and we decode back to ArrayType below. Lance returns ArrayType
natively, so skip
// the rewrite only for Lance base files; log files always go through the
rewrite path.
- val isLogFile = FSUtils.isLogFile(filePath)
- val isLanceBaseFile = !isLogFile && FSUtils.isBaseFile(filePath) &&
- tableConfig.getBaseFileFormat == HoodieFileFormat.LANCE
- val vectorColumnInfo: Map[Int, HoodieSchema.Vector] = if (isLanceBaseFile)
{
+ val isInlineLog = FSUtils.isInlineLogFile(filePath.getName)
+ val isLanceFile = !isInlineLog && tableConfig.getBaseFileFormat ==
HoodieFileFormat.LANCE
Review Comment:
🤖 Now that native Lance logs are enabled, `isLanceFile` keys off
`tableConfig.getBaseFileFormat`, while the `fileFormat` resolution just below
(line 174) keys off the file extension. In a multi-format table whose default
isn't LANCE, a native Lance log would then be read with a Parquet-rewritten
(BinaryType) vector schema but dispatched to the Lance reader by extension — a
schema mismatch on the vector columns. Since the removed
`testShouldWriteInlineLogFormatForMultiFormatLanceWrites` used to keep exactly
that case on inline logs, is multi-format Lance in scope here, or would keying
`isLanceFile` off `filePath.getFileExtension` (like the format resolution) be
safer? (The comment on lines 145-146 that logs 'always go through the rewrite
path' also reads stale now.)
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
