lxy-9602 commented on code in PR #198:
URL: https://github.com/apache/paimon-cpp/pull/198#discussion_r3788466836
##########
src/paimon/format/parquet/parquet_file_batch_reader.cpp:
##########
@@ -762,7 +783,7 @@ Status ParquetFileBatchReader::CollectLeafIndices(const
std::shared_ptr<arrow::D
void ParquetFileBatchReader::SkipLeafIndices(const
std::shared_ptr<arrow::DataType>& file_type,
int32_t* leaf_index) {
if (file_type->id() == arrow::Type::STRUCT || file_type->id() ==
arrow::Type::LIST ||
- file_type->id() == arrow::Type::MAP) {
Review Comment:
For write side, I noticed that the current Paimon library is using Arrow 17,
which does not handle `fixed size list -> list` conversion very well in the
presence of nulls. For example, cases like `[valid vec, null, valid vec]` may
fail.
If that’s the case, then the write path may also need to reuse the current
`ParquetVectorConverter::ConvertToWriteType` logic.
Since this issue has already been fixed in newer Arrow versions, I’d suggest
keeping the write-side handling as isolated as possible within the Parquet
format layer, and adding a TODO so that the extra conversion can be removed
directly once we upgrade Arrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
