lszskye commented on code in PR #257:
URL: https://github.com/apache/paimon-cpp/pull/257#discussion_r3893531047
##########
src/paimon/common/utils/arrow/arrow_utils.cpp:
##########
@@ -500,4 +552,112 @@ Result<arrow::Compression::type>
ArrowUtils::GetCompressionType(const std::strin
return compression_type;
}
+// `is_binary_like()` is BINARY and STRING and nothing else. `LARGE_STRING` is
left out even though
+// it is binary-like: the ORC reader widens strings to `dictionary(int64(),
large_utf8())` under
+// lazy decoding, and a layout reports neither index nor offset width, so
reading that back as
+// `int32` indices over `int32` offsets would silently reinterpret both
buffers instead of failing.
+//
+// This narrows what may be carried; it cannot verify what was. See
+// ResolveParquetDictionaryStructType() for where the index width becomes a
caller contract.
+bool ArrowUtils::IsDictionaryLayoutRecoverableValueType(const arrow::DataType&
type) {
+ return arrow::is_binary_like(type.id());
+}
+
+// Why the header calls the `int32` index width a contract rather than a
check: the value-type
+// check rejects `dictionary(int64(), large_utf8())`, the shape the ORC reader
produces, but
+// nothing here can tell `dictionary(int32(), utf8())` apart from
`dictionary(int64(), utf8())`,
+// and the second would be read as the first.
+//
+// So the contract binds the producer, not the callers:
`ParquetFormatWriter::ResolveBatchSchema`
+// and `DataFileWriterBase::AddFileIndexBatch` see only the layout.
+// `AppendOnlyFileStoreWrite::CompactRewrite` is today's only production path
that can hand over a
+// batch whose dictionaries the schema does not declare, and it honours the
contract by running
+// FlattenUnresolvableDictionaries() first. Closing the hole instead of
narrowing it needs the real
+// `ArrowSchema` to reach the writer, which
`FormatWriter::AddBatch(ArrowArray*)` drops.
+Result<std::shared_ptr<arrow::DataType>>
ArrowUtils::ResolveParquetDictionaryStructType(
Review Comment:
See one function about parquet in framework is a little bit weird
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
