lszskye commented on code in PR #257:
URL: https://github.com/apache/paimon-cpp/pull/257#discussion_r3893433347
##########
src/paimon/common/utils/arrow/arrow_utils.cpp:
##########
@@ -33,11 +35,61 @@
#include "paimon/common/utils/arrow/vector_utils.h"
#include "paimon/common/utils/checked_cast.h"
#include "paimon/common/utils/string_utils.h"
+#include "paimon/core/casting/casting_utils.h"
namespace paimon {
namespace {
+// Whether `type` is a dictionary this can carry across the C data interface
unchanged. The index
+// width is part of the test because nothing in a layout reveals it; see
+// ArrowUtils::IsDictionaryLayoutRecoverableValueType().
+bool IsResolvableDictionary(const arrow::DataType& type) {
Review Comment:
Some format(AliORC), will lazy decoding exposes dictionary-encoded STRING
columns as dictionary<int32, utf8>. This type is considered resolvable by
IsResolvableDictionary(), so FlattenUnresolvableDictionaries() keeps it encoded
during append-only compaction.
However, the non-Parquet veto only disables
parquet.read.enable-dictionary-passthrough; it does not disable
aliorc.read.enable-lazy-decoding. As a result, an AliORC dictionary batch can
still reach OrcFormatWriter::AddBatch(), where it is imported using the logical
utf8 schema. The actual ArrowArray child is dictionary-encoded, so the import
will fail. Even if the dictionary type were recovered during import,
OrcAdapter::WriteBatch() does not support arrow::Type::DICTIONARY.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
