SteNicholas commented on code in PR #392:
URL: https://github.com/apache/paimon-cpp/pull/392#discussion_r4131833247
##########
docs/source/user_guide/write.rst:
##########
@@ -71,6 +71,71 @@ RecordBatch Construction
- Prefer batch sizes tuned for I/O throughput (e.g., tens to hundreds of MB
per flush, depending on filesystem and cluster configuration).
- Maintain stable sort orders within a batch only if required by downstream
merge or compaction logic; otherwise avoid unnecessary ordering costs.
+Writing BLOB Columns
+~~~~~~~~~~~~~~~~~~~~
+
+A ``BLOB`` column is a ``LargeBinary`` field carrying Paimon's BLOB field
+metadata, and an ``ARRAY<BLOB>`` column is a top-level ``List`` field whose
+element field carries it. Build the BLOB field with
``paimon::Blob::ArrowField``
+and import it into Arrow; the element field of an ``ARRAY<BLOB>`` column must
+keep that metadata:
+
+.. code-block:: cpp
+
+ PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<::ArrowSchema> c_element,
+ paimon::Blob::ArrowField("element",
/*nullable=*/true));
+ arrow::Result<std::shared_ptr<arrow::Field>> element =
arrow::ImportField(c_element.get());
+ if (!element.ok()) {
+ return paimon::Status::Invalid(element.status().ToString());
+ }
Review Comment:
Good catch. `arrow::ImportField` returns an `arrow::Result`, so the example
now uses `PAIMON_ASSIGN_OR_RAISE_FROM_ARROW` for it and keeps
`PAIMON_ASSIGN_OR_RAISE` for `paimon::Blob::ArrowField`.
##########
src/paimon/core/append/append_compact_coordinator.cpp:
##########
@@ -203,6 +203,15 @@ Status ValidateTable(const std::shared_ptr<TableSchema>&
table_schema,
const std::shared_ptr<arrow::Schema>& arrow_schema,
const CoreOptions& core_options) {
PAIMON_RETURN_NOT_OK(BlobUtils::ValidateContainerBlobWriteSchema(arrow_schema));
+ // The rewrite reads and writes plain append files, which can neither
merge data-evolution
+ // blob layers nor write blob files; Java compacts these through its
data-evolution
+ // compaction instead.
+ for (const auto& field : arrow_schema->fields()) {
+ if (BlobUtils::IsArrayBlobField(field)) {
+ return Status::NotImplemented(
+ "Compacting a table with ARRAY<BLOB> is not supported by the
C++ writer.");
+ }
+ }
Review Comment:
Done. `ValidateTable` now rejects every table with `data-evolution.enabled`,
which covers all tables with `BLOB`, `ARRAY<BLOB>` or `MAP<..., BLOB>` columns
since they require data evolution, and the container blob special cases are
removed. `TestValidateFailsOnDataEvolutionTable` covers a data-evolution table
with only plain columns, and `TestValidateFailsOnArrayBlobTable` one with an
`ARRAY<BLOB>` column.
##########
src/paimon/common/data/blob_defs.h:
##########
@@ -86,13 +87,15 @@ class BlobDefs {
/// Only the data-evolution blob fallback read path sets this.
static constexpr char kEmitPlaceholderSentinelKey[] =
"blob.internal.emit-placeholder-sentinel";
/// Internal (non user-facing) format option, "false" by default: when
"true", the blob
- /// format writer persists a value exactly equal to kPlaceholderSentinel
as a bin_length -2
- /// entry. Only set for data-evolution partial updates, i.e. blob-only
column writes of a
- /// table with data evolution enabled; all other writes store bytes
verbatim.
+ /// format writer persists a value exactly equal to kPlaceholderSentinel
(or an ARRAY<BLOB>
+ /// whose only element is kPlaceholderSentinel) as a bin_length -2 entry.
Only set for
+ /// data-evolution partial updates, i.e. blob-only column writes of a
table with data evolution
+ /// enabled; no other write interprets a value as a placeholder.
static constexpr char kWritePlaceholderKey[] =
"blob.internal.write-placeholder";
Review Comment:
Fixed. As Java does for its placeholder objects, the writer now recognizes
the sentinel in every write, whatever other columns the write carries: a `BLOB`
value equal to `_PAIMON_BLOB_PLACEHOLDER`, or an `ARRAY<BLOB>` holding only it,
is persisted as a `-2` entry, and the internal
`blob.internal.write-placeholder` option is removed. The new integration test
`BlobTableInteTest.TestDataEvolutionPartialUpdateWithNonBlobColumnFallsBack`
writes an `[f0, b0, a0]` partial update with placeholders and checks that both
the `BLOB` and the `ARRAY<BLOB>` column fall back to the older values. Since
every write now interprets the sentinel, a user value equal to it is persisted
as a placeholder as well; the existing sentinel tests and `write.rst` are
updated accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
