wwj6591812 opened a new pull request, #8391:
URL: https://github.com/apache/paimon/pull/8391
### Purpose
When performing lookup join against a Paimon BLOB table (e.g., storing
images/videos), the full blob content is materialized via
`BlobSerializer.serialize() → blob.toData()` and written into local RocksDB
during bootstrap. For tables with large blob fields, this causes:
- Extremely high local disk usage (e.g., 2 billion images × 200KB = ~400TB
total, ~400GB per subtask)
- Page-size overflow errors (single records exceeding the 64KB default page)
- Prolonged bootstrap time leading to TaskManager heartbeat timeouts
This PR introduces a new table option `lookup.blob-as-descriptor` (default
`false`). When enabled:
1. BLOB fields are stored as their lightweight `BlobDescriptor` bytes (~130
bytes containing file URI, offset, and length) instead of the full blob content.
2. The RowType used for value serialization replaces BLOB with VARBINARY, so
`InternalSerializers.create()` produces `BinarySerializer` (calls
`getBinary()`) instead of `BlobSerializer` (calls `getBlob().toData()`).
3. `BlobRef.toDescriptor()` is a pure in-memory operation (no I/O), so
bootstrap performance is not affected.
The downstream consumer receives the serialized BlobDescriptor bytes and can
resolve the actual blob content on demand via a UDF reading from DFS.
**Example usage:**
```sql
CREATE TABLE dim_images (
url STRING,
image BLOB
) WITH (
'lookup.blob-as-descriptor' = 'true'
);
SELECT s.*, resolve_blob(d.image) AS image_data
FROM stream_table s
LEFT JOIN dim_images FOR SYSTEM_TIME AS OF s.proc_time AS d
ON s.url = d.url;
Impact: Per-subtask RocksDB storage drops from ~400GB to ~3.6GB (over 100x
reduction), making lookup join feasible for tables with large BLOB columns.
Tests
Existing unit tests pass (no behavioral change when the option is disabled).
TODO: Add integration test for NoPrimaryKeyLookupTable with
lookup.blob-as-descriptor = true to verify BlobDescriptor bytes are correctly
stored and returned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
