taoran92 opened a new issue, #10651:
URL: https://github.com/apache/paimon/issues/10651
### Search before asking
- [x] I searched existing issues and found no report of this chain-specific
failure. #10478 describes a related failure involving the record iterator type
on regular primary-key tables; this issue involves the split type used for
chain reads.
### Paimon version
Paimon 2.2-SNAPSHOT.
The affected code is present on master at commit
702810b32b1b0c7189e31191e9234fca3d107c32.
### Compute Engine
Spark 3.5.9, Scala 2.12, Java 11.
### Minimal reproduce step
When a chain table query merges snapshot and delta data, selecting ordinary
columns succeeds. Selecting __paimon_bucket, or using it only in a filter,
fails.
Configure a Paimon catalog named spark_test with the Paimon Spark extensions
enabled:
```properties
spark.sql.catalog.spark_test=org.apache.paimon.spark.SparkCatalog
spark.sql.catalog.spark_test.warehouse=file:/tmp/paimon-chain-bucket-repro
spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions
```
Run the following SQL in a fresh test database:
```sql
CREATE DATABASE spark_test.chain_bucket_repro;
USE spark_test.chain_bucket_repro;
CREATE TABLE T (
id INT,
payload STRING,
seq BIGINT,
dt STRING
)
USING paimon
PARTITIONED BY (dt)
TBLPROPERTIES (
'primary-key' = 'dt,id',
'bucket-key' = 'id',
'bucket' = '1',
'merge-engine' = 'deduplicate',
'sequence.field' = 'seq',
'chain-table.enabled' = 'true',
'partition.timestamp-pattern' = '$dt',
'partition.timestamp-formatter' = 'yyyyMMdd'
);
CALL spark_test.sys.create_branch('chain_bucket_repro.T', 'snapshot');
CALL spark_test.sys.create_branch('chain_bucket_repro.T', 'delta');
ALTER TABLE T SET TBLPROPERTIES (
'scan.fallback-snapshot-branch' = 'snapshot',
'scan.fallback-delta-branch' = 'delta'
);
ALTER TABLE `T$branch_snapshot` SET TBLPROPERTIES (
'scan.fallback-snapshot-branch' = 'snapshot',
'scan.fallback-delta-branch' = 'delta'
);
ALTER TABLE `T$branch_delta` SET TBLPROPERTIES (
'scan.fallback-snapshot-branch' = 'snapshot',
'scan.fallback-delta-branch' = 'delta'
);
-- Two rows in the snapshot.
INSERT INTO `T$branch_snapshot`
VALUES
(1, 'old', 1, '20250810'),
(2, 'keep', 1, '20250810');
-- Update one row in the next day's delta.
INSERT INTO `T$branch_delta`
VALUES
(1, 'new', 2, '20250811');
-- Succeeds: returns (1, 'new') and (2, 'keep').
SELECT id, payload
FROM T
WHERE dt = '20250811';
-- Fails: bucket metadata in the projection.
SELECT id, __paimon_bucket
FROM T
WHERE dt = '20250811';
-- Also fails: bucket metadata only in the filter.
SELECT id, payload
FROM T
WHERE dt = '20250811'
AND __paimon_bucket = 0;
```
Run the two failing queries separately if the SQL client stops after an
error.
### What doesn't meet your expectations?
Both queries involving bucket metadata should succeed. The table has one
bucket, so all records belong to bucket 0.
Expected result for the metadata projection, ignoring row order:
```text
id __paimon_bucket
1 0
2 0
```
Expected result for the bucket filter:
```text
id payload
1 new
2 keep
```
Instead, both queries fail with:
```text
Caused by: java.lang.RuntimeException: There need be DataSplit when path and
index metadata columns are required
at
org.apache.paimon.spark.PaimonRecordReaderIterator.<init>(PaimonRecordReaderIterator.scala:45)
at
org.apache.paimon.spark.PaimonPartitionReader.readSplit(PaimonPartitionReader.scala:121)
at
org.apache.paimon.spark.PaimonPartitionReader.<init>(PaimonPartitionReader.scala:54)
at
org.apache.paimon.spark.PaimonPartitionReaderFactory.createReader(PaimonPartitionReaderFactory.scala:47)
at
org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.advanceToNextIter(DataSourceRDD.scala:125)
```
### Anything else?
The chain merge path produces ChainSplit, wrapped as
FallbackSplitImpl(ChainSplit) by the fallback scan. PaimonPartitionReader
passes that split to PaimonRecordReaderIterator.
When partition or bucket metadata is requested, the iterator requires the
split to be a DataSplit. Neither FallbackSplitImpl nor ChainSplit extends
DataSplit, so initialization fails before returning any rows.
```text
Snapshot DataSplits + Delta DataSplits
|
Group files by bucket
|
ChainSplit
|
FallbackSplitImpl(ChainSplit)
|
Spark metadata reader requires DataSplit
|
RuntimeException
```
Chain splits are constructed per bucket, so the merged rows still have a
well-defined bucket ID. The failure occurs in the Spark metadata adapter.
The exception message mentions “path and index”, but the failing check
actually applies to partition and bucket metadata. A possible fix is to inspect
the underlying split and support partition/bucket metadata from ChainSplit,
while preserving the original wrapper for reading and authorization.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
