wangzhigang1999 opened a new issue, #9819:
URL: https://github.com/apache/paimon/issues/9819
### Search before asking
- [x] I searched in the issues and found nothing similar.
### Paimon version
Apache Paimon master, commit `c081ccd0d7003ceae2517c6b592a947e824acc25`.
### Compute Engine
PyPaimon, Python 3.12.6, PyArrow 19.0.1.
### Minimal reproduce step
A primary-key merge read fails when an input reader returns an empty batch
after a non-empty batch. The following example uses the reader interfaces to
reproduce the batch transition without external storage. Run it with
`PYTHONPATH` pointing to `paimon-python`:
```python
from pypaimon.read.reader.iface.record_iterator import RecordIterator
from pypaimon.read.reader.iface.record_reader import RecordReader
from pypaimon.read.reader.sort_merge_reader import SortMergeReaderWithMinHeap
from pypaimon.schema.data_types import AtomicType, DataField
from pypaimon.schema.table_schema import TableSchema
from pypaimon.table.row.key_value import KeyValue
from pypaimon.table.row.row_kind import RowKind
class BatchIterator(RecordIterator):
def __init__(self, rows):
self.rows = iter(rows)
def next(self):
return next(self.rows, None)
class BatchReader(RecordReader):
def __init__(self, batches):
self.batches = iter(batches)
def read_batch(self):
batch = next(self.batches, None)
return None if batch is None else BatchIterator(batch)
def close(self):
pass
def kv(key):
return KeyValue(1, 1).replace((key, 1, RowKind.INSERT.value, key))
schema = TableSchema(
fields=[DataField(0, "id", AtomicType("INT"))], primary_keys=["id"])
reader = SortMergeReaderWithMinHeap(
[BatchReader([[kv(1)], [], [], [kv(2)], []])], schema)
try:
result = []
while True:
batch = reader.read_batch()
if batch is None:
break
while True:
record = batch.next()
if record is None:
break
result.append(record.key.get_field(0))
assert result == [1, 2], result
print(result)
finally:
reader.close()
```
### What doesn't meet your expectations?
Expected: read both records and print `[1, 2]`. An empty batch should not
end the reader; only `read_batch()` returning `None` indicates EOF.
Actual: the second call to the merge iterator raises:
```text
File "pypaimon/read/reader/sort_merge_reader.py", line 127, in _next_impl
entry = HeapEntry(element.kv.key, element, self.key_comparator,
AttributeError: 'NoneType' object has no attribute 'key'
```
### Anything else?
In `Element.update()`, after exhausting the current batch, we read one more
batch and assign its first record to `self.kv`. If that batch is empty, we
assign `None` but return `True`. The caller then accesses `element.kv.key`.
I also reproduced this through table writes, commits, and a filtered read
using local files: a full scan returns 6,144 rows, while the filtered scan
fails with the same exception. With the fix, the filtered scan returns the
expected four rows and values.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
