zhoulii opened a new issue, #8884: URL: https://github.com/apache/paimon/issues/8884
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version 1.0 ### Compute Engine flink ### Minimal reproduce step 1. Create a primary-key table with: lookup.cache = auto merge-engine = deduplicate sequence.field = <field> 2. Perform a lookup join where the join keys equal the table's primary keys. 3. Partial lookup does not support deduplicate with sequence.field, so AUTO falls back to FullCacheLookupTable. 4. Trigger a compaction and wait for the lookup cache to refresh. ### What doesn't meet your expectations? After falling back to full cache, the table still has lookup.cache=AUTO. Therefore, LookupFileStoreTable.lookupStreamScanMode() incorrectly returns FILE_MONITOR instead of using the scan mode for full cache. FILE_MONITOR treats compaction file replacements as file deletions and additions. As a result, full-cache refresh may process all rows in the compacted files as DELETE and INSERT operations, even when there are very few logical changes. For large tables, this causes excessive CPU and I/O usage in every lookup subtask. With synchronous refresh, it can block the lookup operator, create sustained backpressure, and stall the Flink pipeline. After AUTO falls back to full cache, the effective lookup.cache should be changed to FULL, so the normal full-cache scan-mode selection is used. ### Anything else? _No response_ ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
