Akash3121 commented on code in PR #11144:
URL: https://github.com/apache/paimon/pull/11144#discussion_r4570407006
##########
docs/docs/pypaimon/writing.md:
##########
@@ -90,9 +90,15 @@ the same method. Changing a builder affects only operations
created afterward.
### Native Write and Commit
-With the `pypaimon-rust` extension installed, set `write.native.enabled=true`
-to use Rust for Parquet/BLOB writes and `commit.native.enabled=true` to use
-Rust for REST snapshot publication. Batch and stream writers retain the same
+On Python 3.10+, `pypaimon-rust>=0.5.0` is installed by default.
+`write.native.enabled` and `commit.native.enabled` both default to `true`,
+selecting Rust for supported Parquet/BLOB writes and REST snapshot publication.
+Set either option to `false` to select its Python execution path.
Review Comment:
With `write.native.enabled`, `read.native.enabled`, and
`scan.native-plan.enabled` all defaulting to `True` now, setting
`write.native.enabled=false` alone does not select the Python execution path in
`create_row_id_plan_execution()`
(`paimon-python/pypaimon/write/backend_dispatch.py:351-360`):
```python
def create_row_id_plan_execution(table, plan):
"""Select pure routing once; execution errors never replay another
backend."""
if (table.options.native_write_enabled() or
table.options.native_read_enabled()
or table.options.native_plan_enabled()):
from pypaimon.execution.native.write.row_id_plan import
create_native_row_id_plan, NativeRowIdPlan
native = create_native_row_id_plan(table, plan)
if native is not None:
return NativeRowIdPlan(table, plan, native=native)
from pypaimon.execution.python.write.row_id_plan import PythonRowIdPlan
return PythonRowIdPlan(plan)
```
Previously, when all three options defaulted to `False`,
`native_write_enabled() or native_read_enabled() or native_plan_enabled()`
worked as an opt-in check. Now that all three default to `True`, setting
`write.native.enabled=false` leaves `native_read_enabled()` and
`native_plan_enabled()` as `True`, so `create_row_id_plan_execution()` still
routes row-ID updates (`update_by_row_id`) to `NativeRowIdPlan` unless the user
disables all three options.
Should `create_row_id_plan_execution()` check `if
table.options.native_write_enabled():` (or respect an explicit
`write.native.enabled=false`) so that setting `write.native.enabled=false`
selects `PythonRowIdPlan` as documented here?
##########
paimon-python/pypaimon/common/options/core_options.py:
##########
@@ -723,7 +723,7 @@ class CoreOptions:
SCAN_NATIVE_PLAN_ENABLED: ConfigOption[bool] = (
ConfigOptions.key("scan.native-plan.enabled")
Review Comment:
Flipping both `SCAN_NATIVE_PLAN_ENABLED` and `READ_NATIVE_ENABLED` defaults
to `True` here breaks single-option opt-out (`scan.native-plan.enabled=false`)
in `native_plan_requested()`
(`paimon-python/pypaimon/read/read_dispatch.py:408-415`):
```python
def native_plan_requested(scan) -> bool:
"""Resolve Native planning preference without loading a disabled
backend."""
if scan.table.options.native_plan_enabled():
return True
if not scan.table.options.native_read_enabled():
return False
from pypaimon.execution.native.runtime import native_runtime_available
return native_runtime_available()
```
Previously, when both options defaulted to `False`, falling through to
`scan.table.options.native_read_enabled()` allowed `read.native.enabled=true`
to also opt into native split planning. Now that `READ_NATIVE_ENABLED` defaults
to `True`, if a user explicitly sets `scan.native-plan.enabled=false` (without
also setting `read.native.enabled=false`),
`scan.table.options.native_plan_enabled()` returns `False`, but
`scan.table.options.native_read_enabled()` still returns `True` from its new
default—so `native_plan_requested(scan)` still returns `True` and ignores
`scan.native-plan.enabled=false` (which is also why the tests in this PR had to
add `'read.native.enabled': 'false'` alongside `'scan.native-plan.enabled':
'false'`).
Similarly, `configure_indexed_read(table)` in
`paimon-python/pypaimon/read/read_dispatch.py:250-259` still checks `if not
any(options.contains(option) for option in (CoreOptions.READ_NATIVE_ENABLED,
CoreOptions.SCAN_NATIVE_PLAN_ENABLED)):` and copies
`{CoreOptions.READ_NATIVE_ENABLED.key(): 'true'}` into `table.options.options`,
which overrides the `PYPAIMON_TEST_EXECUTION=python` /
`@pytest.mark.python_read` test-lane default in `conftest.py:246-252` whenever
`pypaimon_rust` is installed.
If `SCAN_NATIVE_PLAN_ENABLED` is explicitly configured in
`scan.table.options`, `native_plan_requested(scan)` should respect it directly:
```python
def native_plan_requested(scan) -> bool:
"""Resolve Native planning preference without loading a disabled
backend."""
if
scan.table.options.options.contains(CoreOptions.SCAN_NATIVE_PLAN_ENABLED):
return scan.table.options.native_plan_enabled()
if not (scan.table.options.native_plan_enabled() or
scan.table.options.native_read_enabled()):
return False
from pypaimon.execution.native.runtime import native_runtime_available
return native_runtime_available()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]