AlexanderNenninger opened a new issue, #27059: URL: https://github.com/apache/datafusion/issues/27059
### Describe the bug I found this inconsistency while checking DataFusion against a [formalization](https://github.com/AlexanderNenninger/substrait-lean) of Substrait plan semantics, when plans that call Substrait's standard shift functions were rejected by `from_substrait_plan`. Substrait's `functions_arithmetic.yaml` defines `shift_left`, `shift_right` and `shift_right_unsigned` (added in Substrait v0.74.0, substrait-io/substrait#799). DataFusion's consumer maps the names `bitwise_shift_left` and `bitwise_shift_right` to its shift operators, and rejects a plan that calls the standard names with `Unsupported function name: "shift_left"`. The producer writes the same non-standard names, so a DataFusion plan with `a << 1` calls `bitwise_shift_left`, which other consumers won't find in the extension. The current names may simply predate the standard functions. ### To Reproduce DataFusion 55.0.0 (Python bindings, `datafusion==55.0.0`, built on DataFusion 55.1.0); the same on 51.0.0. The consumer's mapping on `main` still lists only the `bitwise_` names (`datafusion/substrait/src/logical_plan/consumer/expr/scalar_function.rs`). ```python # uv run --no-project --with datafusion==55.0.0 --with "substrait>=0.37" python DF-10.py import json import datafusion from datafusion import SessionContext from datafusion.substrait import Consumer, Producer, Serde from google.protobuf import json_format from substrait.plan_pb2 import Plan print("datafusion", datafusion.__version__) ctx = SessionContext() # SELECT shift_left(a, 1), shift_right(a, 1) FROM (VALUES (CAST(5 AS INT))) t(a), # written with the functions of Substrait's functions_arithmetic.yaml. urn = "extension:io.substrait:functions_arithmetic" def call(anchor): return {"scalarFunction": {"functionReference": anchor, "outputType": {"i32": {"nullability": "NULLABILITY_REQUIRED"}}, "arguments": [ {"value": {"selection": {"directReference": {"structField": {"field": 0}}, "rootReference": {}}}}, {"value": {"literal": {"i32": 1}}}]}} plan = { "extensionUrns": [{"extensionUrnAnchor": 1, "urn": urn}], "extensions": [ {"extensionFunction": {"extensionUrnReference": 1, "functionAnchor": 1, "name": "shift_left:i32_i32"}}, {"extensionFunction": {"extensionUrnReference": 1, "functionAnchor": 2, "name": "shift_right:i32_i32"}}], "relations": [{"root": {"names": ["l", "r"], "input": {"project": { "common": {"emit": {"outputMapping": [1, 2]}}, "input": {"read": { "baseSchema": {"names": ["a"], "struct": {"types": [{"i32": {"nullability": "NULLABILITY_REQUIRED"}}], "nullability": "NULLABILITY_REQUIRED"}}, "virtualTable": {"expressions": [{"fields": [{"literal": {"i32": 5}}]}]}}}, "expressions": [call(1), call(2)]}}}}]} msg = json_format.ParseDict(plan, Plan()) try: lp = Consumer.from_substrait_plan(ctx, Serde.deserialize_bytes(msg.SerializeToString())) print(ctx.create_dataframe_from_logical_plan(lp).to_pydict()) except Exception as e: print("consumer error:", e) # The producer, for its part, names these functions bitwise_shift_left / bitwise_shift_right. ctx.sql("CREATE TABLE t (a INT) AS VALUES (5)").collect() p = Producer.to_substrait_plan(ctx.sql("SELECT a << 1 AS l, a >> 1 AS r FROM t").logical_plan(), ctx) out = Plan(); out.ParseFromString(p.encode()) print("producer extensions:", [e.extension_function.name for e in out.extensions]) ``` Output: ```text datafusion 55.0.0 consumer error: DataFusion error: This feature is not implemented: Unsupported function name: "shift_left" producer extensions: ['bitwise_shift_left', 'bitwise_shift_right'] ``` With the two names changed to `bitwise_shift_left` and `bitwise_shift_right`, the same plan runs and returns `{'l': [10], 'r': [2]}`, so only the name is in the way. ### Expected behavior The consumer accepts `shift_left` and `shift_right` and maps them to `Operator::BitwiseShiftLeft` and `Operator::BitwiseShiftRight`, so the plan above returns `{'l': [10], 'r': [2]}`. The producer could write the standard names too, as was done for `is_nan`, `sign` and others in #16949. `shift_right_unsigned` has no DataFusion operator, so an explicit "not supported" error would fit it. ### Additional context - The definitions: https://github.com/substrait-io/substrait/blob/main/extensions/functions_arithmetic.yaml (`shift_left`, `shift_right`, `shift_right_unsigned`). - #16949 / #16950 added other missing Substrait-to-DataFusion name mappings in the same table. ### Willingness to contribute I would like to contribute a fix for this bug, but request guidance from the DataFusion community -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
