RamonZhou opened a new pull request, #59563: URL: https://github.com/apache/spark/pull/59563
### What changes were proposed in this pull request? Replace the three `descriptor.label == descriptor.LABEL_REPEATED` checks in `SparkConnectClient._truncate` with `descriptor.is_repeated`. Also tighten `SparkConnectBasicTests.test_truncate_message` so it asserts that the truncated string actually contains `[truncated]`. Before this, the test only checked the length, which the error placeholder also satisfies, so it passed even when truncation failed. ### Why are the changes needed? protobuf 7.34 removed `FieldDescriptor.label`; it had been deprecated since 6.32 in favor of `is_repeated` and `is_required`. The latest `grpcio-status` (1.84.0) and `googleapis-common-protos` (1.75.5) allow `protobuf<8`, so a fresh install of the PySpark Connect client now resolves protobuf 7.36.2. On protobuf 7, `_truncate` raises `AttributeError: 'google._upb._message.FieldDescriptor' object has no attribute 'label'`. `_proto_to_string` catches it, so every "Executing plan" / "Schema for plan" / "Explain" debug log line shows `<Truncated message due to truncation error>` instead of the plan. On protobuf 6.32 - 6.33 the same code emits `DeprecationWarning: label() is deprecated. Use is_required() or is_repeated() instead.` `is_repeated` was added in protobuf 6.31.0. PySpark's minimum `grpcio-status` (1.76.0) already requires `protobuf>=6.31.1`, and the generated `pyspark.sql.connect.proto` modules require a runtime of at least 6.33.5, so no fallback to `label` is needed. ### Does this PR introduce _any_ user-facing change? Yes, for debug logging only. With protobuf 7, the Spark Connect client's plan debug logs show the truncated plan again instead of `<Truncated message due to truncation error>`. On protobuf 6.32 - 6.33 the `label` DeprecationWarning no longer appears. Query behavior is unchanged. ### How was this patch tested? - Updated `test_truncate_message` in `pyspark.sql.tests.connect.test_connect_basic`. - Manually ran `_proto_to_string(plan, truncate=True)` on a hand-built plan (a repeated message field, repeated long strings, and a bytes field), with DeprecationWarnings from `core.py` turned into errors: | protobuf | Before | After | |---|---|---| | 7.36.2 | `<Truncated message due to truncation error>` (AttributeError) | truncated plan | | 6.33.6 | `<Truncated message due to truncation error>` (DeprecationWarning raised as error) | truncated plan | ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac (Claude Opus 5.5) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
