jonathansudhakar1 commented on issue #22480: URL: https://github.com/apache/beam/issues/22480#issuecomment-7016218533
@Abacn, before I write any code I'd like to check a plan for part of this issue with you, since you're driving the move from the native Python Spanner IO to the cross-language one (#35860). It covers the write side of "match the Java connector's configuration options". **1. Failed mutations can't be reached from Python.** `SpannerInsert`, `SpannerUpdate` and the other write wrappers expand through `SpannerTransformRegistrar.WriteBuilder`, which returns `SpannerIO.WriteRows`, a `PDone` ([registrar L313-L314](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerTransformRegistrar.java#L313-L314), [L432](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerTransformRegistrar.java#L432); [SpannerIO L1860-L1868](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java#L1860-L1868)). With `failure_mode=FailureMode.REPORT_FAILURES` the pipeline keeps going, but `SpannerWriteResult.getFailedMutations()` is drop ped, so Python users can't see or dead-letter the failed rows. The `spanner_write:v1` SchemaTransform (used by YAML's `WriteToSpanner`) does have an `errors` output with `failed_row` and `error_message` ([L180-L208](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerWriteSchemaTransformProvider.java#L180-L208)). But it always writes insert-or-update ([L157](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerWriteSchemaTransformProvider.java#L157)) and only takes project, instance, database, table and error handling. **2. Java write options that Python can't set:** `withMaxCommitDelay`, `withTransactionTag`, `withExcludeTxnFromChangeStreams`, `withLowPriority` (Python only has `high_priority`), `withEnableOpenTelemetryTracing` and `SpannerConfig.withDatabaseRole`. Also, the registrar already accepts `experimentalHost`, `plainText`, `clientCertPath` and `clientCertKeyPath` (#36048, #38928), but `ReadFromSpannerSchema` and `WriteToSpannerSchema` in `io/gcp/spanner.py` never send them ([L133-L146](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/python/apache_beam/io/gcp/spanner.py#L133-L146), [L292-L306](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/python/apache_beam/io/gcp/spanner.py#L292-L306)). **Proposed approach, as two PRs:** - **PR 1 (additive):** add the options in (2) to `WriteBuilder.Configuration` and to the Python wrappers as optional keyword arguments, defaulting to today's behavior. I'd fold in two small fixes: `high_priority` is missing from the write docstring (`_INIT_DOC`), and `ReadFromSpanner` validates its arguments with `assert` ([L256-L267](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/python/apache_beam/io/gcp/spanner.py#L256-L267)), where `assert sql or table and not (sql and table)` accepts both. Those would become `ValueError`s. - **PR 2:** have `WriteBuilder` return a `PCollection<Row>` of failed rows, with the same schema as the `errors` output of `spanner_write:v1`. Existing `rows | SpannerInsert(...)` code keeps working, and the output stays empty unless `REPORT_FAILURES` is set. Deletes carry keys rather than column values, so their failed rows would be built from the key. - Alternative: move the Python wrappers onto `spanner_write:v1`, adding the operation type and the options to its configuration. That would also give YAML the options, but it changes a SchemaTransform's configuration, so I'd only do it if you prefer it. **Tests:** the wrapper has no Python unit tests today (only `xlang_spannerio_it_test.py`, which needs the emulator), so I'd add payload tests on the Python side, plus `SpannerTransformRegistrarTest` and `SpannerIOWriteTest` cases with `FakeServiceFactory` on the Java side. I can't run the emulator or the cross-language ITs locally, and the PRs would say so. Two questions: for the failed mutations, does returning a `PCollection<Row>` from the existing registrar work for you, or would you rather go through `spanner_write:v1`? And do you want all of the options in (2), or only some of them? If the direction sounds right, I'd like to take this on. Separately, `ReadFromSpanner` still requires `row_type`, even though Java can now infer the read schema (`readWithSchema()`, #32008). Making it optional in Python would address #20415; I'd leave that for a later PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
