jonathansudhakar1 commented on issue #22480:
URL: https://github.com/apache/beam/issues/22480#issuecomment-7016218533

   @Abacn, before I write any code I'd like to check a plan for part of this 
issue with you, since you're driving the move from the native Python Spanner IO 
to the cross-language one (#35860). It covers the write side of "match the Java 
connector's configuration options".
   
   **1. Failed mutations can't be reached from Python.** `SpannerInsert`, 
`SpannerUpdate` and the other write wrappers expand through 
`SpannerTransformRegistrar.WriteBuilder`, which returns `SpannerIO.WriteRows`, 
a `PDone` ([registrar 
L313-L314](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerTransformRegistrar.java#L313-L314),
 
[L432](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerTransformRegistrar.java#L432);
 [SpannerIO 
L1860-L1868](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java#L1860-L1868)).
 With `failure_mode=FailureMode.REPORT_FAILURES` the pipeline keeps going, but 
`SpannerWriteResult.getFailedMutations()` is drop
 ped, so Python users can't see or dead-letter the failed rows.
   
   The `spanner_write:v1` SchemaTransform (used by YAML's `WriteToSpanner`) 
does have an `errors` output with `failed_row` and `error_message` 
([L180-L208](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerWriteSchemaTransformProvider.java#L180-L208)).
 But it always writes insert-or-update 
([L157](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerWriteSchemaTransformProvider.java#L157))
 and only takes project, instance, database, table and error handling.
   
   **2. Java write options that Python can't set:** `withMaxCommitDelay`, 
`withTransactionTag`, `withExcludeTxnFromChangeStreams`, `withLowPriority` 
(Python only has `high_priority`), `withEnableOpenTelemetryTracing` and 
`SpannerConfig.withDatabaseRole`. Also, the registrar already accepts 
`experimentalHost`, `plainText`, `clientCertPath` and `clientCertKeyPath` 
(#36048, #38928), but `ReadFromSpannerSchema` and `WriteToSpannerSchema` in 
`io/gcp/spanner.py` never send them 
([L133-L146](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/python/apache_beam/io/gcp/spanner.py#L133-L146),
 
[L292-L306](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/python/apache_beam/io/gcp/spanner.py#L292-L306)).
   
   **Proposed approach, as two PRs:**
   - **PR 1 (additive):** add the options in (2) to 
`WriteBuilder.Configuration` and to the Python wrappers as optional keyword 
arguments, defaulting to today's behavior. I'd fold in two small fixes: 
`high_priority` is missing from the write docstring (`_INIT_DOC`), and 
`ReadFromSpanner` validates its arguments with `assert` 
([L256-L267](https://github.com/apache/beam/blob/d44b8088865011b78a61d419c688c97862ce1914/sdks/python/apache_beam/io/gcp/spanner.py#L256-L267)),
 where `assert sql or table and not (sql and table)` accepts both. Those would 
become `ValueError`s.
   - **PR 2:** have `WriteBuilder` return a `PCollection<Row>` of failed rows, 
with the same schema as the `errors` output of `spanner_write:v1`. Existing 
`rows | SpannerInsert(...)` code keeps working, and the output stays empty 
unless `REPORT_FAILURES` is set. Deletes carry keys rather than column values, 
so their failed rows would be built from the key.
     - Alternative: move the Python wrappers onto `spanner_write:v1`, adding 
the operation type and the options to its configuration. That would also give 
YAML the options, but it changes a SchemaTransform's configuration, so I'd only 
do it if you prefer it.
   
   **Tests:** the wrapper has no Python unit tests today (only 
`xlang_spannerio_it_test.py`, which needs the emulator), so I'd add payload 
tests on the Python side, plus `SpannerTransformRegistrarTest` and 
`SpannerIOWriteTest` cases with `FakeServiceFactory` on the Java side. I can't 
run the emulator or the cross-language ITs locally, and the PRs would say so.
   
   Two questions: for the failed mutations, does returning a `PCollection<Row>` 
from the existing registrar work for you, or would you rather go through 
`spanner_write:v1`? And do you want all of the options in (2), or only some of 
them? If the direction sounds right, I'd like to take this on.
   
   Separately, `ReadFromSpanner` still requires `row_type`, even though Java 
can now infer the read schema (`readWithSchema()`, #32008). Making it optional 
in Python would address #20415; I'd leave that for a later PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to