claudevdm opened a new pull request, #40817: URL: https://github.com/apache/beam/pull/40817
Threads a destination-table key through the `AddFiles` registration chain, holding the key constant at the configured table. | stage | before | after | |---|---|---| | `ReadFooterSchema` | `DoFn<String, SchemaGroup>` | `DoFn<KV<String,String>, KV<String,SchemaGroup>>` | | `ConvertToDataFile` | `DoFn<String, SerializableDataFile>` | `DoFn<KV<String,String>, KV<String,SerializableDataFile>>` | | `CreateManifests` | keyed by `specId` | keyed by `KV<table, specId>` | | `CommitManifestFilesDoFn` | `identifier` field | `batch.getKey()` | `AddFiles.expand` supplies the constant with `WithKeys.<String,String>of(tableIdentifier)`, and `distinctSchemas` drops the key again so its global combine and the commit path downstream are untouched. Reviewable as six steps, in this order: 1. key the footer read, add the `fileSchemas` helper 2. key its two test helpers 3. the converter's cached table becomes a `ConcurrentMap<String, TableState>`, name still constant 4. key the converter; `identifier` and its constructor parameters are deleted 5. the manifest key becomes `KV<table, specId>` 6. the committer reads its table from the key -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
