khaledh opened a new issue, #40492: URL: https://github.com/apache/beam/issues/40492
### What would you like to happen? Avoid repeated reflective lookup and string interning in the unknown-column collectors used by `TableRowToStorageApiProto.messageFromMap`. These collectors are `TableRow` instances. Unknown-column capture, nested collector attachment, and empty-STRUCT pruning currently use their `GenericData` map operations. Those operations consult `ClassInfo.getFieldInfo` and intern field names, including ordinary column names that are not annotated Java model fields. With `withAutoSchemaUpdate(true)`, this bookkeeping also runs for known nested fields, so large repeated STRUCTs pay the cost even when no unknown columns appear. ### Proposal Use the existing backing map returned by `TableRow.getUnknownKeys()` for ordinary collector keys. Cache the annotated model-field names once and keep the original `TableRow` setter and map operations for those names. For collector emptiness, use backing-map emptiness plus `getF() == null` only when `f` is the model's sole annotated field. Every non-null `f` counts as an entry, including an empty cell list. If a future model adds fields, fall back to the original `isEmpty()` behavior. Keep the collector representation, eager allocation, repeated positions, empty-container pruning, encoded unknown JSON, and model-field exceptions unchanged. This does not change payload conversion, descriptor lookup, required-field validation, timestamps, the cell-format path, or public converter signatures. It adds no dependency or configuration. ### Measurements A downstream Beam 2.68.0 backport was tested in one matched, backlogged Dataflow staging pair. Both variants used 40 fixed `n2-standard-4` workers in the same zone, Streaming Engine, and no autoscaling. Measurement followed a 600-second warmup and covered five settled intervals, about 1,664 seconds of offset capture per variant. | Metric | Control | Collector change | |---|---:|---:| | Committed Kafka offsets/s | 6,124.13 | 7,784.67 | | Estimated CPU seconds per million committed offsets | 25,252.44 | 20,136.16 | That is 27.1% more committed offsets/s and 20.3% lower estimated CPU per committed offset. The CPU estimate uses observed GCE CPU utilization and the fixed vCPU count, not profiler percentages. All five intervals favored the candidate, but its backlog still grew; this does not establish keep-up. In a matching settled CPU-profile window, control `String.intern` accounted for 15.4% of global self CPU. It fell below the candidate's visible top-100 cutoff of 0.163%, not to a proven zero. An overlap check matched 1,040,000 Kafka coordinates without duplicates or side-only coordinates. Coordinate equality is not full payload parity. These are downstream staging results, not a benchmark of the current master port or a production guarantee. Full pipeline payload parity, long-run validation, and attributable allocation/GC savings remain open. ### Related #40188 / #40203 address descriptor lookup, and #40189 / #40204 address required-field validation. Collector access is an independent cost and does not need either change to land first. Draft implementation: #40491. It includes collector regression tests and documents the local validation setup and full-master build limitation. ### Issue Priority Priority: 2 (default / most feature requests should be filed as P2) ### Issue Components - [ ] Component: Python SDK - [x] Component: Java SDK - [ ] Component: Go SDK - [ ] Component: Typescript SDK - [x] Component: IO connector - [ ] Component: Beam YAML - [ ] Component: Beam examples - [ ] Component: Beam playground - [ ] Component: Beam katas - [ ] Component: Website - [ ] Component: Infrastructure - [ ] Component: Spark Runner - [ ] Component: Flink Runner - [ ] Component: Prism Runner - [ ] Component: Twister2 Runner - [ ] Component: Hazelcast Jet Runner - [ ] Component: Google Cloud Dataflow Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
