bhollis-dbx commented on PR #59283: URL: https://github.com/apache/spark/pull/59283#issuecomment-6105252159
### Changes since the initial commit - Nested updates now operate on the current parent-field value without duplicating the input expression. Restored chain flattening, pruning of overwritten/dropped values, and field-extraction simplification. - Generated copies use typed getters; interpreted evaluation uses arrays rather than repeated sequence indexing. - Preserved common-subexpression elimination through `With`, while respecting lexical bindings, lazy definitions, codegen fallback, and null guards. Shared unconditional UDF values are evaluated once per row. - Flat updates no longer introduce unnecessary `With` scopes. This removed the measured planning regression for deeply nested, grouped updates. - Generated copies use arithmetic indexing when source and output fields are contiguous, avoiding ordinal-array lookups. Sparse copies retain the indexed path. Null checks are omitted for non-nullable fields, and the source row is not rechecked inside the already-guarded copy loop. ### Validation 366 tests passed across 11 SQL, Catalyst, optimizer, and codegen suites. New regressions cover nested chains, overwritten/dropped values, evaluate-once behavior, typed copies, CSE scope safety, nullability, and contiguous copies with shifted output positions. Changed sources were compiled against the cached SBT test classpath and run with ScalaTest. Normal SBT project loading remains blocked by Jackson BOM resolution; `runtests` does not support the OSS checkout. ### Benchmarks Latest local projection microbenchmark, expanded struct construction versus compact execution, in ns/row; lower is better: | Fields | Baseline codegen | Compact codegen | Baseline interpreted | Compact interpreted | | --- | ---: | ---: | ---: | ---: | | 5 | 24 | 29 | 131 | 43 | | 100 | 308 | 243 | 970 | 284 | | 500 | 3295 | 1361 | 5198 | 1356 | This measures replacing the first field of a non-nullable integer struct, leaving contiguous copied fields. Codegen is faster at 100 and 500 fields; the five-field case still costs about 5 ns/row more than baseline. Interpreted evaluation is faster at every measured width. Unrolling small copies did not improve the five-field result, so that experiment was discarded. The latest projection run uses longer warmup and more iterations than the earlier measurements; compare baseline and compact within this table rather than absolute timings across runs. Planning results from the preceding run of `UpdateFieldsBenchmark`, upstream versus compact, in average ms: | Case | Baseline non-nullable | Compact non-nullable | Baseline nullable | Compact nullable | | --- | ---: | ---: | ---: | ---: | | Repeated dotted-path updates, 3 depths | 31 | 2 | 509 | 1 | | Grouped updates per level, 100 depths | 561 | 549 | 603 | 548 | These are local microbenchmarks, not end-to-end SQL throughput measurements. `UpdateFieldsBenchmark` processes zero rows and measures planning only. The projection results do not establish performance for sparse copies, mixed field types, or nullable structs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
