zhengruifeng commented on PR #58336: URL: https://github.com/apache/spark/pull/58336#issuecomment-5436711588
I ran an in-process comparison against a model subclass that recreated the previous bound-method UDF expressions. For closure size, the probe serialized each `ScalaUDF.function` with Spark's closure serializer using a binary model with 4,096 features: | Output columns | Before | After | Delta | |---|---:|---:|---:| | raw prediction | 41,825 B | 35,267 B | -15.7% | | probability | 41,833 B | 35,527 B | -15.1% | | prediction | 41,763 B | 35,184 B | -15.8% | | raw + probability + prediction | 125,532 B (3 UDFs) | 37,605 B (2 UDFs) | -70.0% | For execution time, the probe measured 20 alternating actions over 3,000,000 cached rows with 32 features. The table reports median times: | Transform path | Before | After | Ratio | |---|---:|---:|---:| | prediction only | 214.710 ms | 191.852 ms | 0.894x | | raw prediction to prediction | 501.883 ms | 503.178 ms | 1.003x | | probability to prediction | 528.609 ms | 502.011 ms | 0.950x | The temporary probe was removed after measurement. The affected binary and multinomial correctness tests pass after the final UDF signature adjustment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
