nsivabalan commented on issue #16917: URL: https://github.com/apache/hudi/issues/16917#issuecomment-4800682895
This appears to have been fixed by [PR #13139](https://github.com/apache/hudi/pull/13139) ([HUDI-9309] Fix perf regression from supporting display dag of insert/update statment in spark ui), commit `b91183d764b0`. That change replaced the `sparkAdapter.createSparkRowSerDe(...)` + `query.execute().map(sparkRowSerDe.deserializeRow)` round-trip introduced by #13044 with `sparkSession.internalCreateDataFrame(query.execute(), schema)` in both `InsertIntoHoodieTableCommand` and `UpdateHoodieTableCommand`, eliminating the extra map/WholeStageCodegen stages that caused the 19% regression. Current master (spark3-common and spark4-common variants) uses `internalCreateDataFrame` at line 119 of `InsertIntoHoodieTableCommand.scala`. Should this issue be closed? (Note: the JIRA key referenced in the issue body is HUDI-9310, but the fix landed under HUDI-9309 — likely the same ticket.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
