hansva opened a new issue, #9435: URL: https://github.com/apache/hop/issues/9435
### What needs to happen? The Data set input transform is slow and memory hungry on large data sets, for two reasons. 1. `DataSetCsvUtil.getAllRows(IVariables, DataSet)` clones the value metadata of every field for every row. A fresh clone has no cached number or date format, so every Integer, Number, BigNumber and Date value builds a new `DecimalFormat` or `SimpleDateFormat`. In a profile, 77% of the transform's CPU was spent building formats. 2. `DataSetInput.processRow()` calls `getAllRows`, which parses the complete file into a list, and only then writes the rows. A data set of 2 million rows with 11 fields needs about 2.8 GB before the first row leaves the transform. ### Proposed changes 1. **Value metadata once per field.** Before the read loop, prepare one value meta and one String conversion meta per field, and use them for every row. The other `getAllRows` variant, which unit tests use for input and golden data sets, already converts without a clone per row, so the result does not change. 2. **Stream the rows.** Add a reader to `DataSetCsvUtil` that hands over one row at a time, and let Data set input write each row as it is parsed. The transform should stop reading when the pipeline is stopped. `getAllRows` stays for the callers that need the full list (the data set editor, the location dialog, unit test injection and golden data comparison) and is built on the same reader. Behavior to keep: - A data set whose file does not exist yet gives no rows and no error in Data set input. - Field order, types, the storage masks from `setValueFormats`, hex-encoded Binary fields and the trim of padded BigNumber values stay as they are. - A record with fewer values than the data set has fields leaves the missing fields null. ### Measured gain Change 1 alone, as a prototype with only `DataSetCsvUtil.class` swapped, on a data set of 2 million rows with 11 fields, 3 runs each: | | Before | After | |---|---|---| | Rows/s | 91k-114k | 442k-482k | | Engine time | 17.6-22.1 s | 4.2-4.6 s | | Peak memory | 2.3-3.0 GB | 1.6-2.2 GB | ### Issue Priority Priority: 2 ### Issue Component Component: Transforms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
