mattcasters commented on PR #7539:
URL: https://github.com/apache/hop/pull/7539#issuecomment-4992649253
## Update since PR opened
Follow-up work on the native Spark engine after the original PR (target
streams / multi-input / early docs):
### Engine / runtime
- **Classic I/O on mapPartitions**: stage textfile / getfilenames / excel;
partition-scoped `${Internal.Transform.ID}` / CopyNr / BundleNr + Beam-parity
`beamContext` for Text File Output uniqueness.
- **Accept-filenames (Get File Names → Text File Input)**: re-bind
`accept_transform_name` to the mini-pipeline main injector (`_INJECTOR_`);
preload all filename rows and drive with direct `processRow()` (avoids
`SingleThreadedPipelineExecutor` stall after `createReader` when input hop size
hits 0).
- **Active graph only**: skip disabled/orphaned transforms so sinks without
an enabled input path are not executed.
- **Spark File Output** minor handler tweak; unit tests for accept-filenames
binding, parallel file context, converter active-graph behaviour.
- **`BaseFileInputMeta.setAcceptingTransformName`** for engine rebinding
without protected `getFileInput()` access.
- Small **Text File Input dialog** NPE guard on empty date locale.
### Integration tests (`integration-tests/spark-native`)
- **0007** Text File Input spark readback (create CSVs → GFN → TFI → Parquet
→ validate).
- **0008** Excel catalog as Stream Lookup info + main from 3 Parquet parts →
golden unit test.
- 0099 complex golden/pipeline touch-ups.
### Documentation
- Rewrote **Getting started with the native Spark pipeline engine**: what it
is, native vs Beam, version matrix, `local[*]` run config, fat jar
(`native-provided`), detailed `spark-submit` (incl. **4th arg environment
config** for cluster vars like `PROJECT_HOME=s3://…`), Spark File I/O,
`${Internal.Transform.ID}`, metrics & execution information, limitations (Group
By, Copy/Distribute, **no native Kafka / Spark SQL / MLlib**), and illustrated
IT examples.
- Screenshots under `docs/.../assets/images/pipeline/spark/`.
- Linked from pipelines overview and native Spark run-config page.
Ready for another review pass when convenient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
