CarlosJuncher03 commented on issue #7571: URL: https://github.com/apache/hop/issues/7571#issuecomment-5022510842
Another aspect worth examining is the "data validation" transformation's ability to handle moderate workloads. Currently, because validation is performed row-by-row, a bottleneck arises; furthermore, the use of external sources for validation limits scalability, and memory consumption appears to be quite high. It might be possible to implement a more efficient approach, such as batch processing or parallel execution. Just a thought. Regarding parallel execution: I’m not sure if it’s a bug, but if I use a "Table Input" step for data validation and enable parallel execution, the step fails to locate the data from the "Table Input." -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
