joaomalho commented on issue #4212:
URL: https://github.com/apache/hop/issues/4212#issuecomment-2294340612
My apologies for the previous explanation, i wasn't clear. I have configured
the settings in my project metadata as detailed below:
These are the current configurations of the project:
C:\hop\config\projects\project_name\metadata\pipeline-run-configuration\local.json
```
{
"engineRunConfiguration": {
"Local": {
"feedback_size": "10000000",
"sample_size": "100",
"sample_type_in_gui": "Last",
"wait_time": "20",
"rowset_size": "10000000",
"safe_mode": false,
"show_feedback": false,
"topo_sort": false,
"gather_metrics": false,
"transactional": false
}
},
"defaultSelection": true,
"configurationVariables": [],
"name": "local",
"description": "",
"executionInfoLocationName": "Runs your pipelines locally with the
standard local Hop pipeline engine"
}
```
By default if i dont set my "rowset_size": "" the default rows are 10K:
After got this miss configuration i set "rowset_size": "10000000" (10M), and
it works at least until 400k rows (then i stop it because im in test env.).
Although if i set a very large number in "rowset_size", which will be a
reality in the future once our project will collect huge databases, assuming
"rowset_size" : "100000000" (10OM) will also run:
But if i set it to 1B will not work, nether run the pipeline:
I know it is a huge amount of data, and i can arrange workarounds to manage
it. However i'm concerned that this might be a miss configuration on my side,
and i may need to deploy it on server machine if that's the case, or, if this
is a limitation in apache hop i might need to consider splitting the pipeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
