konradwudkowski opened a new issue, #20741: URL: https://github.com/apache/superset/issues/20741
When superset returns results it converts nested fields to json with [this line of code](https://github.com/apache/superset/blob/f3b289d3c333fe2351e9fbac6fa85b875cb1897c/superset/result_set.py#L65-L67) This results in **excessive** amount of memory used and kills celery worker. The problem manifests itself for large enough number of records. I noticed it on Presto with big highly nested records. I suspect it may show on other databases. I realize that more records means more memory but it my tests selecting 200k records (this value will vary based on data itself) resulted in megabytes of memory used until SupersetResultSet is [created](https://github.com/apache/superset/blob/f3b289d3c333fe2351e9fbac6fa85b875cb1897c/superset/sql_lab.py#L316) and this explodes to gigabytes and was failing when celery worker had 10gb of memory allocated. #### How to reproduce the bug I encountered the problem on Presto with big highly nested records. 1. Create a dataset with many records and nested fields (so that this line is executed https://github.com/apache/superset/blob/f3b289d3c333fe2351e9fbac6fa85b875cb1897c/superset/result_set.py#L142) 2. For high enough number of records selected superset will OOM (in our case celery worker so query fails but superset itself lives on) Error Stack trace ``` celeryworker_1 | [2022-07-12 12:42:15,512: INFO/ForkPoolWorker-1] Query 5: Polling the cursor for progress celeryworker_1 | [2022-07-12 12:42:40,194: ERROR/MainProcess] Process 'ForkPoolWorker-1' pid:12 exited with 'signal 9 (SIGKILL)' celeryworker_1 | [2022-07-12 12:42:41,037: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 9 (SIGKILL) Job: 3.') celeryworker_1 | Traceback (most recent call last): celeryworker_1 | File "/opt/superset/billiard/pool.py", line 1265, in mark_as_worker_lost celeryworker_1 | raise WorkerLostError( celeryworker_1 | billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL) Job: 3. ``` ### Expected results I expect that a result set that takes several megabytes of memory before processing will not blow. ### Actual results OOM on celery worker ### Environment (please complete the following information): - browser type and version: any - superset version: `1.5.1` - python version: `3.8.12` ### Checklist Make sure to follow these steps before submitting your issue - thank you! - [x] I have checked the superset logs for python stacktraces and included it here as text if there are any. - [x] I have reproduced the issue with at least the latest released version of superset. - [x] I have checked the issue tracker for the same issue and I haven't found one similar. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
