xiaokang opened a new pull request #8451: URL: https://github.com/apache/incubator-doris/pull/8451
# Proposed changes Issue Number: NA ## Problem Summary: Due to unlimited queue in OlapScanNode and NodeChannel, memory usage can be very large for reading and writting large table, e.g 'insert into tableB select * from tableA'. ## Checklist(Required) 1. Does it affect the original behavior: (No) 2. Has unit tests been added: (No) 3. Has document been added or modified: (No Need) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments 1. Add bytes limit (2.5% of query_options.mem_limit) for _scan_row_batches queue in OlapScanNode. There was no limit for _scan_row_batches yet. 2. Add bytes limit (2.5% of exec_mem_limit) for _materialized_row_batches queue in OlapScanNode. There was only queue size limit for _materialized_row_batches before, and it's not enough for data with large average bytes of a single row. 3. Add bytes limit for reading a single batch. A new be config doris_scanner_row_bytes is added and its default value is 10MB. There was only row number limit before, and it's also not enough for data with large average bytes of a single row. 5. Add bytes limit (5% of load_mem_limit) for _max_pending_batches queue in NodeChannel. 6. Fixed an exceptional zero value of max_thread in OlapScanNode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
