ByteYue opened a new pull request, #34354: URL: https://github.com/apache/doris/pull/34354
## Proposed changes Issue Number: close #xxx In practice, we've found that if the import frequency to HDFS is too fast, it can cause an OutOfMemoryError (OOM) in the JVM started by the JNI. For this, we should have a method to monitor how much JVM memory is currently being used. The HdfsWriteRateLimit class increments a recorded value during hdfsWrite when writing to HDFS. When hdfsCloseFile is called, all related memory in the JVM will be invalidated, so the recorded value can be decreased at that time. However, if the current usage exceeds the maximum set by the user, the current write will sleep. If the number of sleeps exceeds the number specified by the user, then the current write is considered to have failed. <!--Describe your changes.--> ## Further comments If this is a relatively large or complex change, kick off the discussion at [[email protected]](mailto:[email protected]) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
