saloni-eng commented on PR #12281: URL: https://github.com/apache/seatunnel/pull/12281#issuecomment-5663807721
@goutamadwant @DanielLeens I think I confirmed the root cause for the OOM risk, HttpClientProvider.getResponse() calls EntityUtils.toString(), which buffers the entire entity before my code ever sees it, so my line-splitting in SplunkSourceReader doesn't help. A real fix means avoiding that full buffering, which requires either: a) Adding a new streaming method to HttpClientProvider (e.g. executeLineStreaming, mirroring the existing executeBinaryStreaming pattern) and having SplunkSourceReader override pollAndCollectData() to use it directly. b) Some other approach you'd prefer, if there's already a mechanism for this I'm missing. Option 1 means touching connector-http-base, which is shared across every HTTP connector — is that in scope for this PR, or would you rather I split it into a separate PR against connector-http-base that this one then depends on? Also want to confirm: is there a specific max response size / heap ratio you'd want this to hold up to? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
