[
https://issues.apache.org/jira/browse/KAFKA-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16303141#comment-16303141
]
ASF GitHub Bot commented on KAFKA-4335:
---------------------------------------
phstudy opened a new pull request #4356: KAFKA-4335: Add batch.size to
FileStreamSource connector to prevent OOM
URL: https://github.com/apache/kafka/pull/4356
When the source file of `FileStreamSource` is a large file,
`FileStreamSourceTask.poll()` will result in OOM. This pull request added
`batch.size` parameter which can restrict the poll size.
*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*
*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> FileStreamSource Connector not working for large files (~ 1GB)
> --------------------------------------------------------------
>
> Key: KAFKA-4335
> URL: https://issues.apache.org/jira/browse/KAFKA-4335
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 0.10.0.0
> Reporter: Rahul Shukla
>
> I was trying to sink large file about (1gb). FileStreamSource connector is
> not working for that it's working fine for small files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
