[
https://issues.apache.org/jira/browse/GOBBLIN-968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shirshanka Das resolved GOBBLIN-968.
------------------------------------
Fix Version/s: 0.16.0
Resolution: Fixed
Issue resolved by pull request #2816
[https://github.com/apache/incubator-gobblin/pull/2816]
> Honor file split size for HadoopFileInputSource
> -----------------------------------------------
>
> Key: GOBBLIN-968
> URL: https://issues.apache.org/jira/browse/GOBBLIN-968
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-core
> Affects Versions: 0.14.0
> Reporter: Chirag Todarka
> Assignee: Abhishek Tiwari
> Priority: Major
> Labels: easyfix
> Fix For: 0.16.0
>
> Original Estimate: 24h
> Time Spent: 50m
> Remaining Estimate: 23h 10m
>
> For HadoopFileInputSource it is splitting the source file only in default
> chunk size (which is 128MB) and not honoring the min and mix split size
> mentioned in pull file.
> For example even if below entires are present in pull file it will still
> chunk the source file in smaller 134217728 bytes files and not in 268435456
> bytes files.
> {{source.hadoop.file.input.split.minsize=268435456}}
> {{source.hadoop.file.input.split.maxsize=268435456}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
