Hi Apex Dev community,
Kindly provide with feedback if any for the following approach for
*Input *: BlockMetaData containing offset and length
*Expected Output :* Records in the block
*Approach : *
Similar to approach currently being followed in FSRecordReader.
1) Fetch the block from S3. S3 block fetch size should ideally be large
enough, say 64MB to avoid unnecessary network delays.
2) Search for newline character in the block and emit the record
3) The last record in current block might overflow into subsequent block.
For this, we will get a small part of subsequent block, say 1 MB and search
for newline character and emit the record if newline character is found. We
will fetch additional 1MB blocks till a newline charater is found.
4) We will also avoid reading the first record from all blocks (except
first block) as this set of bytes is a part of last record in previous
On Wed, Oct 19, 2016 at 7:31 AM, Ajay Gupta (JIRA) <j...@apache.org> wrote:
> [ https://issues.apache.org/jira/browse/APEXMALHAR-2303?
> page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
> Ajay Gupta reassigned APEXMALHAR-2303:
> Assignee: Ajay Gupta
> > S3 Line By Line Module
> > ----------------------
> > Key: APEXMALHAR-2303
> > URL: https://issues.apache.org/
> > Project: Apache Apex Malhar
> > Issue Type: Bug
> > Reporter: Ajay Gupta
> > Assignee: Ajay Gupta
> > Original Estimate: 336h
> > Remaining Estimate: 336h
> > This is a new module which will consist of 2 operators
> > 1) File Splitter -- Already existing in Malhar library
> > 2) S3RecordReader -- Read a file from S3 and output the records
> (delimited or fixed width)
> This message was sent by Atlassian JIRA