[
https://issues.apache.org/jira/browse/FLINK-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15258499#comment-15258499
]
ASF GitHub Bot commented on FLINK-3717:
---------------------------------------
Github user kl0u commented on the pull request:
https://github.com/apache/flink/pull/1895#issuecomment-214817636
The problem with this change would be that we do not have yet a way to
deserialize a record, so
when we are trying to advance readRecords in the block, we have a NULL
pointer exception.
Actually this is what we do in the AvroInputFormat case, but there it works.
Let me know if you have any other ideas on how to do it, or if you think
that it may be a problem also
for the Avro format.
> On Apr 25, 2016, at 4:24 PM, Aljoscha Krettek <notificati...@github.com>
wrote:
>
> I had some inline comments but overall the changes look good!
>
> I think can simplify the BinaryInputFormat by getting rid of the filePos
and justReadAllRecords fields and just snapshotting the blockPos. The fieldPos
and justReadAllRecords information functionally depend on the blockPos, so
storing the filePos and justReadAllRecords fields just adds more complexity
since we're keeping track of all of them.
>
> The snapshot would then just be (blockPos, readRecords), upon restore the
correct file read position can be derived from the block/split start position.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly or view it on GitHub
<https://github.com/apache/flink/pull/1895#issuecomment-214358848>
> Add functionality to be a able to restore from specific point in a
> FileInputFormat
> ----------------------------------------------------------------------------------
>
> Key: FLINK-3717
> URL: https://issues.apache.org/jira/browse/FLINK-3717
> Project: Flink
> Issue Type: Sub-task
> Components: Streaming
> Reporter: Kostas Kloudas
> Assignee: Kostas Kloudas
>
> This is the first step in order to make the File Sources fault-tolerant. We
> have to be able to get the start from a specific point in a file despite any
> caching performed during reading. This will guarantee that the task that will
> take over the execution of the failed one will be able to start from the
> correct point in the file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
