[
https://issues.apache.org/jira/browse/BEAM-5309?focusedWorklogId=170643&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-170643
]
ASF GitHub Bot logged work on BEAM-5309:
----------------------------------------
Author: ASF GitHub Bot
Created on: 29/Nov/18 11:26
Start Date: 29/Nov/18 11:26
Worklog Time Spent: 10m
Work Description: b923 commented on issue #6691: WIP:[BEAM-5309] Add
streaming support for HadoopFormatIO
URL: https://github.com/apache/beam/pull/6691#issuecomment-442799898
Sorry for delay,
I pushed last version which is stable for our internal jobs.
The problem with `HDFSSynchronization` was solved by replacing
`FileSystem.get()` by `FileSystem.newInstance()` because `newInstance()` does
not uses cached instance of `FileSystem` and thus its `close()` does not
influences other calls of `FileSystem`s methods.
We also did comparison of `get()` and `newInstance()` methods in our
internal job and we didn't found significant difference in time requirements
for execution. This fact is meaningful since creation of `FileSystem` instance
is called once per Beam bundle and thus it minimizes chance to use shared
instance of `FileSystem` multiple times on one work node since Beam targets to
maximally distribute execution of bundles across work nodes in cluster (in
batch mode).
Do you have any comments @aromanenko-dev ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 170643)
Time Spent: 12h (was: 11h 50m)
> Add streaming support for HadoopOutputFormatIO
> ----------------------------------------------
>
> Key: BEAM-5309
> URL: https://issues.apache.org/jira/browse/BEAM-5309
> Project: Beam
> Issue Type: Sub-task
> Components: io-java-hadoop
> Reporter: Alexey Romanenko
> Assignee: David Hrbacek
> Priority: Minor
> Time Spent: 12h
> Remaining Estimate: 0h
>
> design doc: https://s.apache.org/beam-streaming-hofio
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
