[jira] [Work logged] (BEAM-6608) Flink Runner prepares to-be-staged file too late

Wed, 06 Feb 2019 07:29:09 -0800 


     [ 
https://issues.apache.org/jira/browse/BEAM-6608?focusedWorklogId=195155&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-195155
 ]

ASF GitHub Bot logged work on BEAM-6608:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 06/Feb/19 15:28
            Start Date: 06/Feb/19 15:28
    Worklog Time Spent: 10m 
      Work Description: mxm commented on pull request #7748: [BEAM-6608] 
Prepare to-be-staged files early enough in Flink Runner 
URL: https://github.com/apache/beam/pull/7748
 
 
   #### [BEAM-6608] Prepare to-be-staged files early enough in Flink Runner 
   
   Before execution, the Flink Runner prepares all necessary files, in order for
   them to be shipped to the remote cluster.
   
   Due to a recent regression via BEAM-2873, the preparation is performed after 
the
   execution environment has been initialized. This skips the preparation phase 
and
   results in invalid files being shipped.
   
   #### [BEAM-6609] Default tempLocation in FlinkPipelineOptions to default tmp 
directory
   
   Includes this second commit because it is an easy fix and works on the same 
files. There should be a sensible default, so users don't have to configure the 
temp path by default.
   
   Post-Commit Tests Status (on master branch)
   
------------------------------------------------------------------------------------------------
   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
 </br> [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)
 | --- | --- | ---
   
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


Issue Time Tracking
-------------------

            Worklog Id:     (was: 195155)
            Time Spent: 10m
    Remaining Estimate: 0h

> Flink Runner prepares to-be-staged file too late 
> -------------------------------------------------
>
>                 Key: BEAM-6608
>                 URL: https://issues.apache.org/jira/browse/BEAM-6608
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>    Affects Versions: 2.10.0
>            Reporter: Maximilian Michels
>            Assignee: Maximilian Michels
>            Priority: Major
>             Fix For: 2.11.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Before execution, the Flink Runner prepares all necessary files, in order for 
> them to be shipped to the remote cluster.
> Due to a recent regression via BEAM-2873, the preparation is performed after 
> the execution environment has been initialized. This skips the preparation 
> phase and results in invalid files being shipped.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to