[
https://issues.apache.org/jira/browse/BEAM-8470?focusedWorklogId=349858&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-349858
]
ASF GitHub Bot logged work on BEAM-8470:
----------------------------------------
Author: ASF GitHub Bot
Created on: 26/Nov/19 15:57
Start Date: 26/Nov/19 15:57
Worklog Time Spent: 10m
Work Description: echauchot commented on issue #10221: [BEAM-8470]
Exclude failed ValidatesRunner tests
URL: https://github.com/apache/beam/pull/10221#issuecomment-558695063
> We have disabled a bunch of other tests (that are failing) as well. Do we
need to enable them in this case?
well, you have a point.
> If they are always red, it's difficult to say if we have a regression or
not with every new PR coming. You need to compare test results all the time,
it's not convenient (this is what I already did several times).
you're right here also.
What is the correct way to avoid forgetting fixing the non-passing tests ?
Any suggestion better than listing content of build.gradle exclusions ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 349858)
Time Spent: 15h 10m (was: 15h)
> Create a new Spark runner based on Spark Structured streaming framework
> -----------------------------------------------------------------------
>
> Key: BEAM-8470
> URL: https://issues.apache.org/jira/browse/BEAM-8470
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
> Reporter: Etienne Chauchot
> Assignee: Etienne Chauchot
> Priority: Major
> Time Spent: 15h 10m
> Remaining Estimate: 0h
>
> h1. Why is it worth creating a new runner based on structured streaming:
> Because this new framework brings:
> * Unified batch and streaming semantics:
> * no more RDD/DStream distinction, as in Beam (only PCollection)
> * Better state management:
> * incremental state instead of saving all each time
> * No more synchronous saving delaying computation: per batch and partition
> delta file saved asynchronously + in-memory hashmap synchronous put/get
> * Schemas in datasets:
> * The dataset knows the structure of the data (fields) and can optimize
> later on
> * Schemas in PCollection in Beam
> * New Source API
> * Very close to Beam bounded source and unbounded sources
> h1. Why make a new runner from scratch?
> * Structured streaming framework is very different from the RDD/Dstream
> framework
> h1. We hope to gain
> * More up to date runner in terms of libraries: leverage new features
> * Leverage learnt practices from the previous runners
> * Better performance thanks to the DAG optimizer (catalyst) and by
> simplifying the code.
> * Simplify the code and ease the maintenance
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
