Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/4964#issuecomment-81908574
I thought your approach in
https://github.com/harishreedharan/spark/commit/fa93b871ba0fe22924ff0273e975e492a6a7043c
was to clear metadata on only the latest computed branch, rather than the
computed batch. We could do that in combination with the flag in my PR that
disables cleaning for the pre-batch-start checkpoint.
On Mon, Mar 16, 2015 at 1:01 PM, Hari Shreedharan <[email protected]>
wrote:
> But how would that combination really handle the case where concurrentJobs
> > 1? From what I can see we need either of harishreedharan@fa93b87
>
<https://github.com/harishreedharan/spark/commit/fa93b871ba0fe22924ff0273e975e492a6a7043c>
> and #5008 <https://github.com/apache/spark/pull/5008>, since both solve
> this in pretty much the same way.
>
> I know that concurrentJobs > 1 is not common, but it is non-zero. I have
> seen this being used in production.
>
> â
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/4964#issuecomment-81905729>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
