Hi Kenn, Is there any documentation that needs to accompany the new runner in the pull request or is the documentation added after the pull request is approved?. I would be great if you can point me in the right direction regarding this.
Best Regards, Pulasthi On Mon, Jan 6, 2020 at 9:56 PM Pulasthi Supun Wickramasinghe < pulasthi...@gmail.com> wrote: > Hi Kenn, > > > > On Mon, Jan 6, 2020 at 9:09 PM Kenneth Knowles <k...@apache.org> wrote: > >> >> >> On Mon, Jan 6, 2020 at 8:30 AM Pulasthi Supun Wickramasinghe < >> pulasthi...@gmail.com> wrote: >> >>> Hi Kenn, >>> >>> I was able to solve the problem mentioned above, I am currently running >>> the "ValidatesRunner" tests, I have around 4-5 tests that are failing that >>> I should be able to fix in a couple of days. I wanted to check the next >>> steps I would need to take after all the "ValidatesRunner" tests are >>> passing. I assume that the runner does not need to pass all the >>> "NeedsRunner" tests. >>> >> >> That's correct. You don't need to run the NeedsRunner tests. Those are >> tests of the core SDK's functionality, not the runner. The annotation is a >> workaround for the false cycle in deps "SDK tests" -> "direct runner" -> >> "SDK", which to maven looks like a cyclic dependency. You should run on the >> ValidatesRunner tests. It is fine to also disable some of them, either by >> excluding categories or test classes, or adding new categories to exclude. >> >> >>> The runner is only implemented for the batch mode at the moment because >>> the higher-level API's for streaming on Twister2 are still being finalized. >>> Once that work is done we will add streaming support for the runner as well. >>> >> >> Nice! Batch-only is perfectly fine for a runner. You should be able to >> detect and reject pipelines that the runner cannot execute. >> > > I will make sure that the capability is there. > > >> I re-read the old thread, but I may have missed the answer to a >> fundamental question. Just to get it clear on the mailing list: are you >> intending to submit the runner's code to Apache Beam and ask the community >> to maintain it? >> >> To answer the original question the issue was with forwarding > exceptions that happen during execution since Twister2 has a > distributed model for execution, I added the ability in the Twister2 side > so that the Twister2 job submitting client will receive a job state object > that contains any exceptions thrown during runtime once the job is > completed. > > And about maintaining the twister2 runner, we would like to submit the > runner to the beam codebase but the Twister2 team will maintain and update > it continuously, in that case, we would become part of the Beam community I > suppose. And any contributions from other members of the community are more > than welcome. I hope that answers your question. > > Best Regards, > Pulasthi > > >> Kenn >> >> >> Best Regards, >>> Pulasthi >>> >>> On Thu, Dec 12, 2019 at 11:27 AM Pulasthi Supun Wickramasinghe < >>> pulasthi...@gmail.com> wrote: >>> >>>> Hi Kenn >>>> >>>> We are still working on aspects like automated job monitoring so >>>> currently do not have those capabilities built-in. I discussed with the >>>> Twister2 team on a way we can forward failure information from the workers >>>> to the Jobmaster which would be a solution to this problem. It might take a >>>> little time to develop and test. I will update you after looking into that >>>> solution in a little more detail. >>>> >>>> Best Regards, >>>> Pulasthi >>>> >>>> On Wed, Dec 11, 2019 at 10:51 PM Kenneth Knowles <k...@apache.org> >>>> wrote: >>>> >>>>> I dug in to Twister2 a little bit to understand the question better, >>>>> checking how the various resource managers / launchers are plumbed. >>>>> >>>>> How would a user set up automated monitoring for a job? If that is >>>>> scraping the logs, then it seems unfortunate for users, but I think the >>>>> Beam runner would naturally use whatever a user might use. >>>>> >>>>> Kenn >>>>> >>>>> On Wed, Dec 11, 2019 at 10:45 AM Pulasthi Supun Wickramasinghe < >>>>> pulasthi...@gmail.com> wrote: >>>>> >>>>>> Hi Dev's >>>>>> >>>>>> I have been making some progress on the Twister2 runner for the beam >>>>>> that I mentioned before on the mailing list. The runner is able to run >>>>>> the >>>>>> wordcount example and produce correct results. So I am currently trying >>>>>> to >>>>>> run the runner validation tests. >>>>>> >>>>>> From what I understood looking at a couple examples is that tests are >>>>>> validated based on the exceptions that are thrown (or not) during test >>>>>> runtime. However in Twister2 currently the job submission client does >>>>>> not >>>>>> get failure information such as exceptions back once the job is >>>>>> submitted. >>>>>> These are however recorded in the worker log files. >>>>>> >>>>>> So in order to validate the tests for Twister2 I would have to parse >>>>>> the worker logfile and check what exceptions are in the logs. Would that >>>>>> be >>>>>> an acceptable solution for the validation tests? >>>>>> >>>>>> Best Regards, >>>>>> Pulasthi >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Pulasthi S. Wickramasinghe >>>>>> PhD Candidate | Research Assistant >>>>>> School of Informatics and Computing | Digital Science Center >>>>>> Indiana University, Bloomington >>>>>> cell: 224-386-9035 <(224)%20386-9035> >>>>>> >>>>> >>>> >>>> -- >>>> Pulasthi S. Wickramasinghe >>>> PhD Candidate | Research Assistant >>>> School of Informatics and Computing | Digital Science Center >>>> Indiana University, Bloomington >>>> cell: 224-386-9035 <(224)%20386-9035> >>>> >>> >>> >>> -- >>> Pulasthi S. Wickramasinghe >>> PhD Candidate | Research Assistant >>> School of Informatics and Computing | Digital Science Center >>> Indiana University, Bloomington >>> cell: 224-386-9035 <(224)%20386-9035> >>> >> > > -- > Pulasthi S. Wickramasinghe > PhD Candidate | Research Assistant > School of Informatics and Computing | Digital Science Center > Indiana University, Bloomington > cell: 224-386-9035 > -- Pulasthi S. Wickramasinghe PhD Candidate | Research Assistant School of Informatics and Computing | Digital Science Center Indiana University, Bloomington cell: 224-386-9035
