Re: Adding Twister2 documentation

argument to not help with these > reviews is > the lack of knowledge of the target system (Twister2 in this case) but > anyone > familiar with MapReduce like systems and Beam can definitely give a hand > here > to avoid making me or others a SPOF for reviews. > > On Thu, Sep 10,

Re: Adding Twister2 documentation

@Ismaël Mejía would you have some cycles to validates these changes? Best Regards, Pulasthi On Mon, Aug 31, 2020 at 10:49 PM Pulasthi Supun Wickramasinghe < pulasthi...@gmail.com> wrote: > Hi Devs. > > I created two PR's to add the missing Twister2 runner documentation[1]

Adding Twister2 documentation

Hi Devs. I created two PR's to add the missing Twister2 runner documentation[1] [2], however, would need some guidance on how to fill out the compatibility matrix. 1. Is there some mapping between test classes and entries in the compatibility matrix I can use to crosscheck and tick off

Re: Adding new Jira component for Twister2

Hi Ismaël Yes, just checked and used that for the Jira's. Thanks Best Regards, Pulasthi On Fri, Aug 28, 2020 at 10:59 AM Ismaël Mejía wrote: > Component created I forgot to do this when we merged, please confirm > if it works. > > On Fri, Aug 28, 2020 at 7:47 AM Pulasthi Supun Wi

Adding new Jira component for Twister2

Hi All, While creating a issue for Twister2 i noticed currently there is no component tag for the Twister2 runner. Should we add a new component "runner-twister2", if so what are the steps to creating a component? Best Regards, Pulasthi -- Pulasthi S. Wickramasinghe PhD Candidate | Research

Re: [DISCUSS][BEAM-10670] Migrating BoundedSource/UnboundedSource to execute as a Splittable DoFn for non-portable Java runners

> pipeline support coverage since Spark uses its own tiny suite of tests to > get unbounded pipeline coverage instead of the validates runner set. > Flink: Unstarted. > > @Pulasthi Supun Wickramasinghe , can you help me > with the Twister2 PR[2]? > @Ismaël Mejía , is PR[3] the e

Re: Contributing Twister2 runner to Apache Beam

smaël > > > On Fri, Apr 3, 2020 at 4:16 AM Pulasthi Supun Wickramasinghe > wrote: > > > > Hi Ismaël > > > > Did you get some free time to perform a code review on the pull request > > > > Best Regards > > Pulasthi > > > > On Tue, Mar 10,

Re: Contributing Twister2 runner to Apache Beam

;>> for the >>>>>>> delay). >>>>>>> >>>>>>> > (don't wait for me on code review - if Ismaël said it is good, >>>>>>> then it is >>>>>>> > good.) >>>>>>> >>>>>>>

Re: Contributing Twister2 runner to Apache Beam

ame repository as Beam > >>> core? Can or should they be completely separate products with their > >>> own release cycles? > >>> > >>> Generally, loose coupling leads to more maintainable, reliable > >>> projects. Specifically, Cassandra is holding back some oth

Re: Contributing Twister2 runner to Apache Beam

in an earlier email separately. Best Regards, Pulasthi On Tue, Feb 18, 2020 at 1:45 PM Pulasthi Supun Wickramasinghe < pulasthi...@gmail.com> wrote: > Hi All, > > I have created the initial pull request [1] to contribute the Twister2 > Beam runner to the Apache Beam codebase

Re: [EXTERNAL] Re: Java Build broken

Feb 25, 2020 at 4:33 PM Pulasthi Supun Wickramasinghe < > pulasthi...@gmail.com> wrote: > >> Hi Stefan, >> >> Yes, I am also still getting this error on my local setup, However, >> strangely I am not getting this on my laptop. I tried manually installing >>

Re: Jenkins problems: javaPreCommitPortabilityApiJava11 and No Space left

I got the same build issues for my pull request as well. Java PreCommit also seems to be failing due to a couple of errors in 'BigQureyIO" and "SpannerIO". [1] https://builds.apache.org/job/beam_PreCommit_Java_Commit/10160/java/fixed/ Best Regards, Pulasthi On Wed, Feb 26, 2020 at 1:27 PM Alex

Re: [EXTERNAL] Re: Java Build broken

t; https://oss.sonatype.org/content/repositories/staging/com/google/errorprone/error_prone_check_api/2.3.4/ > > istead of > > > https://mvnrepository.com/artifact/com.google.errorprone/error_prone_check_api/2.3.4 > > > > and the first link returns 404 > > > > >

Contributing Twister2 runner to Apache Beam

Hi All, I have created the initial pull request [1] to contribute the Twister2 Beam runner to the Apache Beam codebase. More information on Twister2 can be found here[2] and the Twister2 codebase is available here[3]. At the moment only batch mode is supported in the runner, but we are planning

Re: Java Build broken

Hi All, Was this issue resolved? I started to get the same error on my local build suddenly. Best Regards, Pulasthi On Thu, Jan 23, 2020 at 10:17 AM Maximilian Michels wrote: > Do you have any overrides in your ~/.m2/settings.xml? The artifacts > should be found as part of Maven central, e.g.

Re: Executing the runner validation tests for the Twister2 runner

ache.org/roadmap/ has the roadmaps for different runners. > For a new runner especially this could be helpful for users. > > Kenn > > On Sun, Jan 12, 2020 at 9:36 AM Pulasthi Supun Wickramasinghe < > pulasthi...@gmail.com> wrote: > >> Hi Kenn, >> >

Re: Executing the runner validation tests for the Twister2 runner

Pulasthi Supun Wickramasinghe < pulasthi...@gmail.com> wrote: > Hi Kenn, > > > > On Mon, Jan 6, 2020 at 9:09 PM Kenneth Knowles wrote: > >> >> >> On Mon, Jan 6, 2020 at 8:30 AM Pulasthi Supun Wickramasinghe < >> pulasthi...@gmail.com> wrote: &

Re: Executing the runner validation tests for the Twister2 runner

Hi Kenn, On Mon, Jan 6, 2020 at 9:09 PM Kenneth Knowles wrote: > > > On Mon, Jan 6, 2020 at 8:30 AM Pulasthi Supun Wickramasinghe < > pulasthi...@gmail.com> wrote: > >> Hi Kenn, >> >> I was able to solve the problem mentioned above, I am currently ru

Re: Executing the runner validation tests for the Twister2 runner

source managers / launchers are plumbed. > > How would a user set up automated monitoring for a job? If that is > scraping the logs, then it seems unfortunate for users, but I think the > Beam runner would naturally use whatever a user might use. > > Kenn > > On Wed, Dec 11, 2

Executing the runner validation tests for the Twister2 runner

Hi Dev's I have been making some progress on the Twister2 runner for the beam that I mentioned before on the mailing list. The runner is able to run the wordcount example and produce correct results. So I am currently trying to run the runner validation tests. >From what I understood looking at

Re: Why is Pipeline not Serializable and can it be changed to be Serializable

in/java/org/apache/beam/runners/core/construction/CoderTranslation.java#L68 > > On Fri, Nov 15, 2019 at 11:00 AM Pulasthi Supun Wickramasinghe < > pulasthi...@gmail.com> wrote: > >> Hi Luke, >> >> Aren't the coders supposed to be serializable? The doc on the Co

Re: Why is Pipeline not Serializable and can it be changed to be Serializable

only class that has this issue. I can work on a pull request to add the private non-args constructors to coders that have them missing if this was not done intentionally. WDYT? Best Regards, Pulasthi On Fri, Nov 15, 2019 at 12:05 AM Pulasthi Supun Wickramasinghe < pulasthi...@gmail

Re: Why is Pipeline not Serializable and can it be changed to be Serializable

BatchPortablePipelineTranslator.java > > On Wed, Nov 13, 2019 at 7:44 PM Pulasthi Supun Wickramasinghe < > pulasthi...@gmail.com> wrote: > >> Hi Dev's >> >> Currently, the Pipeline class in Beam is not Serializable. This is not a >> problem for the current runners

Why is Pipeline not Serializable and can it be changed to be Serializable

Hi Dev's Currently, the Pipeline class in Beam is not Serializable. This is not a problem for the current runners since the pipeline is translated and submitted through a centralized Driver like model. However, if the runner has a decentralized model similar to OpenMPI (MPI), which is also the

Re: why are so many transformation needed for a simple TextIO.write() operation

is about passing around > metadata around what files were written and what errors there were. That > metadata is tiny in comparison and should only be a blip when compared to > writing the files themselves. > > On Sun, Oct 20, 2019 at 10:17 PM Pulasthi Supun Wickramasinghe < >

why are so many transformation needed for a simple TextIO.write() operation

Hi Dev's I was trying to understand the transformations created for the following pipeline, which seems to be pretty simple from the looks of it. But the graph created seems to be pretty complex. I have attached a rough sketch of the graph that I understood from debugging the code below [1]. Was

Re: Using Beam Built-in I/O Transforms with an external framework.

So what you are looking for is a >> way to use a Beam transform in a separate system. >> >> Thanks, >> Cham >> >> [1] >> https://beam.apache.org/contribute/ptransform-style-guide/#exposing-a-ptransform-vs-something-else >> >> On Wed, Sep 1

Using Beam Built-in I/O Transforms with an external framework.

Hi Dev's We have a big data processing framework named Twister2, and wanted to know if there is any way we could leverage the I/O Transforms that are built into Apache Beam externally. That is rather than using it in a Beam pipeline just use them as data sources in our project. Just wanted to

Re: Developing a new beam runner for Twister2

s a "fused" bundle of > > DoFns (called an ExecutableStage). As it looks liek Twister2 is > > written in Java, you can take advantage of much of the existing Java > > libraries that already do this that are shared among the other Java > > runners. &g

Re: Developing a new beam runner for Twister2

asf.slack.com/#beam-portability though I prefer the dev > list > > since it gives visibility to a much greater portion of the community. > > > > Kenn > > > > *volunteers welcome to update the guide to emphasize portability first > > > > *From: *Pu

Developing a new beam runner for Twister2

Hi All, I am Pulasthi a Ph.D. student at Indiana University. We are planning to develop a beam runner for our project Twister2 [1] [2]. Twister2 is a big data framework which supports both batch and stream processing. If you are interested you can find more information on [2] or read some of our