Re: goVet and clickHouse tests failing
Adding +Sameer Abhyankar for visibility. On Fri, Nov 29, 2019 at 4:10 PM Amogh Tiwari wrote: > Hi Elliotte, > Thanks for your reply. > Tried out the steps that you suggested, still not able to resolve this > issue. > Should I use a particular version of go for the build? > What I have noticed is that when I ./gradlew build, it automatically > downloads go 1.12 and sets the go env for it, basically ignoring whatever > version of go that I already have installed. > Please follow the last comment of this > https://issues.apache.org/jira/browse/BEAM-8564. > Regards > Amogh > > > On Wed, Nov 27, 2019 at 5:46 PM Elliotte Rusty Harold > wrote: > >> I did get through this one, and made the classic mistake of not >> immediately committing the steps I took to writing. I believe it >> involved some combination of setting go paths in environment >> variables. I seem to have added this to the end of my .profile: >> >> export GOROOT=/usr/local/go >> export GOPATH=$HOME/go/packages >> export PATH=$GOPATH/bin:$GOROOT/bin:$PATH >> >> According to my history, I also ran >> >> go get github.com/linkedin/goavro >> >> I really need to dedicate some time to cleaning up the build >> documentation. See https://issues.apache.org/jira/browse/BEAM-8798 >> >> Right now we have overlapping and sometimes contradictory docs in four >> different places: >> >> README.md >> CONTRIBUTING.md >> https://cwiki.apache.org/confluence/display/BEAM/Contributor+FAQ >> https://beam.apache.org/contribute/ >> >> We should probably pick one as the source of truth and rewrite the >> other three to simply point to it. I propose putting all checkout, >> build, test, commit, and push instructions in CONTRIBUTING.md in the >> repo. What do folks think? >> >> On Wed, Nov 27, 2019 at 3:51 AM Amogh Tiwari wrote: >> > >> > Hi Elliotte, >> > I am facing a similar goVet issue. It would be great if you can guide >> me through the solution. Please let me know the steps that you followed. >> > Regards, >> > Amogh >> > >> > On Thu, Nov 21, 2019 at 6:06 PM Elliotte Rusty Harold < >> elh...@ibiblio.org> wrote: >> >> >> >> Tentatively, the goVet issue does seem to have been an issue with my >> >> Go install I have now cleaned up. The clickhouse issue remains, as do >> >> several others I'm working through. >> >> >> >> I've filed https://issues.apache.org/jira/browse/BEAM-8798 to >> >> consolidate and update the instructions for getting to a working >> >> build. >> >> >> >> On Thu, Nov 21, 2019 at 6:10 AM Elliotte Rusty Harold >> >> wrote: >> >> > >> >> > I'm slowly working my way through getting the tests to run and pass. >> >> > We have a lot of work to do on the contributing docs to explain how >> to >> >> > setup and run the build. There's clearly a lot of knowledge in >> >> > developers' heads and workstations that hasn't yet made it into the >> >> > docs. >> >> > >> >> > The latest is a problem finding "github.com/linkedin/goavro" when I >> >> > run goVet. I'm not a go person. Is this something that requires an >> >> > extra install? If so, how is it installed? Or is this some error in >> >> > the build.gradles? Or perhaps my go config is borked and gradle is >> >> > looking in the wrong directory? >> >> > >> >> > > Task :sdks:go:examples:resolveBuildDependencies >> >> > Resolving ./github.com/apache/beam/sdks/go@/home/elharo/beam/sdks/go >> >> > .gogradle/project_gopath/src/ >> github.com/apache/beam/sdks/go/examples/vendor/github.com/apache/beam/sdks/go/pkg/beam/io/avroio/avroio.go:28:2 >> : >> >> > cannot find package "github.com/linkedin/goavro" in any of: >> >> > >> /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/ >> github.com/apache/beam/sdks/go/examples/vendor/github.com/linkedin/goavro >> >> > (vendor tree) >> >> > /home/elharo/.gradle/go/binary/1.12/go/src/ >> github.com/linkedin/goavro >> >> > (from $GOROOT) >> >> > >> /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/ >> github.com/linkedin/goavro >> >> > (from $GOPATH) >> >> > >> >> > > Task :sdks:go:examples:goVet FAILED >> >> > >> >> > I'm also seeing failures in ClickHouseIOTest: >> >> > >> >> > > Task :sdks:java:io:clickhouse:test >> >> > >> >> > org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod >> FAILED >> >> > java.lang.IllegalStateException >> >> > >> >> > org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod >> FAILED >> >> > java.lang.NullPointerException >> >> > >> >> > org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod >> FAILED >> >> > java.lang.IllegalStateException >> >> > >> >> > org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod >> FAILED >> >> > java.lang.NullPointerException >> >> > >> >> > -- >> >> > Elliotte Rusty Harold >> >> > elh...@ibiblio.org >> >> >> >> >> >> >> >> -- >> >> Elliotte Rusty Harold >> >> elh...@ibiblio.org >> >> >> >> -- >> Elliotte Rusty Harold >> elh...@ibiblio.org >> >
Re: [DISCUSS] AWS IOs V1 Deprecation Plan
Thanks Alex for the summary, and all for your contributes. I have been waiting for a couple of days, and so far we don't have any objections. So I guess we can move forward with this approach. We can, of course, wait until next week to see if anyone else has any ideas, and then make a final decision to close this discussion :-) Thanks, Cam On Thu, Nov 28, 2019 at 7:47 AM Alexey Romanenko wrote: > Going back to main subject of this thread, I just wanted to make things > clear for all. > > Seems like that everybody is agree that we will *just deprecate* AWS SDK > V1 connectors once the alternative will be available, *don’t remove* them > and still *distribute artifacts* [1] with new releases along with > artifacts with IOs based on AWS SDK V2 [2]. > > Do we need to vote for this or we can accept it without voting if there > are no objections? > > [1] > https://search.maven.org/search?q=a:beam-sdks-java-io-amazon-web-services > [2] > https://search.maven.org/search?q=a:beam-sdks-java-io-amazon-web-services2 > > > On 27 Nov 2019, at 18:44, Kenneth Knowles wrote: > > > On Tue, Nov 26, 2019 at 7:00 PM Chamikara Jayalath > wrote: > >> >> >> On Tue, Nov 26, 2019 at 6:17 PM Reza Rokni wrote: >> >>> With regards to @Experimental there are a couple of discussions around >>> its usage ( or rather over usage! ) on dev@. It is something that we >>> need to clean up ( some of those IO are now being used on production env >>> for years!). >>> >> >> I agree that we should move some IO connectors out of experimental state >> and probably this should be a separate discussion. Also, this issue is >> probably more than for IO connectors since there are other parts of the >> code that is marked as experimental as well, sometimes for a good reason >> (for example, SDF). >> > > Yes, let's have a separate thread on @Experimental. There are a ton of > threads that start talking about it, and they all seem to agree it isn't > working. Only one direct thread* that was about something a bit more > specific > https://lists.apache.org/thread.html/302bd51c77feb5c9ce39882316d391535a0fc92e7608a623d9139160@%3Cdev.beam.apache.org%3E > > > > >> On Tue, Nov 26, 2019 at 8:21 AM Alexey Romanenko < > aromanenko@gmail.com> wrote: > >> AFAICT, all AWS SDK V1 IOs (SnsIO, SqsIO, DynamoDBIO, KinesisIO) are >> marked as "Experimental". So, it should not be a problem to gracefully >> deprecate and finally remove them. We already did the similar procedure >> for >> “HadoopInputFormatIO”, which was renamed to just “HadoopFormatIO” (since >> it >> started to support HadoopOutputFormatI as well). Old >> “HadoopInputFormatIO” >> was deprecated and removed after *3 consecutive* Beam releases (as >> we agreed on mailing list). >> >> In the same time, some users for some reasons would not be able or to >> want to move on AWS SDK V2. So, I’d prefer to just deprecate AWS SDK V1 >> IOs >> and accept new features/fixes *only* for V2 IOs. >> > >> +1 for deprecating AWS V1 IO connectors as opposed to removing as well >> unless we can confirm that usage is extremely limited. >> > > +1 to deprecate as soon as there is an alternative. > > Trying to measure usage is a good idea, but hard. If the maven coordinates > were different we could look at download numbers and dependencies. > > > Talking about “Experimental” annotation. Sorry in advance If I missed that >> and switch a subject a bit, but do we have clear rules or an agreement >> when >> IO becomes stable and should not be marked as experimental anymore? >> *Most* of our Java IOs are marked as Experimental but many of them >> were using in production by real users under real load. Does it mean that >> they are ready to be stable in terms of API? Perhaps, this topic >> deserves a >> new discussion if there are several opinions on that. >> > >> Probably, decision to move component APIs (for example, an IO connector) >> out of experimental state should be done on a case-by-case basis. >> > > Let's repeat these good points on a dedicated thread. > > Kenn > > > >> >> Thanks, >> Cham >> >> >>> >> On 26 Nov 2019, at 00:39, Luke Cwik wrote: >> >> Phase I sounds fine. >> >> Apache Beam follows semantic versioning and I believe removing the >> IOs will be a backwards incompatible change unless they were marked >> experimental which will be a problem for Phase 2. >> >> What is the feasibility of making the V1 transforms wrappers around >> V2? >> >> On Mon, Nov 25, 2019 at 1:46 PM Cam Mach wrote: >> >>> Hello Beam Devs, >>> >>> I have been working on the migration of Amazon Web Services IO >>> connectors into the new AWS SDK for Java V2. The goal is to have an >>> updated >>> implementation aligned with the most recent AWS improvements. So far we >>> have already migrated the connectors for AWS SNS, SQS and
Re: goVet and clickHouse tests failing
Hi Elliotte, Thanks for your reply. Tried out the steps that you suggested, still not able to resolve this issue. Should I use a particular version of go for the build? What I have noticed is that when I ./gradlew build, it automatically downloads go 1.12 and sets the go env for it, basically ignoring whatever version of go that I already have installed. Please follow the last comment of this https://issues.apache.org/jira/browse/BEAM-8564. Regards Amogh On Wed, Nov 27, 2019 at 5:46 PM Elliotte Rusty Harold wrote: > I did get through this one, and made the classic mistake of not > immediately committing the steps I took to writing. I believe it > involved some combination of setting go paths in environment > variables. I seem to have added this to the end of my .profile: > > export GOROOT=/usr/local/go > export GOPATH=$HOME/go/packages > export PATH=$GOPATH/bin:$GOROOT/bin:$PATH > > According to my history, I also ran > > go get github.com/linkedin/goavro > > I really need to dedicate some time to cleaning up the build > documentation. See https://issues.apache.org/jira/browse/BEAM-8798 > > Right now we have overlapping and sometimes contradictory docs in four > different places: > > README.md > CONTRIBUTING.md > https://cwiki.apache.org/confluence/display/BEAM/Contributor+FAQ > https://beam.apache.org/contribute/ > > We should probably pick one as the source of truth and rewrite the > other three to simply point to it. I propose putting all checkout, > build, test, commit, and push instructions in CONTRIBUTING.md in the > repo. What do folks think? > > On Wed, Nov 27, 2019 at 3:51 AM Amogh Tiwari wrote: > > > > Hi Elliotte, > > I am facing a similar goVet issue. It would be great if you can guide me > through the solution. Please let me know the steps that you followed. > > Regards, > > Amogh > > > > On Thu, Nov 21, 2019 at 6:06 PM Elliotte Rusty Harold < > elh...@ibiblio.org> wrote: > >> > >> Tentatively, the goVet issue does seem to have been an issue with my > >> Go install I have now cleaned up. The clickhouse issue remains, as do > >> several others I'm working through. > >> > >> I've filed https://issues.apache.org/jira/browse/BEAM-8798 to > >> consolidate and update the instructions for getting to a working > >> build. > >> > >> On Thu, Nov 21, 2019 at 6:10 AM Elliotte Rusty Harold > >> wrote: > >> > > >> > I'm slowly working my way through getting the tests to run and pass. > >> > We have a lot of work to do on the contributing docs to explain how to > >> > setup and run the build. There's clearly a lot of knowledge in > >> > developers' heads and workstations that hasn't yet made it into the > >> > docs. > >> > > >> > The latest is a problem finding "github.com/linkedin/goavro" when I > >> > run goVet. I'm not a go person. Is this something that requires an > >> > extra install? If so, how is it installed? Or is this some error in > >> > the build.gradles? Or perhaps my go config is borked and gradle is > >> > looking in the wrong directory? > >> > > >> > > Task :sdks:go:examples:resolveBuildDependencies > >> > Resolving ./github.com/apache/beam/sdks/go@/home/elharo/beam/sdks/go > >> > .gogradle/project_gopath/src/ > github.com/apache/beam/sdks/go/examples/vendor/github.com/apache/beam/sdks/go/pkg/beam/io/avroio/avroio.go:28:2 > : > >> > cannot find package "github.com/linkedin/goavro" in any of: > >> > > /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/ > github.com/apache/beam/sdks/go/examples/vendor/github.com/linkedin/goavro > >> > (vendor tree) > >> > /home/elharo/.gradle/go/binary/1.12/go/src/ > github.com/linkedin/goavro > >> > (from $GOROOT) > >> > > /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/ > github.com/linkedin/goavro > >> > (from $GOPATH) > >> > > >> > > Task :sdks:go:examples:goVet FAILED > >> > > >> > I'm also seeing failures in ClickHouseIOTest: > >> > > >> > > Task :sdks:java:io:clickhouse:test > >> > > >> > org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod > FAILED > >> > java.lang.IllegalStateException > >> > > >> > org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod > FAILED > >> > java.lang.NullPointerException > >> > > >> > org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod > FAILED > >> > java.lang.IllegalStateException > >> > > >> > org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod > FAILED > >> > java.lang.NullPointerException > >> > > >> > -- > >> > Elliotte Rusty Harold > >> > elh...@ibiblio.org > >> > >> > >> > >> -- > >> Elliotte Rusty Harold > >> elh...@ibiblio.org > > > > -- > Elliotte Rusty Harold > elh...@ibiblio.org >
