Dear Beams, I've made progress with the Neo4jIO source/sink implementation and left a trail in BEAM-1857 <https://issues.apache.org/jira/browse/BEAM-1857?focusedCommentId=17436020&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17436020> . I'll continue to test a bit more with more data to test some edge cases and then I'll generate a PR so you can review it.
TLDR: I promised you and some folks at Google a review of my struggles to get this IO developed so below is that. It looks like most of my initial de-railing came from an old gradle version which came by default on my Ubuntu 20.04.3 LTS environment. The Beam Contribute <https://beam.apache.org/contribute/> page, while quite lengthy, seems to be accurate but perhaps it would be prudent to also mention a minimal gradle version just to be on the safe side. On a side note I would advise developers to first fork apache/beam instead of cloning it directly so that it's easier to generate PRs later on. Here's an example <http://hop.apache.org/dev-manual/latest/setup-dev-environment.html> on how we approached this in Apache Hop. Perhaps it can remove unnecessary hurdles later on. The next point where things went awry is that the IntelliJ IDEA setup information is a bit outdated for recent IDEA versions. (I'm on 2021.2 myself) When trying it manually by following the IntelliJ IDE instructions <https://cwiki.apache.org/confluence/display/BEAM/Using+IntelliJ+IDE> things seem to go wrong, especially wrt to Gradle and Google AutoValue configurations. The feedback from this thread was very useful to get past this, more in particular this command: *./gradlew idea* A very useful command for sure except that as a non-gradle expert I couldn't for the life of me add the new sdks/java/io/neo4j module in IDEA after a successful import of the whole project. In the end I started from a clean slate and added the directory with a build.gradle file (outside of IntelliJ) and added *include(":sdks:java:io:neo4j")* in gradle.settings.kts. After that I ran *./gradlew idea* again which worked out. It would be great if there was an easier way but I was happy enough to find a solution. AutoValue itself is cool from a technology viewpoint and once it got properly integrated in IntelliJ IDEA it worked well enough. That being said, it has its own learning curve and set of peculiarities. Particularly the mapping between the getters/setters, the with* methods and the actual code in the DoFn classes is rather tedious and making any changes to data types (oops I used a Long instead of an Integer) becomes a lengthy exercise. In general I would have the tendency to spin off a number of these classes but AutoValue only seems to support static inlined classes so I fear that you tend to end up with monolithic classes with in general low code density. I don't mind but I have my doubts that the code is actually easier to maintain this way. Adding more comments would be nitpicking since on the whole I've liked developing this IO ;-) Thanks again for all the help! Cheers, Matt On Tue, Oct 19, 2021 at 9:20 AM Matt Casters <matt.cast...@neo4j.com> wrote: > Thanks Brian, > > I couldn't so I noticed with "git status" that something had changed some > gradle files. Probably an earlier attempt with the older gradle version or > IDEA. Getting everything back in sync with upstream master all of a sudden > made everything come back to life. Thanks for sticking with me! > When I'm done I'll document the whole process. > > Cheers, > Matt > > On Mon, Oct 18, 2021 at 9:10 PM Brian Hulette <bhule...@google.com> wrote: > >> Can you run "./gradlew sdks:java:core:compileJava" or "./gradlew >> sdks:java:core:test"? I'm not sure why gradle is failing to find the idea >> task for you - could you share the output of "./gradlew tasks"? >> >> On Mon, Oct 18, 2021 at 1:19 AM Matt Casters < >> matt.cast...@neotechnology.com> wrote: >> >>> Thanks a lot for the advice given last week. >>> Just to circle back: I've updated gradle to a recent version which >>> appears to be 7.1.1. ... to no avail. >>> >>> tasks like: >>> >>> ./gradlew idea >>> >>> result simply in >>> >>> Task 'idea' not found in root project 'beam'. >>> >>> The same goes for the other suggestions. >>> >>> As for Google Auto: is this project still maintained? The docs and so on >>> seem to be getting quite old. >>> The annotation processor in the latest IntelliJ doesn't seem to get >>> picked up even if you configure it manually in the settings. >>> So I'll skip that one for now. >>> >>> Are there any build instructions I can follow for Beam to at least try >>> to build the Java SDK and go from there? >>> >>> Thanks, >>> >>> Matt >>> >>> >>> On Tue, Oct 12, 2021 at 9:58 PM Evan Galpin <evan.gal...@gmail.com> >>> wrote: >>> >>>> @Matt have you tried any of the "IDE Tasks" available through gradle? >>>> "./gradlew tasks" from beam top-level will list available tasks, and the >>>> IDE Tasks subsection includes tasks specific to trying to bootstrap or >>>> clean up beam project in either Eclipse or Intellij. Ex. "./gradlew idea" >>>> should set up the project files for use in Intellij. There's also >>>> "./gradlew cleanIdea" which may be helpful to you. >>>> >>>> With respect to Google Auto, I've experienced plenty of IDE complaints >>>> around missing types and the like, and those will likely persist until the >>>> code area that you're working on is compiled because the types won't exist >>>> until the sources are generated at pre-processing stage. Not sure if that >>>> was the issue you were having, but if so hopefully this helps. >>>> >>>> Thanks, >>>> Evan >>>> >>>> On Tue, Oct 12, 2021 at 2:51 PM Matt Casters <matt.cast...@neo4j.com> >>>> wrote: >>>> >>>>> Thanks Chamikara but I'm quite familiar with the Beam API and the >>>>> contribution guide did not answer my questions. >>>>> >>>>> On Tue, Oct 12, 2021 at 8:49 PM Chamikara Jayalath < >>>>> chamik...@google.com> wrote: >>>>> >>>>>> If you haven't already, going through Beam contribution guide and >>>>>> varils links from there might help: >>>>>> https://beam.apache.org/contribute/ >>>>>> Regarding developing I/O connectors, please see the guide here: >>>>>> https://beam.apache.org/documentation/io/developing-io-overview/ >>>>>> >>>>>> Thanks, >>>>>> Cham >>>>>> >>>>>> On Tue, Oct 12, 2021 at 6:08 AM Alexey Romanenko < >>>>>> aromanenko....@gmail.com> wrote: >>>>>> >>>>>>> Hi Matt, >>>>>>> >>>>>>> On 12 Oct 2021, at 10:02, Matt Casters < >>>>>>> matt.cast...@neotechnology.com> wrote: >>>>>>> >>>>>>> 1) Setting up my Beam development >>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/Set+up+IntelliJ+from+scratch> >>>>>>> environment >>>>>>> for IDEA 2021.2 is something that's going wrong, probably around Gradle >>>>>>> configurations. >>>>>>> >>>>>>> 2) I can't get Google Auto to work in my IDE (IDEA) because of what >>>>>>> seems outdated documentation >>>>>>> <https://github.com/google/auto/blob/master/value/userguide/index.md> >>>>>>> ? >>>>>>> >>>>>>> >>>>>>> Could you elaborate more what is wrong with 1) and 2) ? >>>>>>> >>>>>> >>>>>>> 3) Since I'm obviously planning to generate a PR at the end of this >>>>>>> exercise: what is the suggested code format for Java in the Beam >>>>>>> project? >>>>>>> >>>>>>> >>>>>>> Please, run this command before committing your changes: >>>>>>> >>>>>>> ./gradlew spotlessApply && ./gradlew >>>>>>> -PenableCheckerFramework=true checkstyleMain checkstyleTest javadoc >>>>>>> spotbugsMain compileJava compileTestJava >>>>>>> >>>>>>> To save a time, run it only against a package where you did the >>>>>>> changes. >>>>>>> >>>>>>> — >>>>>>> Alexey >>>>>>> >>>>>>> >>>>> >>>>> -- >>>>> Neo4j Chief Solutions Architect >>>>> *✉ *matt.cast...@neo4j.com >>>>> >>>>> >>>>> >>>>> > > -- > Neo4j Chief Solutions Architect > *✉ *matt.cast...@neo4j.com > > > >
