This looks great Stephen -- this will make the process of running IO ITs much easier. I'll take a look at the doc and leave comments if I have any.
On Wed, Jul 5, 2017 at 5:02 PM, Kenneth Knowles <[email protected]> wrote: > I have some extra nerdy maven-fu (pom-fu?) to suggest: use -D instead of -P > for a little more flexibility. > > You can't have one profile activate another, but you _can_ activate two > profiles with the same property. [1] > > This doesn't work: mvn -P profile1 > > <profiles> > <profile> > <id>profile1</id> > <properties> > <profile2>true</profile2> > </properties> > </profile> > > <profile> > <id>profile2</id> > <activation> > <property> > <name>profile2</name> > </property> > </activation> > </profile> > </profiles> > > (actually it doesn't work for two reasons - see below about [2]) > > This does work: mvn -D shared-semantic-property > > <profiles> > <profile> > <id>profile1</id> > <activation> > <property> > <name>shared-semantic-property</name> > </property> > </activation> > </profile> > > <profile> > <id>profile2</id> > <activation> > <property> > <name>shared-semantic-property</name> > </property> > </activation> > </profile> > </profiles> > > > Unfortunately, this is only true for _system_ properties, not those set up > in a <properties> block. [2] > > This doesn't work: mvn -D profile1 > > <properties> > <profile2>${profile1}</profile2> > </properties> > > <profiles> > <profile> > <id>profile1</id> > <activation> > <property> > <name>profile1</name> > </property> > </activation> > </profile> > > <profile> > <id>profile2</id> > <activation> > <property> > <name>profile2</name> > </property> > </activation> > </profile> > </profiles> > > So at that point I think you are stuck. But you can use the basic -D trick > to do things like split the various alterations into separate profiles, > which might end up having different boolean combinations with other -D > options. > > Kenn > > [1] > https://stackoverflow.com/questions/943411/can-i-make- > one-maven-profile-activate-another > [2] > https://stackoverflow.com/questions/5676231/how-to- > activate-profile-by-means-of-maven-property > > On Wed, Jul 5, 2017 at 3:11 PM, Stephen Sisk <[email protected]> > wrote: > > > hey all, > > > > I wanted to share an early draft of what it'll be like to invoke mvn for > > the IO integration tests in the future when we have the integration with > > kubernetes going. > > > > I'm really excited about these changes - working on the IO ITs, I have to > > run them frequently, and the command lines to run them can be quite a > bear. > > For example: > > > > mvn -e verify -Dit.test=org.apache.beam.sdk.io.jdbc.JdbcIOIT > > -DskipITs=false -pl sdks/java/io/jdbc -Pio-it -Pdataflow-runner > > -DintegrationTestPipelineOptions=["--project=[project]","-- > > gcpTempLocation=gs://[bucket]/staging","--postgresUsername= > > postgres","--postgresPassword=uuinkks","--postgresDatabaseName=postgres" > > ,"--postgresSsl=False","--postgresServerName=[1.2.3.4]", > > "--runner=TestDataflowRunner","--defaultWorkerLogLevel=INFO"] > > > > Also, in order to run this, I first need to have created an instance of > > this datastore in kubernetes and then copied the parameter and > inevitably I > > mis-copy something in there or something changes, so it doesn't work > > correctly and I have to go back in and edit it. > > > > So that's a pain. > > > > To invoke the IO ITs in the future, it'll be a command like this: > > mvn verify -Pio-it-suite -pl sdks/java/io/jdbc > > -DpkbLocation="path-to-pkb.py" \ > > -DintegrationTestPipelineOptions='["--tempRoot=my-temp-root"]' > > (or at least, that's what I'm proposing :) > > > > This will run the jdbc integration tests, spinning up the data store for > > that integration test in your kubernetes cluster. > > > > This is all enabled by a combination of adding new profiles in maven for > > each IO and changes to the beam benchmarks in pkb (perfkitbenchmarker) to > > control kubernetes. Jason has already done a lot of work to get pkb > working > > to run our regular benchmarks, and I'm excited to continue that work for > IO > > ITs. We use pkb to control kubernetes and capture our benchmark times. > This > > means you'll need to install pkb if you'd like to use this nicer > > experience, however, devs will never have to use pkb if they don't want > to, > > nor is making changes in pkb required when you want to add a new IO IT. > You > > can always spin up the data store yourself, and invoke the integration > test > > directly. > > > > Drafts of these changes can be seen at [0] and [1] - however, I don't > > expect most folks will care about these changes other than "how do I > invoke > > this?", so let me know if you have comments about how this is invoked. > > > > S > > > > [0] pom changes hooking up the call to pkb - > > https://github.com/ssisk/beam/commit/eec7cb5b71330761e71850e8e6f65f > > 34249641b0 > > [1] pkb changes enabling kubernetes spin up- > > https://github.com/ssisk/PerfKitBenchmarker/commits/kubernetes_create > > (last > > 2 changes) > > > -- ------- Jason Kuster Apache Beam / Google Cloud Dataflow
