Indeed, the current directory structure is very Java-centric. We are in
process of fixing that.
We are thinking of going somewhere along the lines of:
sdks
java
py
examples
java
java8
py
runners
google-cloud-dataflow
flink
spark
In a way, we are thinking of having a very limited set of high-level
components (sdks, examples, runners), which are then branched per-language,
as appropriate.
The current top-level directories that should go elsewhere include
java8examples, javadoc, contrib, archetypes.
I think this will get us closer to having other languages as first-class
citizens.
On Tue, Mar 22, 2016 at 7:24 PM, Kenneth Knowles <[email protected]>
wrote:
> +1 JB.
>
> If it works for other incubating projects, then I'm happy to proceed.
>
> On Mon, Mar 21, 2016 at 1:00 PM, Jean-Baptiste Onofré <[email protected]>
> wrote:
>
> > Hi Ben,
> >
> > it works fine with Maven >= 3.2.x (current version is 3.3.9).
> >
> > Most of incubator projects use x.x.x-incubating-SNAPSHOT:
> >
> >
> >
> https://git1-us-west.apache.org/repos/asf?p=incubator-batchee.git;a=blob_plain;f=pom.xml;hb=HEAD
> >
> >
> >
> https://git1-us-west.apache.org/repos/asf?p=incubator-apex-core.git;a=blob_plain;f=pom.xml;hb=HEAD
> >
> >
> >
> https://git1-us-west.apache.org/repos/asf?p=incubator-atlas.git;a=blob_plain;f=pom.xml;hb=HEAD
> >
> >
> >
> https://git1-us-west.apache.org/repos/asf?p=incubator-falcon.git;a=blob_plain;f=pom.xml;hb=HEAD
> >
> > etc
> >
> > And we don't have any problem with Maven, even in OSGi related projects
> > which are a bit "complex" in versioning (as '-' is not allowed).
> >
> > finalName is not a solution, as it's not part of the Maven coordonates.
> >
> > I don't see any valid argument to use a different versioning in Beam, and
> > we will be compliant with release management recommendation (
> > http://incubator.apache.org/guides/releasemanagement.html).
> >
> > IMHO, we should use 0.1.0-incubating-SNAPSHOT (the Maven Parser uses the
> > final -SNAPSHOT, and take 0.1.0-incubating as base version).
> >
> > Regards
> > JB
> >
> >
> > On 03/21/2016 07:22 PM, Ben Chambers wrote:
> >
> >> I don't think Maven will recognize 0.1.0-incubating-SNAPSHOT as a
> >> snapshot.
> >> It will recognize it as 0.1.0 with the "incubating-SNAPSHOT" qualifier.
> >>
> >> For instance, looking at the code for parsing qualifiers, it only
> handles
> >> the string "SNAPSHOT" specially, not "incubating-SNAPSHOT".
> >>
> >>
> http://maven.apache.org/ref/3.0.4/maven-artifact/xref/org/apache/maven/artifact/versioning/ComparableVersion.html#52
> >>
> >> Looking at this Stack Overflow answer (
> >> http://stackoverflow.com/a/31482463/4539304) it looks like support was
> >> improved in Maven 3.2.4 to allow multiple qualifiers (its still unclear
> >> whether incubating would be considered by the code as a qualifier).
> >>
> >> Either way, we shouldn't expect users to upgrade to Maven 3.2.4 or newer
> >> just to get reasonable version number treatment. It seems like sticking
> >> with the standard "-SNAPSHOT" and "" for releases is preferable.
> >>
> >> If the goal is to get incubating into the file names, I think we can
> >> configure the Maven build process to do so. For instance, finalName
> >> defaults to
> >> "<finalName>${project.artifactId}-${project.version}</finalName>". If we
> >> changed that to
> >>
> >>
> "<finalName>${project.artifactId}-incubating-${project.version}</finalName>"
> >> it seems like we'd "incubating" in the file names without needing to
> >> complicate the release numbering.
> >>
> >> On Mon, Mar 21, 2016 at 10:24 AM Jean-Baptiste Onofré <[email protected]>
> >> wrote:
> >>
> >> Hi Ben,
> >>>
> >>> 1. True for Python, but it can go in a folder in sdk (sdk/python)
> >>> anyway. I think the DSLs (Java based) and other languages that we might
> >>> introduce (Scala, ...) can be the same.
> >>>
> >>> 2. The incubating has to be in the released filenames. So it can be in
> >>> the version or name. Anyway, my proposal was 0.1.0-incubating-SNAPSHOT
> >>> for a SNAPSHOT and 0.1.0-incubating for a release (it's what I did in
> >>> the PR). Like this, the Maven standards are still valid.
> >>>
> >>> Regards
> >>> JB
> >>>
> >>> On 03/21/2016 06:20 PM, Ben Chambers wrote:
> >>>
> >>>> 1. Regarding "java" as a module -- are we sure that other languages
> will
> >>>>
> >>> be
> >>>
> >>>> packaged using Maven as well? For instance, Python has its own
> ecosystem
> >>>> which likely doesn't play well with Python.
> >>>>
> >>>> 2. Using the literal "SNAPSHOT" as the qualifier has special meaning
> >>>>
> >>> Maven
> >>>
> >>>> -- it is newer than all other qualified releases, but older than any
> >>>> unqualified release. It feels like we should take advantage of this,
> >>>>
> >>> which
> >>>
> >>>> makes our versioning more consistent with Maven standards.
> Specifically,
> >>>> snapshots should be 0.1.0-SNAPSHOT and releases should be 0.1.0.
> >>>> 0.1.0-SNAPSHOT because that uses the standard definition of
> >>>> SNAPSHOT
> >>>> 0.1.0 because if we had any qualifier than the 0.1.0-SNAPSHOT
> >>>> would
> >>>>
> >>> be
> >>>
> >>>> considered newer
> >>>>
> >>>> Davor's suggestion of putting the "incubating" in the name or
> >>>> description
> >>>> of the artifacts seems like a preferable option.
> >>>>
> >>>> On Mon, Mar 21, 2016 at 7:33 AM Jean-Baptiste Onofré <[email protected]
> >
> >>>> wrote:
> >>>>
> >>>> Hi beamers,
> >>>>>
> >>>>> I updated the PR according to your comments.
> >>>>>
> >>>>> I have couple of points I want to discuss:
> >>>>>
> >>>>> 1. All modules use the same groupId (org.apache.beam). In order to
> have
> >>>>> a cleaner structure on the Maven repo, I wonder if it's not better to
> >>>>> have different groupId depending of the artifacts. For instance,
> >>>>> org.apache.beam.sdk, containing a module with java as artifactId (it
> >>>>> will contain new artifacts with id python, scala, ...),
> >>>>> org.apache.beam.runners containing modules with flink and spark as
> >>>>> artifactId, etc. Thoughts ?
> >>>>> 2. The version has been set to 0.1.0-incubating-SNAPSHOT for all
> >>>>> artifacts, including the runners. It doesn't mean that the runners
> will
> >>>>> have to use the same version as parent (they can have their own
> release
> >>>>> cycle). However, as we "bootstrap" the project, I used the same
> version
> >>>>> in all modules.
> >>>>>
> >>>>> Now, I'm starting two new commits:
> >>>>> - renaming of the packages
> >>>>> - folders re-organization
> >>>>>
> >>>>> Thanks !
> >>>>> Regards
> >>>>> JB
> >>>>>
> >>>>> On 03/21/2016 01:56 PM, Jean-Baptiste Onofré wrote:
> >>>>>
> >>>>>> Hi Davor,
> >>>>>>
> >>>>>> thank you so much for your comments. I'm updating the PR according
> to
> >>>>>> your PR (and will provide explanation to some changes).
> >>>>>>
> >>>>>> Thanks dude !
> >>>>>>
> >>>>>> Regards
> >>>>>> JB
> >>>>>>
> >>>>>> On 03/21/2016 06:29 AM, Davor Bonaci wrote:
> >>>>>>
> >>>>>>> I left a few comments on PR #46.
> >>>>>>>
> >>>>>>> Thanks JB for doing this; a clear improvement.
> >>>>>>>
> >>>>>>> On Mon, Mar 14, 2016 at 6:04 PM, Jean-Baptiste Onofré <
> >>>>>>>
> >>>>>> [email protected]>
> >>>
> >>>> wrote:
> >>>>>>>
> >>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> I started the renaming process from Dataflow to Beam.
> >>>>>>>>
> >>>>>>>> I submitted a first PR about the Maven coordinates:
> >>>>>>>>
> >>>>>>>> https://github.com/apache/incubator-beam/pull/46
> >>>>>>>>
> >>>>>>>> I will start the packages renaming (updating the same PR). For the
> >>>>>>>> directories structure, I would like to talk with Frances, Dan,
> >>>>>>>> Tyler,
> >>>>>>>> and
> >>>>>>>> Davor first.
> >>>>>>>>
> >>>>>>>> Regards
> >>>>>>>> JB
> >>>>>>>> --
> >>>>>>>> Jean-Baptiste Onofré
> >>>>>>>> [email protected]
> >>>>>>>> http://blog.nanthrax.net
> >>>>>>>> Talend - http://www.talend.com
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>> --
> >>>>> Jean-Baptiste Onofré
> >>>>> [email protected]
> >>>>> http://blog.nanthrax.net
> >>>>> Talend - http://www.talend.com
> >>>>>
> >>>>>
> >>>>
> >>> --
> >>> Jean-Baptiste Onofré
> >>> [email protected]
> >>> http://blog.nanthrax.net
> >>> Talend - http://www.talend.com
> >>>
> >>>
> >>
> > --
> > Jean-Baptiste Onofré
> > [email protected]
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
