Re: Understanding the snapshot and release process

[Christofer Dutz]

Ok … reporting some progress here.

I managed to get the build to create the java8 and the java7 artifacts in one 
run and to run the tests in both versions.
The only thing I am currently struggling with is the Maven toolchain support.

Maven provides a mechanism called “toolchains” in which you can for example 
define multiple JDKs and use this to do exactly what we want: run all tests in 
the normal modules with Java8 and the ones in the Java7 modules with Java7. 
Unfortunately, we would need the paths to the JDKs on Travis to use this. It 
would be easy on Apache Jenkins, but right now I think it would probably be 
better to let the tests on Travis all run with Java8 and to run the Java7 tests 
with Java7 on the ASF Jenkins.

What do you think?

Will continue porting all the individual modules … currently only ported the 
api modules . A lot more to go ;-)

Chris



Am 17.06.17, 16:29 schrieb "Christofer Dutz" :

Ok … so I think I might have a possible solution, thanks to Karl Heinz 
Marbaise on the Maven list.

We don’t use profiles at all and build all artifacts in one run. The 
difference would be that we create mini modules in the “java7” directory which 
build the java7 versions of the java8 artifacs by copying and unpacking the 
java8 versions to the mini projects target directory, have retrolambda do it’s 
magic there and package them normally. In this case, I would like to use a 
different groupId for the java7 artifacts instead of using different versions 
as this way we will have far less trouble releasing stuff.

Using maven toolchains I might even be able to run the transformed tests in 
the mini projects with java7 … haven’t used toolchains till now, but it looks 
as if they would deliver exactly the functionality we need.

So I’m going to give this approach a try.

Chris


Am 16.06.17, 18:43 schrieb "Dale LaBossiere" :


> On Jun 16, 2017, at 4:46 AM, Christofer Dutz 
 wrote:
> 
> In case of a variable for the groupId the warning is the same except 
that it mentions groupId instead of version :-(

Hmm…  any thoughts on plausible alternatives? ...and implications to 
dev, release, and consumption of java7 Edgent?

At a high level, something just seems wrong that it’s this hard to
create multiple variants of a product w/mvn, where the variants’
jars need different coordinates along some dimension.
The retrolambda plugin may be complicating things but it doesn't
feel like it’s the cause of the main issue: variable coordinate decls.

Related, without the use of variable-version decls, even ignoring
this j8/j7 thing, there would be 1.2.0-SNAPSHOT
throughout every one of our (~65) poms.  Even using
variable-version decls, that static decl is present in every pom’s
 decl!  Is it the really case that when we want to release
1.2.0 (not SNAPSHOT) or when it’s time for 1.3.0, every pom
has to be edited?

Just brainstorming…   my head hurts :-\   I apologize in advance…  :-)

E means Edgent-java ...

Is there any ~convenient way to synthesize/use alternate pom’s for E7?
Might there be a way to leverage “mvn —file ” and also 
pom.xml:?

e.g., have the pom.xml implicitly be j8/E8 oriented with non-variable 
coordinate decl for the project.

Imagine building / staging a RC for E7 would be something like:

mk-j7-poms  # create pom-j7.xml files adding “-java7” in 
versions and a “relativePath” of ../pom-j7.xml
 # **/pom-7.xml added to .gitignore; target 
dir is still variable

mvn -f pom-j7.xml -Pjava7 clean install -DskipTests
# TBD do j7 testing
# …
mvn -f pom-j7.xml -Pjava7 deploy -Papache-release -DskipTests

Or if we’re going that far, then maybe needing/using a separate 
clone/workspace to build & deploy E7 might be tolerable?

Though… while the gradle/ant tooling created j7 jars using *the* j8 
jars, 
this mvn tooling is already regenerating the j8 classes 
and then running them through retrolambda, right? (if so, ugh)

Kinda feels like E7 artifacts should instead just declare a
dependency on their corresponding E8 jar artifact (not src)
and just do some (retrolambda) transform to generate the E7 jar 
artifacts.
Which seems to imply separate (~parallel) simpler poms for E7
which can then also have non-variable-version decl?
e.g., then something like
cd java7
mvn clean install -DskipTests
# TBD test
mvn 

Re: Understanding the snapshot and release process

[Christofer Dutz]

Just have a look at the last few builds on Travis that my pull request 
initiated. You will see it failing with different tests every time I ran the 
build even if nothing changed (I think in the last commit I simply removed a 
space in one of the poms).

Was successful …

After moving some tests from Unit-Test to Integration-Test phase:
Failure in module “Apache Edgent: Providers: Development”
- DevelopmentPlumbingTest>PlumbingTest.testParallelBalanced:628

After adding the explice lifecycle mapping plugin config (has no effect on the 
build)
Failure in module “Apache Edgent: Connectors: WSClient Websocket”
- [ERROR]   
WebSocketClientGlobalTest>WebSocketClientTest.testReconnect:359->TopologyAbstractTest.completeAndValidate:82->TopologyAbstractTest.completeAndValidate:98
  contents:[一, 二, 四]
- [ERROR]   
WebSocketClientGlobalTest>WebSocketClientTest.testSslReconnect:571->TopologyAbstractTest.completeAndValidate:82->TopologyAbstractTest.completeAndValidate:98
  contents:[一, 二, 四]

After removing a simple space character in a pom
Failure in module “Apache Edgent: Providers: Direct”
- [ERROR]   DirectPlumbingTest>PlumbingTest.testPressureReliever:194 
delay:TAI:332@380

So it seems that there is a hand full of tests that could need some hardening …

Chris

Am 15.06.17, 14:42 schrieb "Dale LaBossiere" :

Thx for the cleanup for Eclipse.

> On Jun 15, 2017, at 7:09 AM, Christofer Dutz  
wrote:
> ...It seems as if some unit-tests tend to fail in random intervals. Right 
now, I had 3 Travis runs without any functional changes and I got failures on 3 
different places. Is this “normal”? If yes … I think I should start 
identifiying the “random failers” and investigate the reason.

What were the failing tests?

MqttOpenTest and Http[Global]Test use external public servers and can be 
impacted by them.  An attempt has been made to desensitize them (e.g., trying 
to connect and then skipping instead of failing tests) but sometimes things 
seem to leak through.

All of the tests have time limits imposed and will “fail" if they don’t 
complete in time (some have had more narrow limits than others).  I’ve seen 
these sort of failures on occasion such as when the test system is heavily 
loaded.

— Dale

Re: Understanding the snapshot and release process

[Dale LaBossiere]

I just pulled down the latest PR contents into a new clone.  I get a bunch of 
warnings like the one below (but things work).

mvn clean install
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for 
org.apache.edgent.analytics:edgent-analytics-sensors:jar:1.2.0-SNAPSHOT
[WARNING] 'version' contains an expression but should be a constant. @ 
org.apache.edgent.analytics:edgent-analytics-sensors:${edgent.version}, 
/Users/dlaboss/git/incubator-edgen\
t-4/analytics/sensors/pom.xml, line 31, column 12

— Dale

Re: Understanding the snapshot and release process

[Dale LaBossiere]

Thx for the cleanup for Eclipse.

> On Jun 15, 2017, at 7:09 AM, Christofer Dutz  
> wrote:
> ...It seems as if some unit-tests tend to fail in random intervals. Right 
> now, I had 3 Travis runs without any functional changes and I got failures on 
> 3 different places. Is this “normal”? If yes … I think I should start 
> identifiying the “random failers” and investigate the reason.

What were the failing tests?

MqttOpenTest and Http[Global]Test use external public servers and can be 
impacted by them.  An attempt has been made to desensitize them (e.g., trying 
to connect and then skipping instead of failing tests) but sometimes things 
seem to leak through.

All of the tests have time limits imposed and will “fail" if they don’t 
complete in time (some have had more narrow limits than others).  I’ve seen 
these sort of failures on occasion such as when the test system is heavily 
loaded.

— Dale

Re: Understanding the snapshot and release process

[Christofer Dutz]

Just another question …

It seems as if some unit-tests tend to fail in random intervals. Right now, I 
had 3 Travis runs without any functional changes and I got failures on 3 
different places. Is this “normal”? If yes … I think I should start 
identifiying the “random failers” and investigate the reason.

Chris 

Am 15.06.17, 12:49 schrieb "Christofer Dutz" :

Your wish is my command ;-)

I just installed the latest Eclipse “Neon.3” in the Developer edition 
(which comes with Maven support).

Then I manually deleted all existing eclipse settings in the Edgent project 
as otherwise Eclipse would have insisted on using them instead. Also, I added 
them to the main gitignore file to prevent them from being tracked when 
re-created.

After that I imported my existing Maven project into my Eclipse workspace. 
It correctly detected all plugins I am using in the build and correctly setup 
the M2E plugins for these (glad I’m just using the usual suspects of maven 
plugins). In the end, I still got 3 errors complaining about the copy plugin 
not being able to copy because the jars not having been assembled yet. So, I 
moved all tests that require the loading of WAR or JAR files to the 
integration-test phase (where they belong anyway) and re-tried. Eclipse still 
had the same issues. In the end, I forced Eclipse to ignore the plugin 
execution. So, If you try to execute any of the “IT” suffixed tests in Eclipse, 
these will probably fail if you haven’t done a Maven build prior to test 
execution.

Right now, I have the project imported in Eclipse and am getting 0 errors. 
I tried running some Junit tests from Eclipse and all worked nicely :-)

So, I guess were coming close to the finishing line.

Chris



Am 14.06.17, 18:09 schrieb "Dale LaBossiere" :

Yup, just noticed it. Looking good!

> On Jun 14, 2017, at 11:36 AM, Christofer Dutz 
 wrote:
> 
> Ok, so I just pushed a major update.
> 
> On my machine with these changes both the java7 and java8 builds seem 
to be working. In the next few days I’ll probably play around with some of the 
excluded modules and see if we can include them in the java7 build. Right now, 
it sort of looks as if they were excluded simply because they didn’t 
immediately work. So I’ll go through them and try to find out which ones we can 
add and write down why some cant (This information seems to be missing at the 
moment)
I suggest deferring the “add more to java7” effort until everything 
else needed to reach the state of being able to develop and make a release with 
the new machinery is done.  Maybe deferring until after actually making a 
release with the new machinery - IMO we should do one asap.

> Another thing I did, was to add the RAT plugin, go through the 
exclusions and clean up the exclusions. Some were excluding stuff that 
shouldn’t be excluded and it reported quite a hand full of files without ASF 
headers. I added them so please have a look if this was ok.
LGTM

> 
> The Eclipse support is also on my list (even if I have to admit that 
I always hate having to open that IDE ;-) )
:-) the sooner the better so that more folks can pull in the PR to 
experiment.

Thanks!
— Dale

Re: Understanding the snapshot and release process

[Christofer Dutz]

Your wish is my command ;-)

I just installed the latest Eclipse “Neon.3” in the Developer edition (which 
comes with Maven support).

Then I manually deleted all existing eclipse settings in the Edgent project as 
otherwise Eclipse would have insisted on using them instead. Also, I added them 
to the main gitignore file to prevent them from being tracked when re-created.

After that I imported my existing Maven project into my Eclipse workspace. It 
correctly detected all plugins I am using in the build and correctly setup the 
M2E plugins for these (glad I’m just using the usual suspects of maven 
plugins). In the end, I still got 3 errors complaining about the copy plugin 
not being able to copy because the jars not having been assembled yet. So, I 
moved all tests that require the loading of WAR or JAR files to the 
integration-test phase (where they belong anyway) and re-tried. Eclipse still 
had the same issues. In the end, I forced Eclipse to ignore the plugin 
execution. So, If you try to execute any of the “IT” suffixed tests in Eclipse, 
these will probably fail if you haven’t done a Maven build prior to test 
execution.

Right now, I have the project imported in Eclipse and am getting 0 errors. I 
tried running some Junit tests from Eclipse and all worked nicely :-)

So, I guess were coming close to the finishing line.

Chris



Am 14.06.17, 18:09 schrieb "Dale LaBossiere" :

Yup, just noticed it. Looking good!

> On Jun 14, 2017, at 11:36 AM, Christofer Dutz  
wrote:
> 
> Ok, so I just pushed a major update.
> 
> On my machine with these changes both the java7 and java8 builds seem to 
be working. In the next few days I’ll probably play around with some of the 
excluded modules and see if we can include them in the java7 build. Right now, 
it sort of looks as if they were excluded simply because they didn’t 
immediately work. So I’ll go through them and try to find out which ones we can 
add and write down why some cant (This information seems to be missing at the 
moment)
I suggest deferring the “add more to java7” effort until everything else 
needed to reach the state of being able to develop and make a release with the 
new machinery is done.  Maybe deferring until after actually making a release 
with the new machinery - IMO we should do one asap.

> Another thing I did, was to add the RAT plugin, go through the exclusions 
and clean up the exclusions. Some were excluding stuff that shouldn’t be 
excluded and it reported quite a hand full of files without ASF headers. I 
added them so please have a look if this was ok.
LGTM

> 
> The Eclipse support is also on my list (even if I have to admit that I 
always hate having to open that IDE ;-) )
:-) the sooner the better so that more folks can pull in the PR to 
experiment.

Thanks!
— Dale

Re: Understanding the snapshot and release process

[Dale LaBossiere]

Yup, just noticed it. Looking good!

> On Jun 14, 2017, at 11:36 AM, Christofer Dutz  
> wrote:
> 
> Ok, so I just pushed a major update.
> 
> On my machine with these changes both the java7 and java8 builds seem to be 
> working. In the next few days I’ll probably play around with some of the 
> excluded modules and see if we can include them in the java7 build. Right 
> now, it sort of looks as if they were excluded simply because they didn’t 
> immediately work. So I’ll go through them and try to find out which ones we 
> can add and write down why some cant (This information seems to be missing at 
> the moment)
I suggest deferring the “add more to java7” effort until everything else needed 
to reach the state of being able to develop and make a release with the new 
machinery is done.  Maybe deferring until after actually making a release with 
the new machinery - IMO we should do one asap.

> Another thing I did, was to add the RAT plugin, go through the exclusions and 
> clean up the exclusions. Some were excluding stuff that shouldn’t be excluded 
> and it reported quite a hand full of files without ASF headers. I added them 
> so please have a look if this was ok.
LGTM

> 
> The Eclipse support is also on my list (even if I have to admit that I always 
> hate having to open that IDE ;-) )
:-) the sooner the better so that more folks can pull in the PR to experiment.

Thanks!
— Dale

Re: Understanding the snapshot and release process

[Christofer Dutz]

Ok, so I just pushed a major update.

On my machine with these changes both the java7 and java8 builds seem to be 
working. In the next few days I’ll probably play around with some of the 
excluded modules and see if we can include them in the java7 build. Right now, 
it sort of looks as if they were excluded simply because they didn’t 
immediately work. So I’ll go through them and try to find out which ones we can 
add and write down why some cant (This information seems to be missing at the 
moment)

Another thing I did, was to add the RAT plugin, go through the exclusions and 
clean up the exclusions. Some were excluding stuff that shouldn’t be excluded 
and it reported quite a hand full of files without ASF headers. I added them so 
please have a look if this was ok.

The Eclipse support is also on my list (even if I have to admit that I always 
hate having to open that IDE ;-) )

Chris

Am 13.06.17, 17:45 schrieb "Dale LaBossiere" :


> On Jun 12, 2017, at 4:56 PM, Christofer Dutz  
wrote:
> ...
> In my opinion IDE settings shouldn’t be checked in and should rather be 
added to the gitignore list. Don’t know how often we had issues in Flex with 
checked in IDE settings :-(
Agreed, with maven projects/pom’s in place, the Eclipse IDE 
settings/.project/.classpath would be removed from the repo.

>  
> The warnings are related to the fact that all Apache Maven configurations 
have the “Apache root pom” as top-most parent. A lot Some things are configured 
there. However I’m using different versions of some plugins and re-defining 
them in the edgent-parent. That’s what the first block of warnings are about.
Since the overrides are intentional, can you add the “annotations” to 
eliminate the warnings?  (and maybe a comment as to why a different version is 
being selected?) 
https://stackoverflow.com/questions/30782453/maven-suppress-overriding-managed-version-warning-in-eclipse
 


>  
> The rest of the blocks are because in order to work correctly the e2m 
plugin has built-in configurations on what to do for which default maven 
plugin. This is how Eclipse knows when to copy resources, compile classes and 
eventually even generate code. However per default not all plugins are handled 
by the m2e plugin. That’s what the second block is about. There is a way to get 
rid of them and eventually even tell Eclipse what to do, but this would need 
some fine tuning. I would like to address that as soon as we’re finished. 
Otherwise I would probably have to re-configure that over and over again. Mabe 
it would be good to tell Eclipse to ignore the plugins it doesn’t know so the 
errors go away and then to fine tune them afterwards.
Dealing with the warnings later seems OK.  Can you make the change to 
eliminate the errors now?  You probably know what needs to be done but fwiw 
here’s what Eclipse quick-fix added:
@dales-mbp:1049> diff test/fvtiot/pom.xml{,.sav}
62,96d61
< 
<   
<   
<   
<   org.eclipse.m2e
<   lifecycle-mapping
<   1.0.0
<   
<   
<   
<   
<   

<   
<   
org.apache.maven.plugins
<   
<   
<   
maven-dependency-plugin
<   

<   

<   
[3.0.1,)
<   

<   
<   
copy
<   
<   

<   
<   

<   
<   
<   
<   
<   
<   
<   
< 
@dales-mbp:1050> 

Thanks,
— Dale

Re: Understanding the snapshot and release process

[Dale LaBossiere]

> On Jun 12, 2017, at 4:56 PM, Christofer Dutz  
> wrote:
> ...
> In my opinion IDE settings shouldn’t be checked in and should rather be added 
> to the gitignore list. Don’t know how often we had issues in Flex with 
> checked in IDE settings :-(
Agreed, with maven projects/pom’s in place, the Eclipse IDE 
settings/.project/.classpath would be removed from the repo.

>  
> The warnings are related to the fact that all Apache Maven configurations 
> have the “Apache root pom” as top-most parent. A lot Some things are 
> configured there. However I’m using different versions of some plugins and 
> re-defining them in the edgent-parent. That’s what the first block of 
> warnings are about.
Since the overrides are intentional, can you add the “annotations” to eliminate 
the warnings?  (and maybe a comment as to why a different version is being 
selected?) 
https://stackoverflow.com/questions/30782453/maven-suppress-overriding-managed-version-warning-in-eclipse
 


>  
> The rest of the blocks are because in order to work correctly the e2m plugin 
> has built-in configurations on what to do for which default maven plugin. 
> This is how Eclipse knows when to copy resources, compile classes and 
> eventually even generate code. However per default not all plugins are 
> handled by the m2e plugin. That’s what the second block is about. There is a 
> way to get rid of them and eventually even tell Eclipse what to do, but this 
> would need some fine tuning. I would like to address that as soon as we’re 
> finished. Otherwise I would probably have to re-configure that over and over 
> again. Mabe it would be good to tell Eclipse to ignore the plugins it doesn’t 
> know so the errors go away and then to fine tune them afterwards.
Dealing with the warnings later seems OK.  Can you make the change to eliminate 
the errors now?  You probably know what needs to be done but fwiw here’s what 
Eclipse quick-fix added:
@dales-mbp:1049> diff test/fvtiot/pom.xml{,.sav}
62,96d61
< 
<   
<   
<   
<   org.eclipse.m2e
<   lifecycle-mapping
<   1.0.0
<   
<   
<   
<   
<   
<   
<   
org.apache.maven.plugins
<   
<   
<   
maven-dependency-plugin
<   
<   
<   [3.0.1,)
<   
<   
<   
copy
<   
<   
<   
<   

<   
<   
<   
<   
<   
<   
<   
< 
@dales-mbp:1050> 

Thanks,
— Dale

Re: Understanding the snapshot and release process

[Dale LaBossiere]

> On Jun 13, 2017, at 10:16 AM, Christofer Dutz  
> wrote:
> ...
> Right now, I am still having issues running all Tests in the java7 version … 
> math3 is sort of causing issues, I haven’t quite figured out why.


Edgent’s java7 platform support doesn’t include everything in the java8 
platform.  See 
https://github.com/apache/incubator-edgent/blob/master/JAVA_SUPPORT.md

For the most part I’m not quite sure why that is.  In some cases I think it was 
just “we’ll confirm module/project xyz works on java7 when there’s a 
demonstrated demand for it.”

I see math3 is one such module, hence it hasn’t been getting included/tested in 
the java7 config.  See the commentary in 
https://github.com/apache/incubator-edgent/blob/master/platform/java7/build.xml#L118
 


The same is true for the android platform support, though apparently JMX isn’t 
supported on android so that explains some of its exclusions.  See 
https://github.com/apache/incubator-edgent/blob/master/platform/android/build.xml#L41
 


The net is that some conditionalization of what’s included in java7 and android 
is needed.

— Dale

Re: Understanding the snapshot and release process

[Christofer Dutz]

Aaahh … well that’s a different thing … that’s already possible with maven :-)

Right now, all Apache releases go to a so-called release-repository. That’s a 
Maven repository containing only the artifacts of the current release. The 
project can vote on this. If the vote doesn’t pass, the repo is simply dropped 
and it was as if nothing had happened. If the release passes, the artifacts are 
released to the Apache Maven Release Repo and this is automatically mirrored to 
Maven Central. 

So, it seems this part is already taken care of … one more thing to check off 
the list ;-)

I guess Travis was chosen, because Github was selected as primary repo for the 
project and the integration with auto-built pull requests is great, I have to 
admit that. But it is problematic to setup auto deployment of SNAPSHOT versions 
as we would have to check in credentials for deploying to the Apache Maven repo 
and that’s not going to happen. As the build setup is quite trivial I think 
using both is the best option. Use Travis for the auto-testing of 
pull-requests. Use Apache Jenkins for deploying SNAPSHOTS and doing 
code-analysis on Sonarqube and stuff like that. 

Right now, I am still having issues running all Tests in the java7 version … 
math3 is sort of causing issues, I haven’t quite figured out why.

Chris



Am 13.06.17, 15:59 schrieb "Dale LaBossiere" :


> On Jun 12, 2017, at 5:04 PM, Christofer Dutz  
wrote:
> ...
> I guess also executing the pre-built test with java7 should also be 
possible, but I’m struggling with one other of your requirements:
> “Only deploy everything if the java8, java7 and android builds succeed”. 
I hope you can see the dilemma.
It’s possible my terminology is confusing/inaccurate.  Mostly I think I 
meant that for releases (not talking about snapshots nor release-candidate 
staging for voting) we should only make the final release artifacts, including 
convenience binaries, public only if/once all configurations have passed 
testing

I think it’s time for me to go back to all the ASF release doc info.  I 
know a lot has changed and earlier I didn’t pay any attention to snapshots nor 
maven related things because we weren’t doing them.

> Here it might be a better option to add a Jenkinsfile to the repository 
and setup a Jenkins job on the ASF Jenkins. With this I am a lot more flexible 
than with Travis. We could continue to use Travis for the auto-testing of pull 
requests, but would live with running the java7 testsuite with a java8 VM, but 
have the Jenkins job deploy the SNAPSHOTS to the ASF repo.
> 
> What do you think about that? Building on Apache Machines also gives a 
lot of other benefits.
I don’t know why Travis was chosen.  Is the ASF Jenkins service relatively 
new or not good/viable for PR-auto-test use?  Regardless, partially or fully 
switching to Jenkins tooling as appropriate is OK with me - unless there are 
some things you forgot to mention :-)

— Dale

Re: Understanding the snapshot and release process

[Dale LaBossiere]

> On Jun 12, 2017, at 5:04 PM, Christofer Dutz  
> wrote:
> ...
> I guess also executing the pre-built test with java7 should also be possible, 
> but I’m struggling with one other of your requirements:
> “Only deploy everything if the java8, java7 and android builds succeed”. I 
> hope you can see the dilemma.
It’s possible my terminology is confusing/inaccurate.  Mostly I think I meant 
that for releases (not talking about snapshots nor release-candidate staging 
for voting) we should only make the final release artifacts, including 
convenience binaries, public only if/once all configurations have passed 
testing

I think it’s time for me to go back to all the ASF release doc info.  I know a 
lot has changed and earlier I didn’t pay any attention to snapshots nor maven 
related things because we weren’t doing them.

> Here it might be a better option to add a Jenkinsfile to the repository and 
> setup a Jenkins job on the ASF Jenkins. With this I am a lot more flexible 
> than with Travis. We could continue to use Travis for the auto-testing of 
> pull requests, but would live with running the java7 testsuite with a java8 
> VM, but have the Jenkins job deploy the SNAPSHOTS to the ASF repo.
> 
> What do you think about that? Building on Apache Machines also gives a lot of 
> other benefits.
I don’t know why Travis was chosen.  Is the ASF Jenkins service relatively new 
or not good/viable for PR-auto-test use?  Regardless, partially or fully 
switching to Jenkins tooling as appropriate is OK with me - unless there are 
some things you forgot to mention :-)

— Dale

Re: Understanding the snapshot and release process

[Christofer Dutz]

Hi Dale,

Glad you like that … I was going to address the “checked in IDE settings” thing 
soon ;-)

I am working with IntelliJ without any issues in my Maven setup. Had to rename 
the Gradle build to make IntelliJ chose Maven though.

In my opinion IDE settings shouldn’t be checked in and should rather be added 
to the gitignore list. Don’t know how often we had issues in Flex with checked 
in IDE settings :-(

The warnings are related to the fact that all Apache Maven configurations have 
the “Apache root pom” as top-most parent. A lot Some things are configured 
there. However I’m using different versions of some plugins and re-defining 
them in the edgent-parent. That’s what the first block of warnings are about.

The rest of the blocks are because in order to work correctly the e2m plugin 
has built-in configurations on what to do for which default maven plugin. This 
is how Eclipse knows when to copy resources, compile classes and eventually 
even generate code. However per default not all plugins are handled by the m2e 
plugin. That’s what the second block is about. There is a way to get rid of 
them and eventually even tell Eclipse what to do, but this would need some fine 
tuning. I would like to address that as soon as we’re finished. Otherwise I 
would probably have to re-configure that over and over again. Mabe it would be 
good to tell Eclipse to ignore the plugins it doesn’t know so the errors go 
away and then to fine tune them afterwards.

Chris



Von: Dale LaBossiere <dml.apa...@gmail.com>
Antworten an: "dev@edgent.apache.org" <dev@edgent.apache.org>
Datum: Montag, 12. Juni 2017 um 22:43
An: "dev@edgent.apache.org" <dev@edgent.apache.org>
Betreff: Re: Understanding the snapshot and release process

Chris, I started to try to get Eclipse working with the maven PR-309 changes.  
Making some progress.

At a high level, I removed all existing .project and .classpath Eclipse files 
from my local git clone.
Then I created a new Eclipse workspace and did an “Import existing Maven 
Projects”.
Eclipse then created a .project and .classpath from the poms.
Seemingly all good stuff.  And presumably analogous to importing a maven 
project into IntelliJ.

But I have 3 build problems and 66 (mostly pom.xml related) warnings.

What do you make of these warnings in the top-level pom.xml?   Googling seems 
to tell me that the “Overriding managed version” warnings are because the 
top-level pom.xml is overriding the versions for the plugins inherited from the 
parent pom.xml (org.apache:apache:18 ???) ???

[cid:image001.png@01D2E3CF.2BA7FA50]


There are a lot (every project/module I assume) of these:
[cid:image002.png@01D2E3CF.2BA7FA50]

Any thoughts on the 3 errors?
[cid:image003.png@01D2E3CF.2BA7FA50]
the full error msgs are:
[cid:image004.png@01D2E3CF.2BA7FA50]

— Dale

Re: Understanding the snapshot and release process

[Dale LaBossiere]

> On Jun 12, 2017, at 10:23 AM, Christofer Dutz  
> wrote:
> 
> Well “mvn test” would run the maven lifecycle up to the test phase, this 
> includes compiling etc … and compiling doesn’t work with Java7.
Maven doesn’t have the notion of build-avoidance / artifact reuse?
I can imagine that even if it did, it might record the JDK version used to 
generate the artifacts and then not reuse them when the current JDK version is 
different.

> We could probably create some solution to manually load and run the tests 
> inside the test-jars but integrating this into one nice Maven build could be 
> a challenging task, so say the least … but we wouldn’t have to, if you would 
> be ok with the signature-verification option. 
I’m holding out till I better understand maven’s behavior/limitations and what 
other strategies might exist :-)

> 
> Seems not, so I’ll continue digging … but this one will probably take a 
> little longer. 

What do other projects do if for example they build/release to a 
least-common-demonitor, e.g., 1.7, and still want to do actual test 
verification against the higher compatible/supported version, e.g., 1.8?  They 
don’t/can’t use standard maven machinery for that?  They must build some 
parallel test execution / reporting machinery?

— Dale

Re: Understanding the snapshot and release process

[Christofer Dutz]

Ok … so I’m working on the open issues … I do have one question though … 
Compiling with one java version and then running a different one with the tests 
will be challenging, if not impossible. 
What I did a while ago was to introduce the animal-sniffer plugin. This very 
strictly checks the resulting class files against the signatures for a given 
JDK to ensure the classes would work in that particular JDK. Would this be 
enough for you? It’s what we settled for in the Flex project. In the past we 
did have to re-release stuff because even if we compiled to output 1.7 code, 
the code wasn’t runnable on 1.7. Since introducing the animal-sniffer plugin we 
never had problems like this again. 

Chris


Am 10.06.17, 22:45 schrieb "Christofer Dutz" :

Hi Dale,

Thanks for that … I had noticed the merged pull request … today I pulled in 
the upstream changes and tweaked the build a little. 
Now both the java8 and java7 builds seem to be working. I also made the 
build compile to different target sub-directories so now you can inspect the 
output from both runs without having them interfering. 

Chris


Am 09.06.17, 22:29 schrieb "Dale LaBossiere" :

Hi Chris,  in case you hadn’t noticed the following merged PR (on 
master) eliminates the
use of “default”.  So you should be able to rebase your branch your 
branch and be all set.

https://github.com/apache/incubator-edgent/pull/310 


> On Jun 8, 2017, at 10:40 AM, Christofer Dutz 
 wrote:
> ...problem. The default methods in DirectTestSetup can’t seem to be 
backported if the interface and the implementing class are not in the same 
retrolambda execution. Eventually refactoring the tests to work without default 
methods would be a good idea.

— Dale

Re: Understanding the snapshot and release process

[Christofer Dutz]

Hi Dale,

Thanks for that … I had noticed the merged pull request … today I pulled in the 
upstream changes and tweaked the build a little. 
Now both the java8 and java7 builds seem to be working. I also made the build 
compile to different target sub-directories so now you can inspect the output 
from both runs without having them interfering. 

Chris


Am 09.06.17, 22:29 schrieb "Dale LaBossiere" :

Hi Chris,  in case you hadn’t noticed the following merged PR (on master) 
eliminates the
use of “default”.  So you should be able to rebase your branch your branch 
and be all set.

https://github.com/apache/incubator-edgent/pull/310 


> On Jun 8, 2017, at 10:40 AM, Christofer Dutz  
wrote:
> ...problem. The default methods in DirectTestSetup can’t seem to be 
backported if the interface and the implementing class are not in the same 
retrolambda execution. Eventually refactoring the tests to work without default 
methods would be a good idea.

— Dale

Re: Understanding the snapshot and release process

[Dale LaBossiere]

Hi Chris,  in case you hadn’t noticed the following merged PR (on master) 
eliminates the
use of “default”.  So you should be able to rebase your branch your branch and 
be all set.

https://github.com/apache/incubator-edgent/pull/310 


> On Jun 8, 2017, at 10:40 AM, Christofer Dutz  
> wrote:
> ...problem. The default methods in DirectTestSetup can’t seem to be 
> backported if the interface and the implementing class are not in the same 
> retrolambda execution. Eventually refactoring the tests to work without 
> default methods would be a good idea.

— Dale

Re: Understanding the snapshot and release process

[Dale LaBossiere]

sorry no idea about travis and memory

On Jun 8, 2017, at 10:40 AM, Christofer Dutz  wrote:
> ...
> Today I had some progress on the java7 side.
...
> problem. The default methods in DirectTestSetup can’t seem to be backported 
> if the interface and the implementing class are not in the same retrolambda 
> execution. Eventually refactoring the tests to work without default methods 
> would be a good idea.
I’ll look into that.

> The problem with classifiers is that maven automatically resolves artifacts. 
> Depending on the scope different artifacts are pulled in. In general, we 
> would be using the “compile” scope. So, if I add a dependency with classifier 
> “java7” then it would pull in that jar in the java7 version, but the 
> transitive dependencies would probably be the java8 ones, so you would have 
> to explicitly define all the transitive dependencies with the classifier. We 
> don’t want that ;-)

Thought I was following it but then not :-)  
I may have to try a test using classifier so as to really understand how/why it 
won’t work. 

But if encoding tgt-kind in the versionId is the way to go… (I think it would 
be nice if it could be avoided for tgt-java8, e.g., just 1.2.0, not 1.2.0.java8)

In travis I don’t think you’ll be able to just/only say
mvn clean deploy# compile/test/deploy java8
mvn clean deploy -Pjava7# compile/test/deploy java7

That’s because the compile for java7, including the tests, must be done with a 
java8 JDK (with retrolambda then applied).  There’s no java7 JDK in the mix for 
java7 compile.

But to really test the compiled java7-compatible tests one has to first switch 
the environment to a java7 JRE and then run them in that environment (they 
can’t be compiled in a java7 env because they use lambdas).

That’s why the travis / gradle/ant stuff was structured out the way it was:
switch to java8
gradlew release   # compiles Edgent into jars for 
java8,7,android.
# NOTE, tests are not bundled into 
jars. So there are just the j8 class files
gradlew test # runs the already compiled Edgent tests 
(with java8)
gradlew test7Compile  # recompile the tests-only to be java7 compatible
switch to java7
gradlew test7Run# runs the already java7-compatible compiled 
Edgent tests (with java7)

Also at a high level I think we don’t want to deploy one config unless all 
configs are OK.

— Dale

Re: Understanding the snapshot and release process

[Dale LaBossiere]

> On Jun 8, 2017, at 10:43 AM, Christofer Dutz  
> wrote:
> 
> Eventually your approach with a sub-directory inside the target would be a 
> valid option … would probably have to adjust the maven-clean-plugin too, but 
> that should work in getting all results inside the target directory in 
> parallel.

maven-clean-plugin seemed to work OK.  Hmm… would there be a way to get a 
“clean all”?  Maybe that’s what you meant.

mvn clean -P tgt-java7  # cleans all target/java7
mvn clean # cleans all target/java8

That was with pom.xml changes:

  
java8
...
  

  

  tgt-java7
  

  tgt
  java7

  
  
java7
  
...


  
target/${edgent.target.kind}
...
  

Re: Understanding the snapshot and release process

[Christofer Dutz]

Hi Dale,

Today I had some progress on the java7 side.
I successfully added the retrolambda-maven-plugin in a dedicated java7 profile 
to the main pom. Now the plugin automatically kicks in and converts the classes 
after compiling them. I also added the animal-sniffer-maven-plugin to 
explicitly check against the signatures of 1.7 or 1.8 JDKs. This all seems to 
be working. Unfortunately, the backporting of default methods in interfaces 
seems to be a problem. The default methods in DirectTestSetup can’t seem to be 
backported if the interface and the implementing class are not in the same 
retrolambda execution. Eventually refactoring the tests to work without default 
methods would be a good idea.

The problem with classifiers is that maven automatically resolves artifacts. 
Depending on the scope different artifacts are pulled in. In general, we would 
be using the “compile” scope. So, if I add a dependency with classifier “java7” 
then it would pull in that jar in the java7 version, but the transitive 
dependencies would probably be the java8 ones, so you would have to explicitly 
define all the transitive dependencies with the classifier. We don’t want that 
;-)

I also contacted the maven guys on their mailing list. They did have the one or 
the other solution. In general, they would all come down to adding the type to 
the version. So, we would start producing “1.2.0.java7” and “1.2.0.java8” or 
“1.2.0.7” and “1.2.0.8” versions (I would prefer the first as it doesn’t 
suggest the java 8 version being slightly newer than the 7 version). Another 
alternative would be to prefix the version with the target version. “7.1.2.0” 
and “8.1.2.0” … but I don’t really like that option. In this case, I would have 
to refactor the build to use variables in all modules except the parent poms 
(you can’t use variables for parent poms). Then travis/Jenkins would run two 
build:

mvn clean deploy
mvn clean deploy –Pjava7

In the end, it would result in both versions being built, tested and deployed, 
but in the end the java7 version would remain in the workspace as the second 
executions “clean” removed any reference to the java8 version.

Hope that explains the dilemma a little more. At least with these 
version-suffixes we would have a fully working solution. I’ll get into changing 
this tomorrow.

By the way … anyone got an idea how I can make Travis not die on out of memory? 
Tried several things, but it didn’t seem to do the job.

Chris




Am 08.06.17, 16:13 schrieb "Dale LaBossiere" :


> On Jun 8, 2017, at 4:19 AM, Christofer Dutz  
wrote:
> 
> Regarding java8 and java7 artifacts …

Coincidentally, I started investigating this space yesterday and 
retrolambda-maven-plugin
and was just composing mail on the question of publishing java8 and java7 
to mvn repos :-)

[ fwiw, so far I had defined a Profile (e.g., mvn install -Ptgt=java7) and
changed the “target” build directory to “target/${edgent.target.kind}”
(e.g., target/{java8,java7}).  Things were working fine up to that point:
generated class files and module jars. ]

> This could be problematic, if we wanted to publish both.
> ...
> I could offer to generally build java7 versions and to test them with 
java7 and java8 … I think this topic should be discussed a little more before I 
start implementing this.
> 
> Right now, I would opt for distributing java8 binaries to Maven Central 
and to offer downloadable java7 binary distributions.

Ugh.  Publishing java8 in a mvn repo but java7/android as a binary tgz 
seems really bad.
It creates two different ways (and needed doc) for users building an Edgent 
app
and installing Edgent jars onto a “device”.

Could you elaborate on the problems using the maven coordinate “classifier”?
The maven doc [1] has this exactly as use case for it (they use “jdk15” vs 
“jdk14”).
At a high (ignorant) level I don’t have a problem with the classifier being 
nil for java8 :-)

If using a coordinate classifier is a no-go, what about (grimace) 
incorporating it into the
groupId:  org.apache.edgent, org.apache.edgent.java7  ?

A bit related, what coordinate component is “incubating” supposed to be 
incorporated into?

— Dale

[1] https://maven.apache.org/pom.html 

Re: Understanding the snapshot and release process

[Dale LaBossiere]

> On Jun 8, 2017, at 4:19 AM, Christofer Dutz  wrote:
> 
> Regarding java8 and java7 artifacts …

Coincidentally, I started investigating this space yesterday and 
retrolambda-maven-plugin
and was just composing mail on the question of publishing java8 and java7 to 
mvn repos :-)

[ fwiw, so far I had defined a Profile (e.g., mvn install -Ptgt=java7) and
changed the “target” build directory to “target/${edgent.target.kind}”
(e.g., target/{java8,java7}).  Things were working fine up to that point:
generated class files and module jars. ]

> This could be problematic, if we wanted to publish both.
> ...
> I could offer to generally build java7 versions and to test them with java7 
> and java8 … I think this topic should be discussed a little more before I 
> start implementing this.
> 
> Right now, I would opt for distributing java8 binaries to Maven Central and 
> to offer downloadable java7 binary distributions.

Ugh.  Publishing java8 in a mvn repo but java7/android as a binary tgz seems 
really bad.
It creates two different ways (and needed doc) for users building an Edgent app
and installing Edgent jars onto a “device”.

Could you elaborate on the problems using the maven coordinate “classifier”?
The maven doc [1] has this exactly as use case for it (they use “jdk15” vs 
“jdk14”).
At a high (ignorant) level I don’t have a problem with the classifier being nil 
for java8 :-)

If using a coordinate classifier is a no-go, what about (grimace) incorporating 
it into the
groupId:  org.apache.edgent, org.apache.edgent.java7  ?

A bit related, what coordinate component is “incubating” supposed to be 
incorporated into?

— Dale

[1] https://maven.apache.org/pom.html 

Re: Understanding the snapshot and release process

[Christofer Dutz]

Regarding java8 and java7 artifacts …

This could be problematic, if we wanted to publish both.

Having a binary tar.gz with a java8 and one with java7 doesn’t seem to be a 
problem, but having two artifacts deployed to Maven-Central is no trivial task.
Exactly this type of jars for different java versions is one of the things the 
Maven guys are currently working on as part of the Java 9 – Jigsaw support. But 
it’s a nightmare. We could start creating java8 jars as default and add a 
second “java7” jar using the “java7” classifier, but then a lot of the 
mechanisms in Maven no longer work. 

I could offer to generally build java7 versions and to test them with java7 and 
java8 … I think this topic should be discussed a little more before I start 
implementing this.

Right now, I would opt for distributing java8 binaries to Maven Central and to 
offer downloadable java7 binary distributions.

Chris



Am 07.06.17, 21:06 schrieb "Christofer Dutz" <christofer.d...@c-ware.de>:

Hi,

Right now I didn't configure anything regarding the assembly of the source 
artifact. It's apaches default config. I'll look into a more fine-tuned version.

And regarding the artifact name ... Justin suggested that change, but 90% 
of the other projects don't do it that way. I was waiting for your opinions 
before doing that. It does make the dependencies a lot more verbose. After all 
you can't use the artifacts without the groupId "org.apache.effect" which 
includes "apache" and "edgent".

But sure, I'll change that if you all want me to.

Chris

Von meinem Samsung Galaxy Smartphone gesendet.


 Ursprüngliche Nachricht 
Von: Dale LaBossiere <dml.apa...@gmail.com>
Datum: 07.06.17 18:45 (GMT+01:00)
An: dev@edgent.apache.org
    Betreff: Re: Understanding the snapshot and release process

OK, so mvn install -Papache-release generates the source release bundle zip.

I poked around for a while but couldn’t come up with the incantation
needed to start excluding things from the zip.  e.g., the zip includes
all of the residual gradle generated **/build/

Chris, can you enhance the configs so as to exclude the various
things that the gradle build was configured to exclude?
See top level build.gradle/srcReleaseTarGz task.
Also exclude src/site (that present in the zip too), right?

The generated zip’s name is:
  target/edgent-parent-1.2.0-SNAPSHOT-source-release.zip

“edgent-parent” ?  today we, I thought correctly, have “apache-edgent” 
there.
Also, the required “incubating” is missing in the release bundle name.

Thanks,
— Dale

> On Jun 7, 2017, at 10:35 AM, Christofer Dutz <christofer.d...@c-ware.de> 
wrote:
>
> Checking our repo:
> https://repository.apache.org/#view-repositories;releases~browsestorage
>
> I can see that fewer than about 10% of all apache projects do that … But 
I wouldn’t mind adding that, if the others agree.
>
> Chris
>
>
>
> Am 07.06.17, 14:54 schrieb "Justin Mclean" <jus...@classsoftware.com>:
>
>Hi,
>
>While not a requirement prefixing with apache may add some legal 
protection and is good from a branding point of of view.
>
>Thanks,
>Justin
>

Re: Understanding the snapshot and release process

[Christofer Dutz]

Hi,

Right now I didn't configure anything regarding the assembly of the source 
artifact. It's apaches default config. I'll look into a more fine-tuned version.

And regarding the artifact name ... Justin suggested that change, but 90% of 
the other projects don't do it that way. I was waiting for your opinions before 
doing that. It does make the dependencies a lot more verbose. After all you 
can't use the artifacts without the groupId "org.apache.effect" which includes 
"apache" and "edgent".

But sure, I'll change that if you all want me to.

Chris

Von meinem Samsung Galaxy Smartphone gesendet.


 Ursprüngliche Nachricht 
Von: Dale LaBossiere <dml.apa...@gmail.com>
Datum: 07.06.17 18:45 (GMT+01:00)
An: dev@edgent.apache.org
Betreff: Re: Understanding the snapshot and release process

OK, so mvn install -Papache-release generates the source release bundle zip.

I poked around for a while but couldn’t come up with the incantation
needed to start excluding things from the zip.  e.g., the zip includes
all of the residual gradle generated **/build/

Chris, can you enhance the configs so as to exclude the various
things that the gradle build was configured to exclude?
See top level build.gradle/srcReleaseTarGz task.
Also exclude src/site (that present in the zip too), right?

The generated zip’s name is:
  target/edgent-parent-1.2.0-SNAPSHOT-source-release.zip

“edgent-parent” ?  today we, I thought correctly, have “apache-edgent” there.
Also, the required “incubating” is missing in the release bundle name.

Thanks,
— Dale

> On Jun 7, 2017, at 10:35 AM, Christofer Dutz <christofer.d...@c-ware.de> 
> wrote:
>
> Checking our repo:
> https://repository.apache.org/#view-repositories;releases~browsestorage
>
> I can see that fewer than about 10% of all apache projects do that … But I 
> wouldn’t mind adding that, if the others agree.
>
> Chris
>
>
>
> Am 07.06.17, 14:54 schrieb "Justin Mclean" <jus...@classsoftware.com>:
>
>Hi,
>
>While not a requirement prefixing with apache may add some legal 
> protection and is good from a branding point of of view.
>
>Thanks,
>Justin
>

Re: Understanding the snapshot and release process

[Christofer Dutz]

No worries,

As long as you didn't do a "deploy" nothing should be uploaded :-)

Chris



Von meinem Samsung Galaxy Smartphone gesendet.


 Ursprüngliche Nachricht 
Von: Dale LaBossiere <dml.apa...@gmail.com>
Datum: 07.06.17 19:35 (GMT+01:00)
An: dev@edgent.apache.org
Betreff: Re: Understanding the snapshot and release process

install -Papache-release is generating the (arc) signature file :-)

Chris, what needs to be done to get it to also generate the .md5 and .sha files?

Also, did I now just populate the ASF snapshot repo? :-)

— Dale

Re: Understanding the snapshot and release process

[Dale LaBossiere]

install -Papache-release is generating the (arc) signature file :-)

Chris, what needs to be done to get it to also generate the .md5 and .sha files?

Also, did I now just populate the ASF snapshot repo? :-)

— Dale

Re: Understanding the snapshot and release process

[Dale LaBossiere]

OK, so mvn install -Papache-release generates the source release bundle zip.

I poked around for a while but couldn’t come up with the incantation
needed to start excluding things from the zip.  e.g., the zip includes
all of the residual gradle generated **/build/

Chris, can you enhance the configs so as to exclude the various
things that the gradle build was configured to exclude?
See top level build.gradle/srcReleaseTarGz task.
Also exclude src/site (that present in the zip too), right?

The generated zip’s name is:
  target/edgent-parent-1.2.0-SNAPSHOT-source-release.zip

“edgent-parent” ?  today we, I thought correctly, have “apache-edgent” there.
Also, the required “incubating” is missing in the release bundle name.

Thanks,
— Dale

> On Jun 7, 2017, at 10:35 AM, Christofer Dutz  
> wrote:
> 
> Checking our repo:
> https://repository.apache.org/#view-repositories;releases~browsestorage
> 
> I can see that fewer than about 10% of all apache projects do that … But I 
> wouldn’t mind adding that, if the others agree.
> 
> Chris
> 
> 
> 
> Am 07.06.17, 14:54 schrieb "Justin Mclean" :
> 
>Hi,
> 
>While not a requirement prefixing with apache may add some legal 
> protection and is good from a branding point of of view.
> 
>Thanks,
>Justin
> 

Re: Understanding the snapshot and release process

[Christofer Dutz]

Checking our repo:
https://repository.apache.org/#view-repositories;releases~browsestorage

I can see that fewer than about 10% of all apache projects do that … But I 
wouldn’t mind adding that, if the others agree.

Chris



Am 07.06.17, 14:54 schrieb "Justin Mclean" :

Hi,

While not a requirement prefixing with apache may add some legal protection 
and is good from a branding point of of view.

Thanks,
Justin

Re: Understanding the snapshot and release process

[Justin Mclean]

Hi,

While not a requirement prefixing with apache may add some legal protection and 
is good from a branding point of of view.

Thanks,
Justin

Re: Understanding the snapshot and release process

[Christofer Dutz]

Well looking through the Apache repo there is no 100% standard, but most 
artifacts are indeed prefixed with the project name but not apache … I’ll add 
the edgent to each of the artifacts.

Chris

Am 07.06.17, 13:47 schrieb "John D. Ament" :

Usually the maven norm is something like
com.mycompany.myproduct:myproduct-some-module.

So I would expect to see groupIds of org.apache.edgent, and artifact id's
of edgent-some-module.  However, I've noticed that both Eclipse and IBM
(obviously not related in any way) seem to use completely different
artifact formats.  I would recommend being an Apache project you adopt the
Apache Maven standard format.

John

On Wed, Jun 7, 2017 at 7:44 AM Christofer Dutz 
wrote:

> Hi,
>
> I’ll answer this one quickly as I can do that without digging in too deep.
>
> I wouldn’t add the apache-edgent to the artifact names as the groupId
> already qualifies them as apache edgent modules.
> However when creating a release zip/tar.gz with the sources or binary
> distribution I would add the “apache-edgent-” prefix.
>
> Chris
>
>
> Am 06.06.17, 23:02 schrieb "Dale LaBossiere" :
>
> Currently the generated JAR/WAR lack an “edgent” or “apache-edgent”
> prefix, e.g., "api-topology-1.2.0-SNAPSHOT.jar".  How important/required 
is
> it to add that before we start publishing these to mvn repos?
>
> What’s the current ASF best-practice in this regard?  e.g., I some
> (limited) # of “apache-XYZZY-…” in maven central.  Presumably there are
> many many others that omit the “apache-" - e.g., “beam-sdks-…”
>
> — Dale
>
>
>
>

Re: Understanding the snapshot and release process

[John D. Ament]

Usually the maven norm is something like
com.mycompany.myproduct:myproduct-some-module.

So I would expect to see groupIds of org.apache.edgent, and artifact id's
of edgent-some-module.  However, I've noticed that both Eclipse and IBM
(obviously not related in any way) seem to use completely different
artifact formats.  I would recommend being an Apache project you adopt the
Apache Maven standard format.

John

On Wed, Jun 7, 2017 at 7:44 AM Christofer Dutz 
wrote:

> Hi,
>
> I’ll answer this one quickly as I can do that without digging in too deep.
>
> I wouldn’t add the apache-edgent to the artifact names as the groupId
> already qualifies them as apache edgent modules.
> However when creating a release zip/tar.gz with the sources or binary
> distribution I would add the “apache-edgent-” prefix.
>
> Chris
>
>
> Am 06.06.17, 23:02 schrieb "Dale LaBossiere" :
>
> Currently the generated JAR/WAR lack an “edgent” or “apache-edgent”
> prefix, e.g., "api-topology-1.2.0-SNAPSHOT.jar".  How important/required is
> it to add that before we start publishing these to mvn repos?
>
> What’s the current ASF best-practice in this regard?  e.g., I some
> (limited) # of “apache-XYZZY-…” in maven central.  Presumably there are
> many many others that omit the “apache-" - e.g., “beam-sdks-…”
>
> — Dale
>
>
>
>

Re: Understanding the snapshot and release process

[Christofer Dutz]

Hi,

I’ll answer this one quickly as I can do that without digging in too deep.

I wouldn’t add the apache-edgent to the artifact names as the groupId already 
qualifies them as apache edgent modules.
However when creating a release zip/tar.gz with the sources or binary 
distribution I would add the “apache-edgent-” prefix.

Chris


Am 06.06.17, 23:02 schrieb "Dale LaBossiere" :

Currently the generated JAR/WAR lack an “edgent” or “apache-edgent” prefix, 
e.g., "api-topology-1.2.0-SNAPSHOT.jar".  How important/required is it to add 
that before we start publishing these to mvn repos?

What’s the current ASF best-practice in this regard?  e.g., I some 
(limited) # of “apache-XYZZY-…” in maven central.  Presumably there are many 
many others that omit the “apache-" - e.g., “beam-sdks-…”

— Dale

Re: Understanding the snapshot and release process

[Dale LaBossiere]

Currently the generated JAR/WAR lack an “edgent” or “apache-edgent” prefix, 
e.g., "api-topology-1.2.0-SNAPSHOT.jar".  How important/required is it to add 
that before we start publishing these to mvn repos?

What’s the current ASF best-practice in this regard?  e.g., I some (limited) # 
of “apache-XYZZY-…” in maven central.  Presumably there are many many others 
that omit the “apache-" - e.g., “beam-sdks-…”

— Dale

Re: Understanding the snapshot and release process

[Dale LaBossiere]

Hi Chris, 

I was trying the different forms of test execution noted in [1].
I’ve been mucking with my (Eclipse) .classpath files but that shouldn’t affect 
the following :-)

I tried running just the jdbc connector tests.

gradlew connectors:jdbc:test   # does everything needed to run the “test” task 
for just this component

mvn -pl components-jdbc test  # from [1]
[ERROR] Could not find the selected project in the reactor: connectors-jdbc @
…

that was when run in the top level directory.
also tried it after ‘cd connectors/jdbc’ and got the same results.

What am I doing wrong?
— Dale

[1] https://cwiki.apache.org/confluence/display/EDGENT/Maven+vs+Gradle

Re: Maven wiki page [was Re: Understanding the snapshot and release process]

[Christofer Dutz]

Hi Queeniw,

thanks … I added most of the things in the upper section. 

For the questions in the lower one, I would like to ask you to watch the little 
maven introduction video I did. It should answer a lot of questions. Questions 
that my answers would probably sound confusing without this minimal Maven 
background.

Chris

Am 05.06.17, 20:39 schrieb "Queenie Ma" :

Hi Chris, I just granted you permissions to make changes to the wiki.

—
Queenie

> On Monday, Jun 05, 2017 at 11:37 AM, Christofer Dutz 
 wrote:
> I’d love to, but you would need to give me the charma to do so (
>
> Chris
>
> Am 05.06.17, 20:14 schrieb "Dale LaBossiere" :
>
> I created a wiki page [1] to help us work through / collaborate on this.
> IMO there are too many questions, etc to effectively deal with it just on 
the dev@ list
>
> Chris, maybe you could fill in some of the (red) blanks on that page 
while continue my maven education.
>
> [1] https://cwiki.apache.org/confluence/display/EDGENT/Maven+vs+Gradle 

>
> — Dale
>
>
>

Re: Maven wiki page [was Re: Understanding the snapshot and release process]

[Christofer Dutz]

I’d love to, but you would need to give me the charma to do so (

Chris

Am 05.06.17, 20:14 schrieb "Dale LaBossiere" :

I created a wiki page [1] to help us work through / collaborate on this.
IMO there are too many questions, etc to effectively deal with it just on 
the dev@ list

Chris, maybe you could fill in some of the (red) blanks on that page while 
continue my maven education.

[1] https://cwiki.apache.org/confluence/display/EDGENT/Maven+vs+Gradle 


— Dale

Re: Understanding the snapshot and release process

[Christofer Dutz]

Hi,

So, I just create a PR as I think I am currently finished with everything I 
wanted to fix. 

Edgent should now be buildable with Maven command. At least I managed to get 
the travis build working with maven ☺

Here some structural changes I did. I refactored the project and extracted the 
TestApplications.java together with its service descriptor into a dedicated 
module located in the test submodule. Wherever a jar was referenced for 
dynamically loading, now I don’t reference the jar where it’s created during 
the build, I use the maven-dependency plugin to copy it to a location local to 
the current module and simply load it from there. This way I didn’t have to 
deal with any too complicated classpath lookups. I could not only run the tests 
as part of the Maven build, but also directly inside my IDE (IntellIJ).

Now some helpful maven commands:

Simply build the project running all unit tests:

mvn clean install

Deploy SNAPSHOTS in Apaches Maven repository:

mvn clean deploy

Additionally, build and deploy source and javadoc artifacts (all signed and 
require correct pgp setup on the client side):

mvn clean deploy -Papache-release

Build without running tests:

mvn clean install -DskipTests

The site generation still needs some love as it would currently run all test 
twice.

Please have a look at this. 

Chris





Am 31.05.17, 18:41 schrieb "Dale LaBossiere" :

Christofer, can you create a [WIP] PR for this work?  Then I’ll have a 
context from which I can review / ask stupid nubie questions :-)

Also, what’s the executive summary for running maven to build a release (or 
at least all of the jars) using this work?  And to run the tests?  I’d like to 
pull the changes into my workspace and really see what’s generated, etc

Thanks!
— Dale

> On May 28, 2017, at 10:03 AM, Christofer Dutz  
wrote:
> 
> had some time to finish the maven migration … I just pushed the changes 
to the “feature/maven” branch of my fork on github: 
https://github.com/chrisdutz/incubator-edgent/tree/feature/maven 

Re: Understanding the snapshot and release process

[Dale LaBossiere]

Christofer, can you create a [WIP] PR for this work?  Then I’ll have a context 
from which I can review / ask stupid nubie questions :-)

Also, what’s the executive summary for running maven to build a release (or at 
least all of the jars) using this work?  And to run the tests?  I’d like to 
pull the changes into my workspace and really see what’s generated, etc

Thanks!
— Dale

> On May 28, 2017, at 10:03 AM, Christofer Dutz  
> wrote:
> 
> had some time to finish the maven migration … I just pushed the changes to 
> the “feature/maven” branch of my fork on github: 
> https://github.com/chrisdutz/incubator-edgent/tree/feature/maven 
> 

Re: Understanding the snapshot and release process

[Christofer Dutz]

Well the big advantage of running the jobs on ASF Jenkins is that you can 
auto-deploy to Nexus (ASF Maven Repo) as well as commit to Git without having 
to store some ones credentials on an external machine. Infra is very helpful in 
getting even complicated setups running (I manage the Windows based flex 
builds, which need Flashplayer, Browsers and Selenium). I would be more than 
happy to help setup everything.

Chris

Am 28.05.17, 17:26 schrieb "John D. Ament" <johndam...@apache.org>:

On Sun, May 28, 2017 at 11:12 AM Christofer Dutz <christofer.d...@c-ware.de>
wrote:

> The Maven build should work on the ASF Jenkins and hereby publish the
> snapshots. I think that in the flex project I added this to ensure its
> added to the generated "site".
>
> Which one is the hosted Jenkins you are referring to? The asf Jenkins?
>
>
Yes, the ASF jenkins.


> And what should be the problem with Travis? Why are you using two ci
> servers?
>

AFAIK, Edgent only uses travis, and doesn't currently run jobs on ASF
Jenkins.  At least if they do, I was unable to find a job with a name that
was easy to identify.


> Chris
>
>
>
> Von meinem Samsung Galaxy Smartphone gesendet.
>
>
>  Ursprüngliche Nachricht 
> Von: "John D. Ament" <johndam...@apache.org>
> Datum: 28.05.17 16:20 (GMT+01:00)
> An: dev@edgent.apache.org
> Betreff: Re: Understanding the snapshot and release process
>
> The maven based build seems to be much more maintainable.  I'm glad you
> guys came together to solve this problem :-)
>
> RE distribution of snapshots.  We only presently support publishing via 
our
> hosted jenkins, however Edgent currently builds in Travis.  So that would
> need to be changed, or some other solution.
>
> You also can drop the repositories/pluginRepositories sections of the new
> pom.  Its not needed.
>
>
> John
>
> On Sun, May 28, 2017 at 10:03 AM Christofer Dutz <
> christofer.d...@c-ware.de>
> wrote:
>
> > Hi guys,
> >
> > Strange people these Germans … if it rains, they complain about it and
> how
> > much they would like to go swimming in the sun … now that it’s 32° and
> the
> > sun is shining, no one wants to go swimming, cause it’s too fu***g hot
> ;-)
> > … well this way I had some time to finish the maven migration … I just
> > pushed the changes to the “feature/maven” branch of my fork on github:
> > https://github.com/chrisdutz/incubator-edgent/tree/feature/maven
> >
> > Here a short list on what little monsters I found siting under some
> rocks:
> >
> > 1) Usage of test-jars --> Bad practice → Move code to separate test-util
> > jar module
> > 2) AppServiceTest in provides/direct relies on system-properties to load
> > jar produced by another module → Ideally this should be refactored to
> find
> > the jar on the classpath
> > 3) HttpServerTest in console/server relies on console.war
> > 4) console/servlets module compiled to something with a name completely
> > unrelated to the project
> > 5) test/svt requires samples project → Some modules rely on samples.
> While
> > I think, this is ok for Samples themselves, nothing outside the samples
> > should rely on samples (my opinion)
> > 6) Almost all tests relying on successful SSL handshakes in
> > wsclient-javax.websocket are failing (I’ll investigate this as soon as I
> > have time, I could imagine this to be related to non US JVM encryption
> > issues we are also seeing in the Flex project)
> > 7) Some unit-tests should more be integration-tests
> >
> > I didn’t migrate the stuff in “platform” as I didn’t quite understand
> what
> > they are used for and what they should do.
> >
> > Hope you like the changes. With a build like this it should be easy to
> > setup the CI to auto distribute SNAPSHOTs and execute releases on ASF
> > machines with a simple one or two-liner.
> >
> > Chris
> >
> >
> > Am 27.05.17, 19:16 schrieb "Christofer Dutz" <christofer.d...@c-ware.de
> >:
> >
> > I did have some free time this weekend, so I thought I’d give the
> > Maven thing a try. And right now I haven finished, but I think I’m about
> > 80% done. So far there have been one or tw

Re: Understanding the snapshot and release process

[John D. Ament]

On Sun, May 28, 2017 at 11:12 AM Christofer Dutz <christofer.d...@c-ware.de>
wrote:

> The Maven build should work on the ASF Jenkins and hereby publish the
> snapshots. I think that in the flex project I added this to ensure its
> added to the generated "site".
>
> Which one is the hosted Jenkins you are referring to? The asf Jenkins?
>
>
Yes, the ASF jenkins.


> And what should be the problem with Travis? Why are you using two ci
> servers?
>

AFAIK, Edgent only uses travis, and doesn't currently run jobs on ASF
Jenkins.  At least if they do, I was unable to find a job with a name that
was easy to identify.


> Chris
>
>
>
> Von meinem Samsung Galaxy Smartphone gesendet.
>
>
>  Ursprüngliche Nachricht 
> Von: "John D. Ament" <johndam...@apache.org>
> Datum: 28.05.17 16:20 (GMT+01:00)
> An: dev@edgent.apache.org
> Betreff: Re: Understanding the snapshot and release process
>
> The maven based build seems to be much more maintainable.  I'm glad you
> guys came together to solve this problem :-)
>
> RE distribution of snapshots.  We only presently support publishing via our
> hosted jenkins, however Edgent currently builds in Travis.  So that would
> need to be changed, or some other solution.
>
> You also can drop the repositories/pluginRepositories sections of the new
> pom.  Its not needed.
>
>
> John
>
> On Sun, May 28, 2017 at 10:03 AM Christofer Dutz <
> christofer.d...@c-ware.de>
> wrote:
>
> > Hi guys,
> >
> > Strange people these Germans … if it rains, they complain about it and
> how
> > much they would like to go swimming in the sun … now that it’s 32° and
> the
> > sun is shining, no one wants to go swimming, cause it’s too fu***g hot
> ;-)
> > … well this way I had some time to finish the maven migration … I just
> > pushed the changes to the “feature/maven” branch of my fork on github:
> > https://github.com/chrisdutz/incubator-edgent/tree/feature/maven
> >
> > Here a short list on what little monsters I found siting under some
> rocks:
> >
> > 1) Usage of test-jars --> Bad practice → Move code to separate test-util
> > jar module
> > 2) AppServiceTest in provides/direct relies on system-properties to load
> > jar produced by another module → Ideally this should be refactored to
> find
> > the jar on the classpath
> > 3) HttpServerTest in console/server relies on console.war
> > 4) console/servlets module compiled to something with a name completely
> > unrelated to the project
> > 5) test/svt requires samples project → Some modules rely on samples.
> While
> > I think, this is ok for Samples themselves, nothing outside the samples
> > should rely on samples (my opinion)
> > 6) Almost all tests relying on successful SSL handshakes in
> > wsclient-javax.websocket are failing (I’ll investigate this as soon as I
> > have time, I could imagine this to be related to non US JVM encryption
> > issues we are also seeing in the Flex project)
> > 7) Some unit-tests should more be integration-tests
> >
> > I didn’t migrate the stuff in “platform” as I didn’t quite understand
> what
> > they are used for and what they should do.
> >
> > Hope you like the changes. With a build like this it should be easy to
> > setup the CI to auto distribute SNAPSHOTs and execute releases on ASF
> > machines with a simple one or two-liner.
> >
> > Chris
> >
> >
> > Am 27.05.17, 19:16 schrieb "Christofer Dutz" <christofer.d...@c-ware.de
> >:
> >
> > I did have some free time this weekend, so I thought I’d give the
> > Maven thing a try. And right now I haven finished, but I think I’m about
> > 80% done. So far there have been one or two little monsters under some
> > rocks … but they weren’t too nasty ;-)
> >
> > Regarding the classpath stuff in the manifest: I really dislike this
> > way of setting the classpath as it usually causes a lot of problems when
> > using it. It requires libs to be in a predefined directory structure and
> > follow a given naming convention. Try to use this in a project built with
> > Maven (just picking this as I saw this would be a problem for a Maven
> built
> > project)
> >
> > Regarding tests: I did see that there are several Unit tests I would
> > probably more call integration-tests. Especially the ones that require
> > loading of a war (console.war). So I guess you have both some classical
> > unit-tests as well as integration-tests … no need to treat them the same
> > way. Maven ususally has the unit-test and the integra

Re: Understanding the snapshot and release process

[John D. Ament]

The maven based build seems to be much more maintainable.  I'm glad you
guys came together to solve this problem :-)

RE distribution of snapshots.  We only presently support publishing via our
hosted jenkins, however Edgent currently builds in Travis.  So that would
need to be changed, or some other solution.

You also can drop the repositories/pluginRepositories sections of the new
pom.  Its not needed.


John

On Sun, May 28, 2017 at 10:03 AM Christofer Dutz 
wrote:

> Hi guys,
>
> Strange people these Germans … if it rains, they complain about it and how
> much they would like to go swimming in the sun … now that it’s 32° and the
> sun is shining, no one wants to go swimming, cause it’s too fu***g hot ;-)
> … well this way I had some time to finish the maven migration … I just
> pushed the changes to the “feature/maven” branch of my fork on github:
> https://github.com/chrisdutz/incubator-edgent/tree/feature/maven
>
> Here a short list on what little monsters I found siting under some rocks:
>
> 1) Usage of test-jars --> Bad practice → Move code to separate test-util
> jar module
> 2) AppServiceTest in provides/direct relies on system-properties to load
> jar produced by another module → Ideally this should be refactored to find
> the jar on the classpath
> 3) HttpServerTest in console/server relies on console.war
> 4) console/servlets module compiled to something with a name completely
> unrelated to the project
> 5) test/svt requires samples project → Some modules rely on samples. While
> I think, this is ok for Samples themselves, nothing outside the samples
> should rely on samples (my opinion)
> 6) Almost all tests relying on successful SSL handshakes in
> wsclient-javax.websocket are failing (I’ll investigate this as soon as I
> have time, I could imagine this to be related to non US JVM encryption
> issues we are also seeing in the Flex project)
> 7) Some unit-tests should more be integration-tests
>
> I didn’t migrate the stuff in “platform” as I didn’t quite understand what
> they are used for and what they should do.
>
> Hope you like the changes. With a build like this it should be easy to
> setup the CI to auto distribute SNAPSHOTs and execute releases on ASF
> machines with a simple one or two-liner.
>
> Chris
>
>
> Am 27.05.17, 19:16 schrieb "Christofer Dutz" :
>
> I did have some free time this weekend, so I thought I’d give the
> Maven thing a try. And right now I haven finished, but I think I’m about
> 80% done. So far there have been one or two little monsters under some
> rocks … but they weren’t too nasty ;-)
>
> Regarding the classpath stuff in the manifest: I really dislike this
> way of setting the classpath as it usually causes a lot of problems when
> using it. It requires libs to be in a predefined directory structure and
> follow a given naming convention. Try to use this in a project built with
> Maven (just picking this as I saw this would be a problem for a Maven built
> project)
>
> Regarding tests: I did see that there are several Unit tests I would
> probably more call integration-tests. Especially the ones that require
> loading of a war (console.war). So I guess you have both some classical
> unit-tests as well as integration-tests … no need to treat them the same
> way. Maven ususally has the unit-test and the integration-test phase for
> exactly this.
>
> Regarding jacoco: So I’ll activate that in the maven build too
>
> Regarding automatic build: For Flex I setup the build to do a
> multibranch pipeline build. So whenever someone checks something in
> “develop”, “releae/*” or “feature/*” it automatically builds that
> particular branch. However develop and “release/*” are the only ones
> uploading SNAPSHOT versions to the ASF Nexus.
>
> Regarding my commitment: I think Edgent is the tool I was looking for
> in order to make my SCADA project feature-perfect. So I have a strong
> interest in the project maturing. I do have plans for a longer involvement
> in the project. Probably I would be trying to create an open-source “plc”
> connector (which would probably have to be developed outside of Apache due
> to libraries still being GPL). But I do plan on staying around some time ;-)
>
> Regarding the usage: Maven or Gradle based application developers
> could directly utilize SNAPSHOT and release artifacts as they are released.
> Additionally, I would create a so-called assembly that produces the same
> zip or tar-gz archives the users have been using. So there shouldn’t be too
> much of a change.
>
> So now I’ll try to finish the rest of the connector modules maven
> conversion (
>
> Chris
>
> Am 25.05.17, 19:37 schrieb "Dale LaBossiere" :
>
> Awaiting others to chime in on this “switch to mvn” subject.
> A “don’t care” (+/- 0) response is preferred over silence.
>
> In the mean time...
>
> > On May 24, 2017, at 11:41 AM, 

Re: Understanding the snapshot and release process

[Christofer Dutz]

Hi guys,

Strange people these Germans … if it rains, they complain about it and how much 
they would like to go swimming in the sun … now that it’s 32° and the sun is 
shining, no one wants to go swimming, cause it’s too fu***g hot ;-) … well this 
way I had some time to finish the maven migration … I just pushed the changes 
to the “feature/maven” branch of my fork on github: 
https://github.com/chrisdutz/incubator-edgent/tree/feature/maven

Here a short list on what little monsters I found siting under some rocks:

1) Usage of test-jars --> Bad practice → Move code to separate test-util jar 
module
2) AppServiceTest in provides/direct relies on system-properties to load jar 
produced by another module → Ideally this should be refactored to find the jar 
on the classpath
3) HttpServerTest in console/server relies on console.war
4) console/servlets module compiled to something with a name completely 
unrelated to the project
5) test/svt requires samples project → Some modules rely on samples. While I 
think, this is ok for Samples themselves, nothing outside the samples should 
rely on samples (my opinion)
6) Almost all tests relying on successful SSL handshakes in 
wsclient-javax.websocket are failing (I’ll investigate this as soon as I have 
time, I could imagine this to be related to non US JVM encryption issues we are 
also seeing in the Flex project)
7) Some unit-tests should more be integration-tests

I didn’t migrate the stuff in “platform” as I didn’t quite understand what they 
are used for and what they should do.

Hope you like the changes. With a build like this it should be easy to setup 
the CI to auto distribute SNAPSHOTs and execute releases on ASF machines with a 
simple one or two-liner.

Chris


Am 27.05.17, 19:16 schrieb "Christofer Dutz" :

I did have some free time this weekend, so I thought I’d give the Maven 
thing a try. And right now I haven finished, but I think I’m about 80% done. So 
far there have been one or two little monsters under some rocks … but they 
weren’t too nasty ;-)

Regarding the classpath stuff in the manifest: I really dislike this way of 
setting the classpath as it usually causes a lot of problems when using it. It 
requires libs to be in a predefined directory structure and follow a given 
naming convention. Try to use this in a project built with Maven (just picking 
this as I saw this would be a problem for a Maven built project)

Regarding tests: I did see that there are several Unit tests I would 
probably more call integration-tests. Especially the ones that require loading 
of a war (console.war). So I guess you have both some classical unit-tests as 
well as integration-tests … no need to treat them the same way. Maven ususally 
has the unit-test and the integration-test phase for exactly this.

Regarding jacoco: So I’ll activate that in the maven build too 

Regarding automatic build: For Flex I setup the build to do a multibranch 
pipeline build. So whenever someone checks something in “develop”, “releae/*” 
or “feature/*” it automatically builds that particular branch. However develop 
and “release/*” are the only ones uploading SNAPSHOT versions to the ASF Nexus.

Regarding my commitment: I think Edgent is the tool I was looking for in 
order to make my SCADA project feature-perfect. So I have a strong interest in 
the project maturing. I do have plans for a longer involvement in the project. 
Probably I would be trying to create an open-source “plc” connector (which 
would probably have to be developed outside of Apache due to libraries still 
being GPL). But I do plan on staying around some time ;-)

Regarding the usage: Maven or Gradle based application developers could 
directly utilize SNAPSHOT and release artifacts as they are released. 
Additionally, I would create a so-called assembly that produces the same zip or 
tar-gz archives the users have been using. So there shouldn’t be too much of a 
change.

So now I’ll try to finish the rest of the connector modules maven 
conversion (

Chris

Am 25.05.17, 19:37 schrieb "Dale LaBossiere" :

Awaiting others to chime in on this “switch to mvn” subject.
A “don’t care” (+/- 0) response is preferred over silence.

In the mean time...

> On May 24, 2017, at 11:41 AM, Christofer Dutz 
 wrote:
> ...
> I just checked out everything and managed to get things imported in 
IntelliJ after a little struggle … this is a good job you did :-)
Can’t take all the credit (or blame :-)
> 
> Right now, it seems as if there were Ant+Gradle+Eclipse build files 
in there. While it might seem ... 
gradle is the only CLI way to build edgent.  Any ant build.xml files 
are either (a) residual cruft that should be removed (my bad) or (b) needed to 
leverage the ant machinery 

Re: Understanding the snapshot and release process

[Dale LaBossiere]

Awaiting others to chime in on this “switch to mvn” subject.
A “don’t care” (+/- 0) response is preferred over silence.

In the mean time...

> On May 24, 2017, at 11:41 AM, Christofer Dutz  
> wrote:
> ...
> I just checked out everything and managed to get things imported in IntelliJ 
> after a little struggle … this is a good job you did :-)
Can’t take all the credit (or blame :-)
> 
> Right now, it seems as if there were Ant+Gradle+Eclipse build files in there. 
> While it might seem ... 
gradle is the only CLI way to build edgent.  Any ant build.xml files are either 
(a) residual cruft that should be removed (my bad) or (b) needed to leverage 
the ant machinery for invoking retrolambda in building java7/android compatible 
versions of the jars (machinery that wasn’t converted to pure gradle due to 
time/effort/value at the time).  Attempting to use ant at the top level tells 
you it doesn’t work :-)  As for Eclipse .project/.classpath, yup those are live 
and haven’t been a maintenance issue.

fwiw, I just removed all build.xml except the top level one and those under 
platform and “gradle release” worked fine so I’ll create a PR to clean them up 
as a first step.

> ...JavaDoc is generated automatically when running a release build together 
> with the usual Maven project reports (is even configured in the apache parent 
> POM together with rat, deployment etc.)
Javadoc was complicated by creating groupings as well as excluding it for 
non-API classes.

> ...I didn’t quite understand the “Manifest” thing, but the jar plugin does 
> generate this with reasonable defaults, and can be extended to also export 
> the dependencies into that (even if I don’t recommend that). 
An edgent jar’s manifest class-path includes references to its immediate 
dependent edgent jars (not transitive) as well as references to its external 
jar dependencies (transitively).  I agree that "compiled in" references to 
specific versions of external dependencies may not be a great idea / is perhaps 
best eliminated.

My recollection is that by default gradle did not generate any manifest 
class-path.

e.g., just to make this a bit more concrete, this is from 
edgent.connectors.kafka.jar/MANIFEST.MF (see connectors/kafka/build.gradle for 
more info)

Class-Path: ../../../lib/edgent.api.topology.jar ../../../ext/gson-2.2
 .4.jar ../../../ext/slf4j-api-1.7.12.jar ../../../ext/metrics-core-3.
 1.2.jar ../ext/kafka_2.10-0.8.2.2.jar ../ext/kafka-clients-0.8.2.2.ja
 r ../ext/log4j-1.2.16.jar ../ext/metrics-core-2.2.0.jar ../ext/scala-
 library-2.10.4.jar ../ext/zkclient-0.3.jar ../ext/zookeeper-3.4.6.jar

edgent.api.topology.jar itself has lots of edgent jar dependencies (captured in 
its manifest) but that’s of no concern to the kafka connector.  I’m not sure 
maintaining this is a requirement and, at least for gradle, eliminating it 
would have greatly simplified things.

Test execution environment was also an issue.  perhaps more a result of 
structural issues.  i.e., the gradle config treated a component’s tests as unit 
tests, run against the associated component’s .class files.  The ant system 
treated them more like integration tests, run against the jar files that would 
be bundled into a binary-release bundle; more like the environment that 
Edgent-based applications would use.  That was very important.  And due to some 
manual setup requirements for some connector tests are excluded from execution 
by default.

> …
> In the Flex project, I also setup the build to run SonarQube analysis and 
> automatically generate the documentation from markdown and/or asciidoctor 
> (which I think is very convenient) even automatically update and deploy the 
> project website. 

We include use of jacoco
We have travis integration for auto-PR validation.
It would be nice to have some periodic build/regression testing run on the main 
(master) branch.

> I could offer to create a fork on GitHub, create a feature branch there and 
> try to whip up a set of poms that add Maven as fourth build system to the 
> list … you could check it out and play around with it. But I’d only do this, 
> if there is any interest in it.

I don’t blame you!

One worry I have is that it ends up like the gradle effort… it was ~trivial to 
get started but there was then a lot of effort required identify and flesh out 
equivalence with the ant based build result artifacts.
If you’re on board for the long haul in doing a conversion then that lessens my 
concern.

I’m on board to help but don’t want to inherit a “now finish it” task :-)

A full switch to a ~simple mvn-based build system would be the goal IMO — we’d 
want to toss gradle. The effort ultimately also entails updating a bunch of 
doc.  Not a killer, just not to be overlooked.

It’s unclear to me how the Edgent-based app developers will be affected by all 
of this.  OK, they build their Edgent-based app with some tool that can utilize 
a maven repo.  Then what?

Today the story is 

Re: Understanding the snapshot and release process

[Dale LaBossiere]

On May 24, 2017, at 9:38 AM, Christofer Dutz  wrote:
...
Regarding the build system … what is the projects general opinion on a Maven 
build? I know that Ant and Gradle are a lot more flexible as Maven, but I think 
it is a good thing that Maven is this strict, as it forces the developers to 
address structural issues instead of using workarounds. Especially in 
Open-Source projects with a lot of different levels of build system expertise I 
usually always had problems with non Maven builds. But this is just a general 
question without having seen the current state


I think we headed down the gradle path because someone stepped up and
contributed an initial set of gradle build files.  And it was simple.  But it 
omitted 
building javadoc, release bundles, the jar manifest class-path that I’ve
mentioned, and maybe some additional stuff (oh, like building java7 based
jars using retrolambda, as well as building for an android tgt).  

I don’t know that the project has an overall position on gradle vs maven.  
Folks?

I don’t have a strong opinion (though I was happy to see the elimination of an 
xml based build spec :-)

Once “forced to address structural issues instead of using gradle workarounds",
might one then end up with an equivalently simple/clean gradle based system?
Though the path to get to that point may be less clear :-)

— Dale

Re: Understanding the snapshot and release process

[Dale LaBossiere]

Looks like we might also get some help from christofer.d...@c-ware.de in 
cleaning up the build system!

I hope my previous message clarifies why things are the way they are.
I don’t know if there are simpler ways to achieve the same effects or
if it would be best to abandon some of them.

e.g., instead of having the “accurate” auto-generated MANIFEST.MF/Class-Path
in the Edgent jars, we could either have a general, include everything,
Class-Path or omit Class-Path and require applications to set CLASSPATH 
themselves.
In either case we could provide a manually created & maintained file identifying
all of these dependencies.

I believe John would like to get to a state where 3rd party dependencies
are not included in the convenience binary tgz.  And/or eliminate the binary 
tgz?

Ultimately there’s a need to be to generate some bundle that includes
the Edgent jars and their external dependencies for download / install
onto an edge device.  And the app won’t be built on the device.
It should be ~easy for an Edgent user to identify and exclude optional
Edgent components (the component’s Edgent jar and its deps) from
that bundle.

— Dale

Re: Understanding the snapshot and release process

[Dale LaBossiere]

> On May 22, 2017, at 9:54 PM, John D. Ament  wrote:
> ...
> I'm trying to understand the current snapshot and release process for
> Edgent, specifically to make it so that we can easily publish JARs into
> maven central.
> ...

Yeah, the gradle stuff ended up that complicated as a result of preserving 
certain behaviors of the original ant-based build, and being a gradle/maven 
newbie. It would be great to be able to simplify it!  [I wasn’t the author of 
the ant builds.]

The binary build (current and previous) results in a layout of Edgent and 3rd 
party jars:
java8//
  [/ext/]

The generated Edgent jars include MANIFEST.MF/Class-Path for all immediate 
edgent-jar dependencies and the component’s transitive external dependencies.  
All are specified as relative
paths given the above layout.

Edgent applications need to only include an Edgent component that they directly 
use on the their classpath.  Internal and external transitive dependencies are 
automatically handled by virtue of the manifests' classpath.  In practice they 
need only include the path to the Edgent provider jar (which
takes care of all of the core/lib Edgent jars), and then Edgent jars for 
individual connectors or other
optional components. See [1].

The release mgmt guide [2] steps related to source-release and binary-release 
bundle creation and signing are:
./gradlew -Dedgent.snapshotId=“” release
./gradlew signAll —no-daemon

See [2] for staging and publishing steps too.

Oh, and there was some additional complication as a result of preserving the 
ability of Edgent-runtime developers to continue to use Eclipse without 
switching to gradle and/or maven [3]

[1] “Building Applications” at the end of 
https://edgent.apache.org/javadoc/latest/index.html 

[2] 
https://cwiki.apache.org/confluence/display/EDGENT/Release+Manager%27s+Guide 

[3] 
https://github.com/apache/incubator-edgent/blob/master/DEVELOPMENT.md#using-eclipse

Hope that helps clarify things.
— Dale