+1 on suggestions and approach. We may need to iron out details about exact paths etc. Which can be done on jira / PR comments. Is that right @dev?
On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <[email protected]> wrote: > Inline comments... > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar < > [email protected]> > wrote: > > > @Thomas, Not all the jar present in DT community edition will be included > > there. > > > > Here is the list of jars I found from common dependencies between of apex > > and DT Community edition: > > netlet-1.2.0.jar > > kryo-2.24.0.jar > > jackson-core-asl-1.9.2.jar > > jackson-mapper-asl-1.9.2.jar > > async-http-client-1.7.20.jar > > netty-3.6.6.Final.jar > > validation-api-1.1.0.Final.jar > > bval-jsr303-0.5.jar > > bval-core-0.5.jar > > commons-lang3-3.1.jar > > commons-beanutils-1.8.3.jar > > httpclient-4.3.5.jar > > commons-codec-1.10.jar > > zip4j-1.3.2.jar > > jetty-websocket-8.1.10.v20130312.jar > > xbean-asm5-shaded-4.3.jar > > jersey-apache-client4-1.9.jar > > jline-2.11.jar > > ant-1.9.2.jar > > ant-launcher-1.9.2.jar > > mbassador-1.1.9.jar > > jackson-jaxrs-1.9.2.jar > > jackson-xc-1.9.2.jar > > hadoop-common-2.2.0-tests.jar > > > > Ofcourse, I'll be running some tests do check that dtcli works properly > for > > launch+shutdown+kill of apps with only these libraries present in > isolation > > without dependency on local m2. > > I'm believe that there are unwanted jars which are used for compile time > > dependency and not runtime in above list which I can drop to keep package > > size minimal. > > > > > > @Anniruddha, > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is required. Only > > required values I see in it are classpath. and dtcli is capable of > building > > it on the fly. > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find the jar path, > > dt-env.sh and all such conf files. If we need a change in dtcli, then > > community's opinion is required for whether dtcli should change in our > repo > > or a copy of that with required changes exist in bigtop repo until we > make > > the dtcli generic enough. > > > > Can we have these file paths set using environment variables with some > default values in dtcli? > Then we can set such params in dt-env.sh. > > > > 3. dt-sited.xml -> How is this file different from dt-site.xml? Not sure > if > > adding in /etc/skel would help. Files & Dirs from /etc/skel are copied to > > home of new user when useradd program is called. But for existing users > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on the fly if > not > > encountered for the first time. Again correct me if I'm wrong. > > > > Yes, you are right. However this would probably be necessary step for > rpm/deb, as they may also get installed during OS install and before user > accounts were created. > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the best idea. > We'll > > anyway put dtcli in location which is by default present in path i.e. > > /usr/bin. > > For other env variables specific to apex, I'll prefer to use dt-env.sh and > > source it in dtcli rather than changing bashrc etc... > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the cleaner way to > achieve the same. > > > > > Thanks, > > Chinmay. > > > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare < > > [email protected]> wrote: > > > > > Hi, > > > > > > @Chinmay, > > > > > > We need to consider following: > > > System wide default config files can be located at following locations: > > > > > > /etc/apex/dt-env.sh (we may have to change dtcli behaviour on how it > > finds > > > those locations) > > > /etc/apex/dt-site.xml (we may have to change dtcli behaviour on how it > > > finds those locations) > > > /etc/skel/.dt/dt-sited.xml and other files (for new users that will be > > > created in system in future.) > > > > > > We may also have to modify bashrc / bashprofile for population any > > > variables that are required. > > > > > > @dev, please put in your comments / suggestion. > > > > > > > > > > > > Thanks, > > > > > > > > > Aniruddha > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <[email protected]> > > > wrote: > > > > > > > --> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar < > > > [email protected] > > > > > > > > > wrote: > > > > > > > > > Hello Everyone!! > > > > > > > > > > Continuing with packaging effort (rpm+deb) of apex, here are some > > > > proposals > > > > > about package structure etc.. > > > > > Before posting it on bbigtop mailing list, I have some question for > > > apex > > > > > community. > > > > > > > > > > Proposed Directory structure of apex package for both deb & rpm: > > > > > > > > > > /usr/lib/apex/bin/dtcli > > > > > /usr/lib/apex/lib/apex-api-<version>.jar > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar > > > > > /usr/lib/apex/lib/apex-common-<version>.jar > > > > > /usr/lib/apex/lib/<other dependent jars> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli > > > > > /usr/share/doc/man/man1/dtcli.1.gz > > > > > /usr/share/doc/apex/license/LICENSE.txt > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt > > > > > /usr/share/doc/apex/CHANGELOG > > > > > /usr/share/doc/apex/NOTICE > > > > > > > > > > > > > > > <version> = 3.3.0-incubating. > > > > > <other dependent jars> = All the 3rd party jars which are required > > for > > > > apex > > > > > to run. Usually the dependencies are packaged as part of rpm/deb by > > any > > > > > software in bigtop. > > > > > > > > > > > > > Can you specify what those jars are in Bigtop context? Same as > shipped > > > with > > > > DT community addition under lib/ ? > > > > > > > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd party jars > which > > > > needs > > > > > to included while packaging. > > > > > > > > > > Questions related to this: > > > > > 1. Should we call the cli of apex as "apex" instead of "dtcli" in > > > bigtop > > > > > package? > > > > > > > > > > > > > I think we should keep the name until we are able to change it in > Apex. > > > > Otherwise this may get confusing. > > > > > > > > > > > > > 2. I see that all softwares in bigtop have man page for their > > > > executables. > > > > > I think we should have it too for dtcli. Is there any documentation > > > > which I > > > > > can convert to man page? or can I use output of "dtcli --help"? > > > > > 3. Do we want to call version of apex in Bigtop as 3.3.0 OR > > > > > 3.3.0-incubating? > > > > > > > > > > > > > Has to be -incubating > > > > > > > > > > > > > 4. Is it ok for apex package of bigtop to depend on 2.7.1 version > of > > > > > bigtop hadoop? Any problems that we see with this dependency? > > > > > > > > > > > > > Should work without changed. I thought we certified against 2.7.0? > > > > > > > > > > > > > 5. Following is the apache mirror from which bigtop will pick the > > apex > > > > > source for compilation and packaging. Please correct if incorrect: > > > > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/ > > > > > NOTE: apache.osuosl.org is the mirror used by all the > softwares > > in > > > > > bigtop. > > > > > > > > > > > > > > > Please share your opinion. > > > > > > > > > > Thanks, > > > > > Chinmay. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar < > > > > > [email protected]> > > > > > wrote: > > > > > > > > > > > Yes.. Starting to work on the packaging. > > > > > > > > > > > > I've already started discussion on bigtop dev mailing list for > > > > > > integration. Also created a Jira for the same. For this > communities > > > > > > reference, here is the bigtop Jira: BIGTOP-2313. > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > >> Chinmay, > > > > > >> > > > > > >> I don't see anything under prerequisites that would be a > problem. > > We > > > > > >> looked > > > > > >> at the ASF licencing compatibility as part of becoming an > > incubator > > > > > >> project. > > > > > >> > > > > > >> Please focus on the packaging during the next weeks. Since the > > work > > > > will > > > > > >> be > > > > > >> part of Bigtop, related discussions and JIRAs should also be > > there. > > > > > >> > > > > > >> Would be good to have the packaging in place by end Feb. > > > > > >> > > > > > >> Thanks, > > > > > >> Thomas > > > > > >> > > > > > >> > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar < > > > > > >> [email protected]> > > > > > >> wrote: > > > > > >> > > > > > >> > Hi All, > > > > > >> > > > > > > >> > We're planning a work on adding Apache Apex as a component to > > > Apache > > > > > >> > Bigtop. > > > > > >> > Bigtop is the packaging system for the Apache big data > > ecosystem. > > > > > >> Several > > > > > >> > Hadoop distros use it, most recently EMR. > > > > > >> > > > > > > >> > Here is the tracking Jira task in APEXCORE for the same: > > > > > >> > https://issues.apache.org/jira/browse/APEXCORE-331 > > > > > >> > > > > > > >> > Proposed plan of execution is as follows: > > > > > >> > *Step 1) Handle prerequisites* > > > > > >> > Apache bigtop has some hard and soft expectation for new > > > components > > > > to > > > > > >> get > > > > > >> > integrated into Bigtop. > > > > > >> > Here is the list of it: > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution > > > > > >> > > > > > > >> > Most of them seems to be standard ASF based requirements, but > > few > > > > need > > > > > >> to > > > > > >> > be checked for: > > > > > >> > 1. Software projects are expected to be Licensed under Apache > > > > License, > > > > > >> > Version 2.0 (and their dependencies are expected to be > > compatible > > > > with > > > > > >> this > > > > > >> > license) > > > > > >> > - Apex is under ASL 2.0 but need to check if dependencies > of > > > > Apex > > > > > >> are > > > > > >> > compatible with ASL 2.0. This I guess would be a verification > > > check. > > > > > >> > 2. Software projects are expected to be compatible with all of > > the > > > > > >> > supported platforms that Bigtop distribution is targeting > > > > > >> > - This needs verifying whether our software runs fine in > > > > centos-6 > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2. > > > > > >> > 3. What smoke tests that should be added for deployment. > > > > > >> > 4. Identifying the test artifacts which goes beyond smoke test > > > > > >> > - These are basically the integration tests for > verification > > > > after > > > > > >> the > > > > > >> > deployment. This is a soft requirement, but aim is to achieve > > this > > > > as > > > > > >> well > > > > > >> > or at least have explanation why not to include. > > > > > >> > > > > > > >> > If there are any from the link which explicitly needs to be > > > checked > > > > > >> other > > > > > >> > than above 4, please let us know. > > > > > >> > > > > > > >> > *Step 2) Adding Apex as component to Bigtop* > > > > > >> > From one of the mail archive of Bigtop, it was learnt that the > > > > bigtop > > > > > >> > community want to see the addition of new components in > phases. > > > Here > > > > > are > > > > > >> > the phases: > > > > > >> > 1. Packaging > > > > > >> > - This needs creating of package i.e. rpm & deb files. > > > > > >> > - documentations/READMEs, LICENSE, DISCLAMER, NOTES etc if > > any > > > > > >> needed. > > > > > >> > - Any documentation that need to be added to distribution > of > > > our > > > > > >> > software. > > > > > >> > - Any license information of dependencies required to be > > added > > > > to > > > > > >> > package > > > > > >> > 2. Smoke tests (at very least) > > > > > >> > - Adding smoke test for packaging. > > > > > >> > 3. Puppet recipes for automatic deployment and configuration > > > > > >> > - Add puppet recipes for automatic deployment > > > > > >> > 4. Integration tests > > > > > >> > - For verification of deployments. > > > > > >> > 5. license clearance: > > > > > >> > Run 'gradle rat' to make sure all new stuff is compliant > > with > > > > ASF > > > > > >> > license requirements. If you add code licenses under different > > > > > licenses, > > > > > >> > those would need to be listed in the NOTICE. > > > > > >> > > > > > > >> > Please share your thoughts on the approach. > > > > > >> > We'll start corresponding communication on bigtop mailing list > > as > > > > > well. > > > > > >> > > > > > > >> > We have some specific questions/suggestions related to what > > should > > > > be > > > > > >> the > > > > > >> > content of the package and what should be the smoke tests, but > > in > > > > the > > > > > >> > interest of not having too much content here, we'll put the > > > > questions > > > > > >> as a > > > > > >> > separate mail in this mailthread. > > > > > >> > > > > > > >> > Thanks, > > > > > >> > Chinmay. > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
