@Thomas, Not all the jar present in DT community edition will be included there.
Here is the list of jars I found from common dependencies between of apex and DT Community edition: netlet-1.2.0.jar kryo-2.24.0.jar jackson-core-asl-1.9.2.jar jackson-mapper-asl-1.9.2.jar async-http-client-1.7.20.jar netty-3.6.6.Final.jar validation-api-1.1.0.Final.jar bval-jsr303-0.5.jar bval-core-0.5.jar commons-lang3-3.1.jar commons-beanutils-1.8.3.jar httpclient-4.3.5.jar commons-codec-1.10.jar zip4j-1.3.2.jar jetty-websocket-8.1.10.v20130312.jar xbean-asm5-shaded-4.3.jar jersey-apache-client4-1.9.jar jline-2.11.jar ant-1.9.2.jar ant-launcher-1.9.2.jar mbassador-1.1.9.jar jackson-jaxrs-1.9.2.jar jackson-xc-1.9.2.jar hadoop-common-2.2.0-tests.jar Ofcourse, I'll be running some tests do check that dtcli works properly for launch+shutdown+kill of apps with only these libraries present in isolation without dependency on local m2. I'm believe that there are unwanted jars which are used for compile time dependency and not runtime in above list which I can drop to keep package size minimal. @Anniruddha, 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is required. Only required values I see in it are classpath. and dtcli is capable of building it on the fly. 2. dt-site.xml -> Yes, we'll have to see how dtcli can find the jar path, dt-env.sh and all such conf files. If we need a change in dtcli, then community's opinion is required for whether dtcli should change in our repo or a copy of that with required changes exist in bigtop repo until we make the dtcli generic enough. 3. dt-sited.xml -> How is this file different from dt-site.xml? Not sure if adding in /etc/skel would help. Files & Dirs from /etc/skel are copied to home of new user when useradd program is called. But for existing users this won't be of any use. Moreover, dtcli creates ~/.dt/ on the fly if not encountered for the first time. Again correct me if I'm wrong. 4. Changing bashrc & bash_profile -> I'm not sure its the best idea. We'll anyway put dtcli in location which is by default present in path i.e. /usr/bin. For other env variables specific to apex, I'll prefer to use dt-env.sh and source it in dtcli rather than changing bashrc etc... Thanks, Chinmay. On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare < [email protected]> wrote: > Hi, > > @Chinmay, > > We need to consider following: > System wide default config files can be located at following locations: > > /etc/apex/dt-env.sh (we may have to change dtcli behaviour on how it finds > those locations) > /etc/apex/dt-site.xml (we may have to change dtcli behaviour on how it > finds those locations) > /etc/skel/.dt/dt-sited.xml and other files (for new users that will be > created in system in future.) > > We may also have to modify bashrc / bashprofile for population any > variables that are required. > > @dev, please put in your comments / suggestion. > > > > Thanks, > > > Aniruddha > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <[email protected]> > wrote: > > > --> > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar < > [email protected] > > > > > wrote: > > > > > Hello Everyone!! > > > > > > Continuing with packaging effort (rpm+deb) of apex, here are some > > proposals > > > about package structure etc.. > > > Before posting it on bbigtop mailing list, I have some question for > apex > > > community. > > > > > > Proposed Directory structure of apex package for both deb & rpm: > > > > > > /usr/lib/apex/bin/dtcli > > > /usr/lib/apex/lib/apex-api-<version>.jar > > > /usr/lib/apex/lib/apex-engine-<version>.jar > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar > > > /usr/lib/apex/lib/apex-common-<version>.jar > > > /usr/lib/apex/lib/<other dependent jars> > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli > > > /usr/share/doc/man/man1/dtcli.1.gz > > > /usr/share/doc/apex/license/LICENSE.txt > > > /usr/share/doc/apex/license/<package>-LICENSE.txt > > > /usr/share/doc/apex/CHANGELOG > > > /usr/share/doc/apex/NOTICE > > > > > > > > > <version> = 3.3.0-incubating. > > > <other dependent jars> = All the 3rd party jars which are required for > > apex > > > to run. Usually the dependencies are packaged as part of rpm/deb by any > > > software in bigtop. > > > > > > > Can you specify what those jars are in Bigtop context? Same as shipped > with > > DT community addition under lib/ ? > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd party jars which > > needs > > > to included while packaging. > > > > > > Questions related to this: > > > 1. Should we call the cli of apex as "apex" instead of "dtcli" in > bigtop > > > package? > > > > > > > I think we should keep the name until we are able to change it in Apex. > > Otherwise this may get confusing. > > > > > > > 2. I see that all softwares in bigtop have man page for their > > executables. > > > I think we should have it too for dtcli. Is there any documentation > > which I > > > can convert to man page? or can I use output of "dtcli --help"? > > > 3. Do we want to call version of apex in Bigtop as 3.3.0 OR > > > 3.3.0-incubating? > > > > > > > Has to be -incubating > > > > > > > 4. Is it ok for apex package of bigtop to depend on 2.7.1 version of > > > bigtop hadoop? Any problems that we see with this dependency? > > > > > > > Should work without changed. I thought we certified against 2.7.0? > > > > > > > 5. Following is the apache mirror from which bigtop will pick the apex > > > source for compilation and packaging. Please correct if incorrect: > > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/ > > > NOTE: apache.osuosl.org is the mirror used by all the softwares in > > > bigtop. > > > > > > > > > Please share your opinion. > > > > > > Thanks, > > > Chinmay. > > > > > > > > > > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar < > > > [email protected]> > > > wrote: > > > > > > > Yes.. Starting to work on the packaging. > > > > > > > > I've already started discussion on bigtop dev mailing list for > > > > integration. Also created a Jira for the same. For this communities > > > > reference, here is the bigtop Jira: BIGTOP-2313. > > > > > > > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <[email protected] > > > > > > wrote: > > > > > > > >> Chinmay, > > > >> > > > >> I don't see anything under prerequisites that would be a problem. We > > > >> looked > > > >> at the ASF licencing compatibility as part of becoming an incubator > > > >> project. > > > >> > > > >> Please focus on the packaging during the next weeks. Since the work > > will > > > >> be > > > >> part of Bigtop, related discussions and JIRAs should also be there. > > > >> > > > >> Would be good to have the packaging in place by end Feb. > > > >> > > > >> Thanks, > > > >> Thomas > > > >> > > > >> > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar < > > > >> [email protected]> > > > >> wrote: > > > >> > > > >> > Hi All, > > > >> > > > > >> > We're planning a work on adding Apache Apex as a component to > Apache > > > >> > Bigtop. > > > >> > Bigtop is the packaging system for the Apache big data ecosystem. > > > >> Several > > > >> > Hadoop distros use it, most recently EMR. > > > >> > > > > >> > Here is the tracking Jira task in APEXCORE for the same: > > > >> > https://issues.apache.org/jira/browse/APEXCORE-331 > > > >> > > > > >> > Proposed plan of execution is as follows: > > > >> > *Step 1) Handle prerequisites* > > > >> > Apache bigtop has some hard and soft expectation for new > components > > to > > > >> get > > > >> > integrated into Bigtop. > > > >> > Here is the list of it: > > > >> > > > > >> > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution > > > >> > > > > >> > Most of them seems to be standard ASF based requirements, but few > > need > > > >> to > > > >> > be checked for: > > > >> > 1. Software projects are expected to be Licensed under Apache > > License, > > > >> > Version 2.0 (and their dependencies are expected to be compatible > > with > > > >> this > > > >> > license) > > > >> > - Apex is under ASL 2.0 but need to check if dependencies of > > Apex > > > >> are > > > >> > compatible with ASL 2.0. This I guess would be a verification > check. > > > >> > 2. Software projects are expected to be compatible with all of the > > > >> > supported platforms that Bigtop distribution is targeting > > > >> > - This needs verifying whether our software runs fine in > > centos-6 > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2. > > > >> > 3. What smoke tests that should be added for deployment. > > > >> > 4. Identifying the test artifacts which goes beyond smoke test > > > >> > - These are basically the integration tests for verification > > after > > > >> the > > > >> > deployment. This is a soft requirement, but aim is to achieve this > > as > > > >> well > > > >> > or at least have explanation why not to include. > > > >> > > > > >> > If there are any from the link which explicitly needs to be > checked > > > >> other > > > >> > than above 4, please let us know. > > > >> > > > > >> > *Step 2) Adding Apex as component to Bigtop* > > > >> > From one of the mail archive of Bigtop, it was learnt that the > > bigtop > > > >> > community want to see the addition of new components in phases. > Here > > > are > > > >> > the phases: > > > >> > 1. Packaging > > > >> > - This needs creating of package i.e. rpm & deb files. > > > >> > - documentations/READMEs, LICENSE, DISCLAMER, NOTES etc if any > > > >> needed. > > > >> > - Any documentation that need to be added to distribution of > our > > > >> > software. > > > >> > - Any license information of dependencies required to be added > > to > > > >> > package > > > >> > 2. Smoke tests (at very least) > > > >> > - Adding smoke test for packaging. > > > >> > 3. Puppet recipes for automatic deployment and configuration > > > >> > - Add puppet recipes for automatic deployment > > > >> > 4. Integration tests > > > >> > - For verification of deployments. > > > >> > 5. license clearance: > > > >> > Run 'gradle rat' to make sure all new stuff is compliant with > > ASF > > > >> > license requirements. If you add code licenses under different > > > licenses, > > > >> > those would need to be listed in the NOTICE. > > > >> > > > > >> > Please share your thoughts on the approach. > > > >> > We'll start corresponding communication on bigtop mailing list as > > > well. > > > >> > > > > >> > We have some specific questions/suggestions related to what should > > be > > > >> the > > > >> > content of the package and what should be the smoke tests, but in > > the > > > >> > interest of not having too much content here, we'll put the > > questions > > > >> as a > > > >> > separate mail in this mailthread. > > > >> > > > > >> > Thanks, > > > >> > Chinmay. > > > >> > > > > >> > > > > > > > > > > > > > >
