Hi Thomas,
Thanks for the feedback.
First, we're not changing the name anywhere. We'll follow what currently is
in source tarball for 3.3.0-incubating version.
Secondly, I've mentioned a directory structure below which is inline with
other existing integrations with bigtop.
The need for each file is as follows:
1. bin/dtcli -> CLI for apex picked from engine/src/main/scripts/ of source
code apex 3.3.0-incubating.
2. conf/dt-env.sh -> This is source from dtcli. This searches for hadoop
binary path and exports an env variable for dtcli to use it.
3. lib/ -> Libraries for CLI. This list is extracted in a similar way to
how DT Community Edition finds dependency jars.
Please note that this is the first iteration list of jars. I'm trying to
narrow this down to only those which are really required.
The test that I'm running to check if dtcli runs fine with given dependency
is to launch+shutdown+kill+status for pi demo.
Above is the directory structure which is required for CLI to work.
What aniruddha and pradeep has mentioned are some additional files which
makes the like of administrator easier. For eg. /etc/skel
Please let me know if this is inline with what you're thinking.
Thanks,
Chinmay.
.
|-- bin
| `-- dtcli
|-- conf
| `-- dt-env.sh
`-- lib
|-- ant-1.9.2.jar
|-- ant-launcher-1.9.2.jar
|-- apex-api-3.3.0-incubating.jar
|-- apex-bufferserver-3.3.0-incubating.jar
|-- apex-common-3.3.0-incubating.jar
|-- apex-engine.jar
|-- apex-engine-tests.jar
|-- async-http-client-1.7.20.jar
|-- bval-core-0.5.jar
|-- bval-jsr303-0.5.jar
|-- commons-beanutils-1.8.3.jar
|-- commons-codec-1.10.jar
|-- commons-lang3-3.1.jar
|-- commons-logging-1.1.3.jar
|-- grizzly-http-servlet-2.1.2.jar
|-- hadoop-common-2.2.0-tests.jar
|-- httpclient-4.3.5.jar
|-- httpcore-4.3.2.jar
|-- jackson-core-asl-1.9.2.jar
|-- jackson-mapper-asl-1.9.2.jar
|-- javax.servlet-3.1.jar
|-- javax.servlet-api-3.0.1.jar
|-- jersey-apache-client4-1.9.jar
|-- jersey-client-1.9.jar
|-- jetty-http-8.1.10.v20130312.jar
|-- jetty-io-8.1.10.v20130312.jar
|-- jetty-util-8.1.10.v20130312.jar
|-- jetty-websocket-8.1.10.v20130312.jar
|-- jline-2.11.jar
|-- kryo-2.24.0.jar
|-- mbassador-1.1.9.jar
|-- minlog-1.2.jar
|-- netlet-1.2.0.jar
|-- netty-3.6.6.Final.jar
|-- objenesis-2.1.jar
|-- validation-api-1.1.0.Final.jar
|-- xbean-asm5-shaded-4.3.jar
`-- zip4j-1.3.2.jar
3 directories, 40 files
On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise <[email protected]>
wrote:
> Chinmay,
>
> Before discussing where to put the files, let's make sure they are really
> needed for the operation of the CLI. As for names, anything new needs to
> reflect Apex in the name and should follow common conventions, especially
> in Bigtop where there are many existing integrations to look at.
>
> Thanks,
> Thomas
>
>
> On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar <
> [email protected]>
> wrote:
>
> > Really good points Pradeep and Aniruddha.
> >
> > 1. I believe we won't need to change the dtcli considering it works with
> DT
> > Community edition. We can keep the directory structure similar to that.
> > dt-env.sh has variables which contains information required for dtcli to
> > launch.
> >
> > 2. Let me check with bigtop community that whether they facilitate the
> > installation of rpms/debs before user is created. In either case, current
> > dtcli creates a .dt folder in home directory. Also before putting
> anything
> > in /etc/skel we need to define what are the default contents that should
> go
> > to ~/.dt/ folder. If there is no defaults, probably we should not
> > explicitly add it in /etc/skel.
> >
> > 3. /etc/profile.d approach looks nice. This way contents of dt-env.sh
> are
> > present as env variables. But I see a catch there. Adding dt-env.sh to
> > /etc/profile.d would make all the variables available at runtime all the
> > time. I feel a little skeptical about that. Maybe a possible collision
> can
> > occur with other application vars.
> > Moreover, current dtcli does source "../conf/dt-env.sh" and
> > "~/.dt/dt-env.sh". So those variables are anyway available.
> >
> > @Aniruddha, the Jira and PR effort is happening at Bigtop
> Jira/repository.
> > Packaging related code usually goes there. (That's what all the
> components
> > in bigtop does).
> > Having said that, once a PR is created, I'll be sharing the link of PR
> > here, so that apex community as well can review it.
> >
> > Thanks,
> > Chinmay.
> >
> >
> > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare <
> > [email protected]> wrote:
> >
> > > +1 on suggestions and approach.
> > > We may need to iron out details about exact paths etc.
> > > Which can be done on jira / PR comments.
> > > Is that right @dev?
> > >
> > >
> > >
> > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi <[email protected]>
> > > wrote:
> > >
> > > > Inline comments...
> > > >
> > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar <
> > > > [email protected]>
> > > > wrote:
> > > >
> > > > > @Thomas, Not all the jar present in DT community edition will be
> > > included
> > > > > there.
> > > > >
> > > > > Here is the list of jars I found from common dependencies between
> of
> > > apex
> > > > > and DT Community edition:
> > > > > netlet-1.2.0.jar
> > > > > kryo-2.24.0.jar
> > > > > jackson-core-asl-1.9.2.jar
> > > > > jackson-mapper-asl-1.9.2.jar
> > > > > async-http-client-1.7.20.jar
> > > > > netty-3.6.6.Final.jar
> > > > > validation-api-1.1.0.Final.jar
> > > > > bval-jsr303-0.5.jar
> > > > > bval-core-0.5.jar
> > > > > commons-lang3-3.1.jar
> > > > > commons-beanutils-1.8.3.jar
> > > > > httpclient-4.3.5.jar
> > > > > commons-codec-1.10.jar
> > > > > zip4j-1.3.2.jar
> > > > > jetty-websocket-8.1.10.v20130312.jar
> > > > > xbean-asm5-shaded-4.3.jar
> > > > > jersey-apache-client4-1.9.jar
> > > > > jline-2.11.jar
> > > > > ant-1.9.2.jar
> > > > > ant-launcher-1.9.2.jar
> > > > > mbassador-1.1.9.jar
> > > > > jackson-jaxrs-1.9.2.jar
> > > > > jackson-xc-1.9.2.jar
> > > > > hadoop-common-2.2.0-tests.jar
> > > > >
> > > > > Ofcourse, I'll be running some tests do check that dtcli works
> > properly
> > > > for
> > > > > launch+shutdown+kill of apps with only these libraries present in
> > > > isolation
> > > > > without dependency on local m2.
> > > > > I'm believe that there are unwanted jars which are used for compile
> > > time
> > > > > dependency and not runtime in above list which I can drop to keep
> > > package
> > > > > size minimal.
> > > > >
> > > > >
> > > > > @Anniruddha,
> > > > > 1. dt-env.sh -> I'm not sure if all of the dt-env.sh is required.
> > Only
> > > > > required values I see in it are classpath. and dtcli is capable of
> > > > building
> > > > > it on the fly.
> > > > >
> > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli can find the jar
> > > path,
> > > > > dt-env.sh and all such conf files. If we need a change in dtcli,
> then
> > > > > community's opinion is required for whether dtcli should change in
> > our
> > > > repo
> > > > > or a copy of that with required changes exist in bigtop repo until
> we
> > > > make
> > > > > the dtcli generic enough.
> > > > >
> > > >
> > > > Can we have these file paths set using environment variables with
> some
> > > > default values in dtcli?
> > > > Then we can set such params in dt-env.sh.
> > > >
> > > >
> > > > > 3. dt-sited.xml -> How is this file different from dt-site.xml? Not
> > > sure
> > > > if
> > > > > adding in /etc/skel would help. Files & Dirs from /etc/skel are
> > copied
> > > to
> > > > > home of new user when useradd program is called. But for existing
> > users
> > > > > this won't be of any use. Moreover, dtcli creates ~/.dt/ on the fly
> > if
> > > > not
> > > > > encountered for the first time. Again correct me if I'm wrong.
> > > > >
> > > >
> > > > Yes, you are right. However this would probably be necessary step for
> > > > rpm/deb, as they may also get installed during OS install and before
> > user
> > > > accounts were created.
> > > >
> > > >
> > > > > 4. Changing bashrc & bash_profile -> I'm not sure its the best
> idea.
> > > > We'll
> > > > > anyway put dtcli in location which is by default present in path
> i.e.
> > > > > /usr/bin.
> > > >
> > > > For other env variables specific to apex, I'll prefer to use
> dt-env.sh
> > > and
> > > > > source it in dtcli rather than changing bashrc etc...
> > > >
> > > >
> > > > Instead, coping dt-env.sh to /etc/profile.d/ shall be the cleaner way
> > to
> > > > achieve the same.
> > > >
> > > >
> > > > >
> > > > Thanks,
> > > > > Chinmay.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha Thombare <
> > > > > [email protected]> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > @Chinmay,
> > > > > >
> > > > > > We need to consider following:
> > > > > > System wide default config files can be located at following
> > > locations:
> > > > > >
> > > > > > /etc/apex/dt-env.sh (we may have to change dtcli behaviour on
> how
> > it
> > > > > finds
> > > > > > those locations)
> > > > > > /etc/apex/dt-site.xml (we may have to change dtcli behaviour on
> how
> > > it
> > > > > > finds those locations)
> > > > > > /etc/skel/.dt/dt-sited.xml and other files (for new users that
> will
> > > be
> > > > > > created in system in future.)
> > > > > >
> > > > > > We may also have to modify bashrc / bashprofile for population
> any
> > > > > > variables that are required.
> > > > > >
> > > > > > @dev, please put in your comments / suggestion.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > >
> > > > > > Aniruddha
> > > > > >
> > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise <
> > > [email protected]>
> > > > > > wrote:
> > > > > >
> > > > > > > -->
> > > > > > >
> > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay Kolhatkar <
> > > > > > [email protected]
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Everyone!!
> > > > > > > >
> > > > > > > > Continuing with packaging effort (rpm+deb) of apex, here are
> > some
> > > > > > > proposals
> > > > > > > > about package structure etc..
> > > > > > > > Before posting it on bbigtop mailing list, I have some
> question
> > > for
> > > > > > apex
> > > > > > > > community.
> > > > > > > >
> > > > > > > > Proposed Directory structure of apex package for both deb &
> > rpm:
> > > > > > > >
> > > > > > > > /usr/lib/apex/bin/dtcli
> > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar
> > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar
> > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar
> > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar
> > > > > > > > /usr/lib/apex/lib/<other dependent jars>
> > > > > > >
> > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli
> > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz
> > > > > > > > /usr/share/doc/apex/license/LICENSE.txt
> > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt
> > > > > > > > /usr/share/doc/apex/CHANGELOG
> > > > > > > > /usr/share/doc/apex/NOTICE
> > > > > > > >
> > > > > > > >
> > > > > > > > <version> = 3.3.0-incubating.
> > > > > > > > <other dependent jars> = All the 3rd party jars which are
> > > required
> > > > > for
> > > > > > > apex
> > > > > > > > to run. Usually the dependencies are packaged as part of
> > rpm/deb
> > > by
> > > > > any
> > > > > > > > software in bigtop.
> > > > > > > >
> > > > > > >
> > > > > > > Can you specify what those jars are in Bigtop context? Same as
> > > > shipped
> > > > > > with
> > > > > > > DT community addition under lib/ ?
> > > > > > >
> > > > > > >
> > > > > > > > <package LICENSE> = Licenses of corresponding of 3rd party
> jars
> > > > which
> > > > > > > needs
> > > > > > > > to included while packaging.
> > > > > > > >
> > > > > > > > Questions related to this:
> > > > > > > > 1. Should we call the cli of apex as "apex" instead of
> "dtcli"
> > in
> > > > > > bigtop
> > > > > > > > package?
> > > > > > > >
> > > > > > >
> > > > > > > I think we should keep the name until we are able to change it
> in
> > > > Apex.
> > > > > > > Otherwise this may get confusing.
> > > > > > >
> > > > > > >
> > > > > > > > 2. I see that all softwares in bigtop have man page for their
> > > > > > > executables.
> > > > > > > > I think we should have it too for dtcli. Is there any
> > > documentation
> > > > > > > which I
> > > > > > > > can convert to man page? or can I use output of "dtcli
> --help"?
> > > > > > > > 3. Do we want to call version of apex in Bigtop as 3.3.0 OR
> > > > > > > > 3.3.0-incubating?
> > > > > > > >
> > > > > > >
> > > > > > > Has to be -incubating
> > > > > > >
> > > > > > >
> > > > > > > > 4. Is it ok for apex package of bigtop to depend on 2.7.1
> > > version
> > > > of
> > > > > > > > bigtop hadoop? Any problems that we see with this dependency?
> > > > > > > >
> > > > > > >
> > > > > > > Should work without changed. I thought we certified against
> > 2.7.0?
> > > > > > >
> > > > > > >
> > > > > > > > 5. Following is the apache mirror from which bigtop will pick
> > the
> > > > > apex
> > > > > > > > source for compilation and packaging. Please correct if
> > > incorrect:
> > > > > > > >
> http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/
> > > > > > > > NOTE: apache.osuosl.org is the mirror used by all the
> > > > softwares
> > > > > in
> > > > > > > > bigtop.
> > > > > > > >
> > > > > > > >
> > > > > > > > Please share your opinion.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Chinmay.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay Kolhatkar <
> > > > > > > > [email protected]>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Yes.. Starting to work on the packaging.
> > > > > > > > >
> > > > > > > > > I've already started discussion on bigtop dev mailing list
> > for
> > > > > > > > > integration. Also created a Jira for the same. For this
> > > > communities
> > > > > > > > > reference, here is the bigtop Jira: BIGTOP-2313.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas Weise <
> > > > > [email protected]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Chinmay,
> > > > > > > > >>
> > > > > > > > >> I don't see anything under prerequisites that would be a
> > > > problem.
> > > > > We
> > > > > > > > >> looked
> > > > > > > > >> at the ASF licencing compatibility as part of becoming an
> > > > > incubator
> > > > > > > > >> project.
> > > > > > > > >>
> > > > > > > > >> Please focus on the packaging during the next weeks. Since
> > the
> > > > > work
> > > > > > > will
> > > > > > > > >> be
> > > > > > > > >> part of Bigtop, related discussions and JIRAs should also
> be
> > > > > there.
> > > > > > > > >>
> > > > > > > > >> Would be good to have the packaging in place by end Feb.
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> Thomas
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay Kolhatkar <
> > > > > > > > >> [email protected]>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi All,
> > > > > > > > >> >
> > > > > > > > >> > We're planning a work on adding Apache Apex as a
> component
> > > to
> > > > > > Apache
> > > > > > > > >> > Bigtop.
> > > > > > > > >> > Bigtop is the packaging system for the Apache big data
> > > > > ecosystem.
> > > > > > > > >> Several
> > > > > > > > >> > Hadoop distros use it, most recently EMR.
> > > > > > > > >> >
> > > > > > > > >> > Here is the tracking Jira task in APEXCORE for the same:
> > > > > > > > >> > https://issues.apache.org/jira/browse/APEXCORE-331
> > > > > > > > >> >
> > > > > > > > >> > Proposed plan of execution is as follows:
> > > > > > > > >> > *Step 1) Handle prerequisites*
> > > > > > > > >> > Apache bigtop has some hard and soft expectation for new
> > > > > > components
> > > > > > > to
> > > > > > > > >> get
> > > > > > > > >> > integrated into Bigtop.
> > > > > > > > >> > Here is the list of it:
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution
> > > > > > > > >> >
> > > > > > > > >> > Most of them seems to be standard ASF based
> requirements,
> > > but
> > > > > few
> > > > > > > need
> > > > > > > > >> to
> > > > > > > > >> > be checked for:
> > > > > > > > >> > 1. Software projects are expected to be Licensed under
> > > Apache
> > > > > > > License,
> > > > > > > > >> > Version 2.0 (and their dependencies are expected to be
> > > > > compatible
> > > > > > > with
> > > > > > > > >> this
> > > > > > > > >> > license)
> > > > > > > > >> > - Apex is under ASL 2.0 but need to check if
> > > dependencies
> > > > of
> > > > > > > Apex
> > > > > > > > >> are
> > > > > > > > >> > compatible with ASL 2.0. This I guess would be a
> > > verification
> > > > > > check.
> > > > > > > > >> > 2. Software projects are expected to be compatible with
> > all
> > > of
> > > > > the
> > > > > > > > >> > supported platforms that Bigtop distribution is
> targeting
> > > > > > > > >> > - This needs verifying whether our software runs
> fine
> > in
> > > > > > > centos-6
> > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2.
> > > > > > > > >> > 3. What smoke tests that should be added for deployment.
> > > > > > > > >> > 4. Identifying the test artifacts which goes beyond
> smoke
> > > test
> > > > > > > > >> > - These are basically the integration tests for
> > > > verification
> > > > > > > after
> > > > > > > > >> the
> > > > > > > > >> > deployment. This is a soft requirement, but aim is to
> > > achieve
> > > > > this
> > > > > > > as
> > > > > > > > >> well
> > > > > > > > >> > or at least have explanation why not to include.
> > > > > > > > >> >
> > > > > > > > >> > If there are any from the link which explicitly needs to
> > be
> > > > > > checked
> > > > > > > > >> other
> > > > > > > > >> > than above 4, please let us know.
> > > > > > > > >> >
> > > > > > > > >> > *Step 2) Adding Apex as component to Bigtop*
> > > > > > > > >> > From one of the mail archive of Bigtop, it was learnt
> that
> > > the
> > > > > > > bigtop
> > > > > > > > >> > community want to see the addition of new components in
> > > > phases.
> > > > > > Here
> > > > > > > > are
> > > > > > > > >> > the phases:
> > > > > > > > >> > 1. Packaging
> > > > > > > > >> > - This needs creating of package i.e. rpm & deb
> files.
> > > > > > > > >> > - documentations/READMEs, LICENSE, DISCLAMER, NOTES
> > etc
> > > if
> > > > > any
> > > > > > > > >> needed.
> > > > > > > > >> > - Any documentation that need to be added to
> > > distribution
> > > > of
> > > > > > our
> > > > > > > > >> > software.
> > > > > > > > >> > - Any license information of dependencies required
> to
> > be
> > > > > added
> > > > > > > to
> > > > > > > > >> > package
> > > > > > > > >> > 2. Smoke tests (at very least)
> > > > > > > > >> > - Adding smoke test for packaging.
> > > > > > > > >> > 3. Puppet recipes for automatic deployment and
> > configuration
> > > > > > > > >> > - Add puppet recipes for automatic deployment
> > > > > > > > >> > 4. Integration tests
> > > > > > > > >> > - For verification of deployments.
> > > > > > > > >> > 5. license clearance:
> > > > > > > > >> > Run 'gradle rat' to make sure all new stuff is
> > compliant
> > > > > with
> > > > > > > ASF
> > > > > > > > >> > license requirements. If you add code licenses under
> > > different
> > > > > > > > licenses,
> > > > > > > > >> > those would need to be listed in the NOTICE.
> > > > > > > > >> >
> > > > > > > > >> > Please share your thoughts on the approach.
> > > > > > > > >> > We'll start corresponding communication on bigtop
> mailing
> > > list
> > > > > as
> > > > > > > > well.
> > > > > > > > >> >
> > > > > > > > >> > We have some specific questions/suggestions related to
> > what
> > > > > should
> > > > > > > be
> > > > > > > > >> the
> > > > > > > > >> > content of the package and what should be the smoke
> tests,
> > > but
> > > > > in
> > > > > > > the
> > > > > > > > >> > interest of not having too much content here, we'll put
> > the
> > > > > > > questions
> > > > > > > > >> as a
> > > > > > > > >> > separate mail in this mailthread.
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> > Chinmay.
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
