Please follow the recommendation and continue discussion on the Bigtop list.
-- sent from mobile On Feb 16, 2016 8:17 AM, "Chinmay Kolhatkar" <[email protected]> wrote: > I got a recommendation from bigtop community to drop "-incubating" from > version. > > Here is the mail thread: > > http://mail-archives.apache.org/mod_mbox/bigtop-dev/201602.mbox/%3C20160216154548.GA4472%40tpx%3E > > Please let me know if removing "-incubating" is fine. > > Thanks, > Chinmay. > > > On Tue, Feb 16, 2016 at 9:06 PM, Chinmay Kolhatkar <[email protected]> > wrote: > > > I've already started communication there. > > > > On Tue, Feb 16, 2016 at 8:33 PM, Thomas Weise <[email protected]> > > wrote: > > > >> This is a question you should ask on the Bigtop mailing list. > >> > >> -- > >> sent from mobile > >> On Feb 16, 2016 5:25 AM, "Pradeep A. Dalvi" <[email protected]> wrote: > >> > >> > Yes. In case of RPM, the only restriction on version is that it can > not > >> > contain '-'. > >> > However, as a workaround, one can use 'incubating' as a part of > >> %{release}. > >> > > >> > Version: 3.3.0 > >> > Release: incubating > >> > > >> > apex-3.3.0-incubating.noarch.rpm > >> > > >> > It is the same practice followed for pre-release packages e.g. with > rcX, > >> > alphaX, betaX being used in %{release}. > >> > > >> > Regards, > >> > Pradeep A. Dalvi > >> > > >> > On Tue, Feb 16, 2016 at 6:35 PM, Chinmay Kolhatkar < > [email protected]> > >> > wrote: > >> > > >> > > Hi Everyone, > >> > > > >> > > In one of the previous mails in this thread, we decided that version > >> of > >> > the > >> > > package will be "3.3.0-incubating". > >> > > The problem that I'm facing is in RPM base version "-" is not > allowed. > >> > > > >> > > Can we have package version to "3.3.0"? > >> > > i.e. the packages created will be as follows: > >> > > For RPM: apex-3.3.0-1.noarch.rpm > >> > > For DEB: apex_3.3.0-1_all.deb > >> > > > >> > > There is another integration for Zepplin in Bigtop which is > incubating > >> > and > >> > > is having base version as "0.5.6". > >> > > > >> > > Please let me know your thoughts. > >> > > > >> > > Thanks, > >> > > Chinmay. > >> > > > >> > > > >> > > > >> > > On Tue, Feb 16, 2016 at 11:48 AM, Chinmay Kolhatkar < > >> [email protected]> > >> > > wrote: > >> > > > >> > > > Thanks everyone for the feedback. > >> > > > > >> > > > Also as Ram and David mentioned, I'll start creating the man page > as > >> > > well. > >> > > > > >> > > > Thanks, > >> > > > Chinmay. > >> > > > > >> > > > > >> > > > On Tue, Feb 16, 2016 at 6:30 AM, Thomas Weise < > >> [email protected]> > >> > > > wrote: > >> > > > > >> > > >> I think we already agreed that we will retain the current script > >> name > >> > in > >> > > >> this bigtop iteration. There is a ticket for renaming, add you > >> > > suggestions > >> > > >> there. > >> > > >> > >> > > >> -- > >> > > >> sent from mobile > >> > > >> On Feb 15, 2016 4:56 PM, "Sandesh Hegde" < > [email protected]> > >> > > wrote: > >> > > >> > >> > > >> > +1 for Symlink from apex to dtcli or start the transition > >> process by > >> > > >> making > >> > > >> > dtcli as the symlink to apex cli. > >> > > >> > > >> > > >> > Renaming is a breaking change should be done during major > version > >> > > number > >> > > >> > change. > >> > > >> > > >> > > >> > On Mon, Feb 15, 2016 at 9:57 AM David Yan < > [email protected] > >> > > >> > > >> wrote: > >> > > >> > > >> > > >> > > Apologies for hitting the "Send" button too early. > >> > > >> > > > >> > > >> > > What is the scope of the name change? If we simply have a > >> symlink > >> > > >> from > >> > > >> > > apex to dtcli, then it should have no backward compatibility > >> > problem > >> > > >> at > >> > > >> > > all. > >> > > >> > > > >> > > >> > > But the proper name change would involve: > >> > > >> > > - Change the name of DTCli class to ApexCli > >> > > >> > > - Change the prompt from dt> to apex> > >> > > >> > > - Change the initial greeting when running the CLI from > >> something > >> > > like > >> > > >> > "DT > >> > > >> > > CLI 3.3.0-incubating ..." to "Apex CLI ...". > >> > > >> > > > >> > > >> > > The above changes may break existing scripts that rely on the > >> the > >> > > >> string > >> > > >> > > "DT". > >> > > >> > > But if folks are okay with this potential problem, I am okay > >> with > >> > > the > >> > > >> > name > >> > > >> > > change. > >> > > >> > > > >> > > >> > > For the man page, we can start with the content of this page: > >> > > >> > > > >> > > >> > > http://docs.datatorrent.com/dtcli/ > >> > > >> > > > >> > > >> > > Plus the command line options when doing dtcli -h: > >> > > >> > > > >> > > >> > > usage: DTCli > >> > > >> > > -e <arg> Commands are read from the argument > >> > > >> > > -f <arg> Use the specified prompt at all time > >> > > >> > > -h Print this help > >> > > >> > > -kp <arg> Use the specified kerberos principal > >> > > >> > > -kt <arg> Use the specified kerberos keytab > >> > > >> > > -p <arg> JSONP padding function > >> > > >> > > -r JSON Raw mode > >> > > >> > > -v Verbose mode level 1 > >> > > >> > > -vv Verbose mode level 2 > >> > > >> > > -vvv Verbose mode level 3 > >> > > >> > > -vvvv Verbose mode level 4 > >> > > >> > > > >> > > >> > > David > >> > > >> > > > >> > > >> > > > >> > > >> > > On Mon, Feb 15, 2016 at 9:56 AM, David Yan < > >> [email protected] > >> > > > >> > > >> > wrote: > >> > > >> > > > >> > > >> > > > What is the scope of the name change? If we simply have a > >> > symlink > >> > > >> from > >> > > >> > > > apex to dtcli, then it should have no backward > compatibility > >> > > >> problem at > >> > > >> > > all. > >> > > >> > > > > >> > > >> > > > But the proper name change would involve: > >> > > >> > > > - Change the name of DTCli class to ApexCli > >> > > >> > > > - Change the prompt from dt> to apex> > >> > > >> > > > - Change the initial greeting when running the CLI from > >> > something > >> > > >> like > >> > > >> > > "DT > >> > > >> > > > CLI 3.3.0-incubating ..." to "Apex CLI ...". > >> > > >> > > > > >> > > >> > > > The above changes may break existing scripts that rely on > the > >> > the > >> > > >> > string > >> > > >> > > > "DT". > >> > > >> > > > But if folks are okay with this potential problem, I am > okay > >> > with > >> > > >> the > >> > > >> > > name > >> > > >> > > > change. > >> > > >> > > > > >> > > >> > > > For the man page, we can start with the content of this > page: > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar < > >> > > >> [email protected] > >> > > >> > > > >> > > >> > > > wrote: > >> > > >> > > > > >> > > >> > > >> Here is a suggestion I have related to name change: > >> > > >> > > >> > >> > > >> > > >> 1. While packaging we change the name of "dtcli" script to > >> > > "apex". > >> > > >> > This > >> > > >> > > is > >> > > >> > > >> done so that in future when we change the name to "apex", > >> users > >> > > of > >> > > >> > > bigtop > >> > > >> > > >> apex don't have to transition much. > >> > > >> > > >> 2. We also keep a symlink named dtcli which points to apex > >> > > script. > >> > > >> > This > >> > > >> > > is > >> > > >> > > >> for backward compatibility. > >> > > >> > > >> 3. We still keep conf/dt-env.sh for first integration in > >> > bigtop. > >> > > >> This > >> > > >> > is > >> > > >> > > >> in > >> > > >> > > >> the interest of not changing the content of dtcli or apex > >> > script. > >> > > >> > > >> 4. In apex 3.4.0, we take care of all these naming related > >> > > changes > >> > > >> and > >> > > >> > > >> update bigtop repository later to remove references to > dtcli > >> > all > >> > > >> > > together. > >> > > >> > > >> > >> > > >> > > >> Please share your thoughts on above approach. > >> > > >> > > >> > >> > > >> > > >> Also, please share what could be the man page content for > >> > "apex". > >> > > >> > > >> > >> > > >> > > >> Thanks, > >> > > >> > > >> Chinmay. > >> > > >> > > >> > >> > > >> > > >> > >> > > >> > > >> > >> > > >> > > >> On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar < > >> > > >> > [email protected]> > >> > > >> > > >> wrote: > >> > > >> > > >> > >> > > >> > > >> > Yes.. It'll be build from source tar downloaded from one > >> of > >> > the > >> > > >> > apache > >> > > >> > > >> > mirror: apache.osuosl.org > >> > > >> > > >> > > >> > > >> > > >> > The reason for dt-env.sh is it's sourced from dtcli. > This > >> > > leaves > >> > > >> us > >> > > >> > 3 > >> > > >> > > >> > options: > >> > > >> > > >> > If we should not have any dt-env.sh, dtcli would need a > >> > change > >> > > >> while > >> > > >> > > >> > packaging. > >> > > >> > > >> > > >> > > >> > > >> > But if we want to use dtcli as it is, then we would need > >> > > >> dt-env.sh > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise < > >> > > >> > [email protected] > >> > > >> > > > > >> > > >> > > >> > wrote: > >> > > >> > > >> > > >> > > >> > > >> >> Looks good overall, though there shouldn't be any > >> dt-env.sh > >> > > >> > > >> >> > >> > > >> > > >> >> What will the bigtop package be built from, the source > >> tar? > >> > > >> > > >> >> > >> > > >> > > >> >> > >> > > >> > > >> >> > >> > > >> > > >> >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar < > >> > > >> > > [email protected] > >> > > >> > > >> > > >> > > >> > > >> >> wrote: > >> > > >> > > >> >> > >> > > >> > > >> >> > Hi Thomas, > >> > > >> > > >> >> > > >> > > >> > > >> >> > Thanks for the feedback. > >> > > >> > > >> >> > > >> > > >> > > >> >> > First, we're not changing the name anywhere. We'll > >> follow > >> > > what > >> > > >> > > >> >> currently is > >> > > >> > > >> >> > in source tarball for 3.3.0-incubating version. > >> > > >> > > >> >> > > >> > > >> > > >> >> > Secondly, I've mentioned a directory structure below > >> which > >> > > is > >> > > >> > > inline > >> > > >> > > >> >> with > >> > > >> > > >> >> > other existing integrations with bigtop. > >> > > >> > > >> >> > > >> > > >> > > >> >> > The need for each file is as follows: > >> > > >> > > >> >> > 1. bin/dtcli -> CLI for apex picked from > >> > > >> engine/src/main/scripts/ > >> > > >> > > of > >> > > >> > > >> >> source > >> > > >> > > >> >> > code apex 3.3.0-incubating. > >> > > >> > > >> >> > > >> > > >> > > >> >> > 2. conf/dt-env.sh -> This is source from dtcli. This > >> > > searches > >> > > >> for > >> > > >> > > >> hadoop > >> > > >> > > >> >> > binary path and exports an env variable for dtcli to > >> use > >> > it. > >> > > >> > > >> >> > > >> > > >> > > >> >> > 3. lib/ -> Libraries for CLI. This list is extracted > >> in a > >> > > >> similar > >> > > >> > > >> way to > >> > > >> > > >> >> > how DT Community Edition finds dependency jars. > >> > > >> > > >> >> > Please note that this is the first iteration list of > >> jars. > >> > > I'm > >> > > >> > > >> trying to > >> > > >> > > >> >> > narrow this down to only those which are really > >> required. > >> > > >> > > >> >> > The test that I'm running to check if dtcli runs fine > >> with > >> > > >> given > >> > > >> > > >> >> dependency > >> > > >> > > >> >> > is to launch+shutdown+kill+status for pi demo. > >> > > >> > > >> >> > > >> > > >> > > >> >> > Above is the directory structure which is required > for > >> CLI > >> > > to > >> > > >> > work. > >> > > >> > > >> >> > What aniruddha and pradeep has mentioned are some > >> > additional > >> > > >> > files > >> > > >> > > >> which > >> > > >> > > >> >> > makes the like of administrator easier. For eg. > >> /etc/skel > >> > > >> > > >> >> > > >> > > >> > > >> >> > Please let me know if this is inline with what you're > >> > > >> thinking. > >> > > >> > > >> >> > > >> > > >> > > >> >> > Thanks, > >> > > >> > > >> >> > Chinmay. > >> > > >> > > >> >> > > >> > > >> > > >> >> > > >> > > >> > > >> >> > . > >> > > >> > > >> >> > |-- bin > >> > > >> > > >> >> > | `-- dtcli > >> > > >> > > >> >> > |-- conf > >> > > >> > > >> >> > | `-- dt-env.sh > >> > > >> > > >> >> > `-- lib > >> > > >> > > >> >> > |-- ant-1.9.2.jar > >> > > >> > > >> >> > |-- ant-launcher-1.9.2.jar > >> > > >> > > >> >> > |-- apex-api-3.3.0-incubating.jar > >> > > >> > > >> >> > |-- apex-bufferserver-3.3.0-incubating.jar > >> > > >> > > >> >> > |-- apex-common-3.3.0-incubating.jar > >> > > >> > > >> >> > |-- apex-engine.jar > >> > > >> > > >> >> > |-- apex-engine-tests.jar > >> > > >> > > >> >> > |-- async-http-client-1.7.20.jar > >> > > >> > > >> >> > |-- bval-core-0.5.jar > >> > > >> > > >> >> > |-- bval-jsr303-0.5.jar > >> > > >> > > >> >> > |-- commons-beanutils-1.8.3.jar > >> > > >> > > >> >> > |-- commons-codec-1.10.jar > >> > > >> > > >> >> > |-- commons-lang3-3.1.jar > >> > > >> > > >> >> > |-- commons-logging-1.1.3.jar > >> > > >> > > >> >> > |-- grizzly-http-servlet-2.1.2.jar > >> > > >> > > >> >> > |-- hadoop-common-2.2.0-tests.jar > >> > > >> > > >> >> > |-- httpclient-4.3.5.jar > >> > > >> > > >> >> > |-- httpcore-4.3.2.jar > >> > > >> > > >> >> > |-- jackson-core-asl-1.9.2.jar > >> > > >> > > >> >> > |-- jackson-mapper-asl-1.9.2.jar > >> > > >> > > >> >> > |-- javax.servlet-3.1.jar > >> > > >> > > >> >> > |-- javax.servlet-api-3.0.1.jar > >> > > >> > > >> >> > |-- jersey-apache-client4-1.9.jar > >> > > >> > > >> >> > |-- jersey-client-1.9.jar > >> > > >> > > >> >> > |-- jetty-http-8.1.10.v20130312.jar > >> > > >> > > >> >> > |-- jetty-io-8.1.10.v20130312.jar > >> > > >> > > >> >> > |-- jetty-util-8.1.10.v20130312.jar > >> > > >> > > >> >> > |-- jetty-websocket-8.1.10.v20130312.jar > >> > > >> > > >> >> > |-- jline-2.11.jar > >> > > >> > > >> >> > |-- kryo-2.24.0.jar > >> > > >> > > >> >> > |-- mbassador-1.1.9.jar > >> > > >> > > >> >> > |-- minlog-1.2.jar > >> > > >> > > >> >> > |-- netlet-1.2.0.jar > >> > > >> > > >> >> > |-- netty-3.6.6.Final.jar > >> > > >> > > >> >> > |-- objenesis-2.1.jar > >> > > >> > > >> >> > |-- validation-api-1.1.0.Final.jar > >> > > >> > > >> >> > |-- xbean-asm5-shaded-4.3.jar > >> > > >> > > >> >> > `-- zip4j-1.3.2.jar > >> > > >> > > >> >> > > >> > > >> > > >> >> > 3 directories, 40 files > >> > > >> > > >> >> > > >> > > >> > > >> >> > > >> > > >> > > >> >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise < > >> > > >> > > >> [email protected]> > >> > > >> > > >> >> > wrote: > >> > > >> > > >> >> > > >> > > >> > > >> >> > > Chinmay, > >> > > >> > > >> >> > > > >> > > >> > > >> >> > > Before discussing where to put the files, let's > make > >> > sure > >> > > >> they > >> > > >> > > are > >> > > >> > > >> >> really > >> > > >> > > >> >> > > needed for the operation of the CLI. As for names, > >> > > anything > >> > > >> new > >> > > >> > > >> needs > >> > > >> > > >> >> to > >> > > >> > > >> >> > > reflect Apex in the name and should follow common > >> > > >> conventions, > >> > > >> > > >> >> especially > >> > > >> > > >> >> > > in Bigtop where there are many existing > integrations > >> to > >> > > look > >> > > >> > at. > >> > > >> > > >> >> > > > >> > > >> > > >> >> > > Thanks, > >> > > >> > > >> >> > > Thomas > >> > > >> > > >> >> > > > >> > > >> > > >> >> > > > >> > > >> > > >> >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar > < > >> > > >> > > >> >> > > [email protected]> > >> > > >> > > >> >> > > wrote: > >> > > >> > > >> >> > > > >> > > >> > > >> >> > > > Really good points Pradeep and Aniruddha. > >> > > >> > > >> >> > > > > >> > > >> > > >> >> > > > 1. I believe we won't need to change the dtcli > >> > > >> considering it > >> > > >> > > >> works > >> > > >> > > >> >> > with > >> > > >> > > >> >> > > DT > >> > > >> > > >> >> > > > Community edition. We can keep the directory > >> structure > >> > > >> > similar > >> > > >> > > to > >> > > >> > > >> >> that. > >> > > >> > > >> >> > > > dt-env.sh has variables which contains > information > >> > > >> required > >> > > >> > for > >> > > >> > > >> >> dtcli > >> > > >> > > >> >> > to > >> > > >> > > >> >> > > > launch. > >> > > >> > > >> >> > > > > >> > > >> > > >> >> > > > 2. Let me check with bigtop community that > whether > >> > they > >> > > >> > > >> facilitate > >> > > >> > > >> >> the > >> > > >> > > >> >> > > > installation of rpms/debs before user is created. > >> In > >> > > >> either > >> > > >> > > case, > >> > > >> > > >> >> > current > >> > > >> > > >> >> > > > dtcli creates a .dt folder in home directory. > Also > >> > > before > >> > > >> > > putting > >> > > >> > > >> >> > > anything > >> > > >> > > >> >> > > > in /etc/skel we need to define what are the > default > >> > > >> contents > >> > > >> > > that > >> > > >> > > >> >> > should > >> > > >> > > >> >> > > go > >> > > >> > > >> >> > > > to ~/.dt/ folder. If there is no defaults, > >> probably we > >> > > >> should > >> > > >> > > not > >> > > >> > > >> >> > > > explicitly add it in /etc/skel. > >> > > >> > > >> >> > > > > >> > > >> > > >> >> > > > 3. /etc/profile.d approach looks nice. This way > >> > > contents > >> > > >> of > >> > > >> > > >> >> dt-env.sh > >> > > >> > > >> >> > > are > >> > > >> > > >> >> > > > present as env variables. But I see a catch > there. > >> > > Adding > >> > > >> > > >> dt-env.sh > >> > > >> > > >> >> to > >> > > >> > > >> >> > > > /etc/profile.d would make all the variables > >> available > >> > at > >> > > >> > > runtime > >> > > >> > > >> all > >> > > >> > > >> >> > the > >> > > >> > > >> >> > > > time. I feel a little skeptical about that. > Maybe a > >> > > >> possible > >> > > >> > > >> >> collision > >> > > >> > > >> >> > > can > >> > > >> > > >> >> > > > occur with other application vars. > >> > > >> > > >> >> > > > Moreover, current dtcli does source > >> > "../conf/dt-env.sh" > >> > > >> and > >> > > >> > > >> >> > > > "~/.dt/dt-env.sh". So those variables are anyway > >> > > >> available. > >> > > >> > > >> >> > > > > >> > > >> > > >> >> > > > @Aniruddha, the Jira and PR effort is happening > at > >> > > Bigtop > >> > > >> > > >> >> > > Jira/repository. > >> > > >> > > >> >> > > > Packaging related code usually goes there. > (That's > >> > what > >> > > >> all > >> > > >> > the > >> > > >> > > >> >> > > components > >> > > >> > > >> >> > > > in bigtop does). > >> > > >> > > >> >> > > > Having said that, once a PR is created, I'll be > >> > sharing > >> > > >> the > >> > > >> > > link > >> > > >> > > >> of > >> > > >> > > >> >> PR > >> > > >> > > >> >> > > > here, so that apex community as well can review > it. > >> > > >> > > >> >> > > > > >> > > >> > > >> >> > > > Thanks, > >> > > >> > > >> >> > > > Chinmay. > >> > > >> > > >> >> > > > > >> > > >> > > >> >> > > > > >> > > >> > > >> >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha > >> Thombare < > >> > > >> > > >> >> > > > [email protected]> wrote: > >> > > >> > > >> >> > > > > >> > > >> > > >> >> > > > > +1 on suggestions and approach. > >> > > >> > > >> >> > > > > We may need to iron out details about exact > paths > >> > etc. > >> > > >> > > >> >> > > > > Which can be done on jira / PR comments. > >> > > >> > > >> >> > > > > Is that right @dev? > >> > > >> > > >> >> > > > > > >> > > >> > > >> >> > > > > > >> > > >> > > >> >> > > > > > >> > > >> > > >> >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi < > >> > > >> > > >> >> > [email protected]> > >> > > >> > > >> >> > > > > wrote: > >> > > >> > > >> >> > > > > > >> > > >> > > >> >> > > > > > Inline comments... > >> > > >> > > >> >> > > > > > > >> > > >> > > >> >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay > >> > Kolhatkar < > >> > > >> > > >> >> > > > > > [email protected]> > >> > > >> > > >> >> > > > > > wrote: > >> > > >> > > >> >> > > > > > > >> > > >> > > >> >> > > > > > > @Thomas, Not all the jar present in DT > >> community > >> > > >> > edition > >> > > >> > > >> will > >> > > >> > > >> >> be > >> > > >> > > >> >> > > > > included > >> > > >> > > >> >> > > > > > > there. > >> > > >> > > >> >> > > > > > > > >> > > >> > > >> >> > > > > > > Here is the list of jars I found from > common > >> > > >> > dependencies > >> > > >> > > >> >> between > >> > > >> > > >> >> > > of > >> > > >> > > >> >> > > > > apex > >> > > >> > > >> >> > > > > > > and DT Community edition: > >> > > >> > > >> >> > > > > > > netlet-1.2.0.jar > >> > > >> > > >> >> > > > > > > kryo-2.24.0.jar > >> > > >> > > >> >> > > > > > > jackson-core-asl-1.9.2.jar > >> > > >> > > >> >> > > > > > > jackson-mapper-asl-1.9.2.jar > >> > > >> > > >> >> > > > > > > async-http-client-1.7.20.jar > >> > > >> > > >> >> > > > > > > netty-3.6.6.Final.jar > >> > > >> > > >> >> > > > > > > validation-api-1.1.0.Final.jar > >> > > >> > > >> >> > > > > > > bval-jsr303-0.5.jar > >> > > >> > > >> >> > > > > > > bval-core-0.5.jar > >> > > >> > > >> >> > > > > > > commons-lang3-3.1.jar > >> > > >> > > >> >> > > > > > > commons-beanutils-1.8.3.jar > >> > > >> > > >> >> > > > > > > httpclient-4.3.5.jar > >> > > >> > > >> >> > > > > > > commons-codec-1.10.jar > >> > > >> > > >> >> > > > > > > zip4j-1.3.2.jar > >> > > >> > > >> >> > > > > > > jetty-websocket-8.1.10.v20130312.jar > >> > > >> > > >> >> > > > > > > xbean-asm5-shaded-4.3.jar > >> > > >> > > >> >> > > > > > > jersey-apache-client4-1.9.jar > >> > > >> > > >> >> > > > > > > jline-2.11.jar > >> > > >> > > >> >> > > > > > > ant-1.9.2.jar > >> > > >> > > >> >> > > > > > > ant-launcher-1.9.2.jar > >> > > >> > > >> >> > > > > > > mbassador-1.1.9.jar > >> > > >> > > >> >> > > > > > > jackson-jaxrs-1.9.2.jar > >> > > >> > > >> >> > > > > > > jackson-xc-1.9.2.jar > >> > > >> > > >> >> > > > > > > hadoop-common-2.2.0-tests.jar > >> > > >> > > >> >> > > > > > > > >> > > >> > > >> >> > > > > > > Ofcourse, I'll be running some tests do > check > >> > that > >> > > >> > dtcli > >> > > >> > > >> works > >> > > >> > > >> >> > > > properly > >> > > >> > > >> >> > > > > > for > >> > > >> > > >> >> > > > > > > launch+shutdown+kill of apps with only > these > >> > > >> libraries > >> > > >> > > >> >> present in > >> > > >> > > >> >> > > > > > isolation > >> > > >> > > >> >> > > > > > > without dependency on local m2. > >> > > >> > > >> >> > > > > > > I'm believe that there are unwanted jars > >> which > >> > are > >> > > >> used > >> > > >> > > for > >> > > >> > > >> >> > compile > >> > > >> > > >> >> > > > > time > >> > > >> > > >> >> > > > > > > dependency and not runtime in above list > >> which I > >> > > can > >> > > >> > drop > >> > > >> > > >> to > >> > > >> > > >> >> keep > >> > > >> > > >> >> > > > > package > >> > > >> > > >> >> > > > > > > size minimal. > >> > > >> > > >> >> > > > > > > > >> > > >> > > >> >> > > > > > > > >> > > >> > > >> >> > > > > > > @Anniruddha, > >> > > >> > > >> >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the > >> > > >> dt-env.sh is > >> > > >> > > >> >> required. > >> > > >> > > >> >> > > > Only > >> > > >> > > >> >> > > > > > > required values I see in it are classpath. > >> and > >> > > >> dtcli is > >> > > >> > > >> >> capable > >> > > >> > > >> >> > of > >> > > >> > > >> >> > > > > > building > >> > > >> > > >> >> > > > > > > it on the fly. > >> > > >> > > >> >> > > > > > > > >> > > >> > > >> >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see > how > >> > dtcli > >> > > >> can > >> > > >> > > find > >> > > >> > > >> >> the > >> > > >> > > >> >> > jar > >> > > >> > > >> >> > > > > path, > >> > > >> > > >> >> > > > > > > dt-env.sh and all such conf files. If we > >> need a > >> > > >> change > >> > > >> > in > >> > > >> > > >> >> dtcli, > >> > > >> > > >> >> > > then > >> > > >> > > >> >> > > > > > > community's opinion is required for whether > >> > dtcli > >> > > >> > should > >> > > >> > > >> >> change > >> > > >> > > >> >> > in > >> > > >> > > >> >> > > > our > >> > > >> > > >> >> > > > > > repo > >> > > >> > > >> >> > > > > > > or a copy of that with required changes > >> exist in > >> > > >> bigtop > >> > > >> > > >> repo > >> > > >> > > >> >> > until > >> > > >> > > >> >> > > we > >> > > >> > > >> >> > > > > > make > >> > > >> > > >> >> > > > > > > the dtcli generic enough. > >> > > >> > > >> >> > > > > > > > >> > > >> > > >> >> > > > > > > >> > > >> > > >> >> > > > > > Can we have these file paths set using > >> environment > >> > > >> > > variables > >> > > >> > > >> >> with > >> > > >> > > >> >> > > some > >> > > >> > > >> >> > > > > > default values in dtcli? > >> > > >> > > >> >> > > > > > Then we can set such params in dt-env.sh. > >> > > >> > > >> >> > > > > > > >> > > >> > > >> >> > > > > > > >> > > >> > > >> >> > > > > > > 3. dt-sited.xml -> How is this file > different > >> > from > >> > > >> > > >> >> dt-site.xml? > >> > > >> > > >> >> > Not > >> > > >> > > >> >> > > > > sure > >> > > >> > > >> >> > > > > > if > >> > > >> > > >> >> > > > > > > adding in /etc/skel would help. Files & > Dirs > >> > from > >> > > >> > > /etc/skel > >> > > >> > > >> >> are > >> > > >> > > >> >> > > > copied > >> > > >> > > >> >> > > > > to > >> > > >> > > >> >> > > > > > > home of new user when useradd program is > >> called. > >> > > But > >> > > >> > for > >> > > >> > > >> >> existing > >> > > >> > > >> >> > > > users > >> > > >> > > >> >> > > > > > > this won't be of any use. Moreover, dtcli > >> > creates > >> > > >> > ~/.dt/ > >> > > >> > > on > >> > > >> > > >> >> the > >> > > >> > > >> >> > fly > >> > > >> > > >> >> > > > if > >> > > >> > > >> >> > > > > > not > >> > > >> > > >> >> > > > > > > encountered for the first time. Again > >> correct me > >> > > if > >> > > >> I'm > >> > > >> > > >> wrong. > >> > > >> > > >> >> > > > > > > > >> > > >> > > >> >> > > > > > > >> > > >> > > >> >> > > > > > Yes, you are right. However this would > >> probably be > >> > > >> > > necessary > >> > > >> > > >> >> step > >> > > >> > > >> >> > for > >> > > >> > > >> >> > > > > > rpm/deb, as they may also get installed > during > >> OS > >> > > >> install > >> > > >> > > and > >> > > >> > > >> >> > before > >> > > >> > > >> >> > > > user > >> > > >> > > >> >> > > > > > accounts were created. > >> > > >> > > >> >> > > > > > > >> > > >> > > >> >> > > > > > > >> > > >> > > >> >> > > > > > > 4. Changing bashrc & bash_profile -> I'm > not > >> > sure > >> > > >> its > >> > > >> > the > >> > > >> > > >> best > >> > > >> > > >> >> > > idea. > >> > > >> > > >> >> > > > > > We'll > >> > > >> > > >> >> > > > > > > anyway put dtcli in location which is by > >> default > >> > > >> > present > >> > > >> > > in > >> > > >> > > >> >> path > >> > > >> > > >> >> > > i.e. > >> > > >> > > >> >> > > > > > > /usr/bin. > >> > > >> > > >> >> > > > > > > >> > > >> > > >> >> > > > > > For other env variables specific to apex, > I'll > >> > > prefer > >> > > >> to > >> > > >> > > use > >> > > >> > > >> >> > > dt-env.sh > >> > > >> > > >> >> > > > > and > >> > > >> > > >> >> > > > > > > source it in dtcli rather than changing > >> bashrc > >> > > >> etc... > >> > > >> > > >> >> > > > > > > >> > > >> > > >> >> > > > > > > >> > > >> > > >> >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ > >> shall > >> > > be > >> > > >> the > >> > > >> > > >> >> cleaner > >> > > >> > > >> >> > way > >> > > >> > > >> >> > > > to > >> > > >> > > >> >> > > > > > achieve the same. > >> > > >> > > >> >> > > > > > > >> > > >> > > >> >> > > > > > > >> > > >> > > >> >> > > > > > > > >> > > >> > > >> >> > > > > > Thanks, > >> > > >> > > >> >> > > > > > > Chinmay. > >> > > >> > > >> >> > > > > > > > >> > > >> > > >> >> > > > > > > > >> > > >> > > >> >> > > > > > > > >> > > >> > > >> >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha > >> > > Thombare > >> > > >> < > >> > > >> > > >> >> > > > > > > [email protected]> wrote: > >> > > >> > > >> >> > > > > > > > >> > > >> > > >> >> > > > > > > > Hi, > >> > > >> > > >> >> > > > > > > > > >> > > >> > > >> >> > > > > > > > @Chinmay, > >> > > >> > > >> >> > > > > > > > > >> > > >> > > >> >> > > > > > > > We need to consider following: > >> > > >> > > >> >> > > > > > > > System wide default config files can be > >> > located > >> > > at > >> > > >> > > >> following > >> > > >> > > >> >> > > > > locations: > >> > > >> > > >> >> > > > > > > > > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-env.sh (we may have to > change > >> > > dtcli > >> > > >> > > >> behaviour > >> > > >> > > >> >> on > >> > > >> > > >> >> > > how > >> > > >> > > >> >> > > > it > >> > > >> > > >> >> > > > > > > finds > >> > > >> > > >> >> > > > > > > > those locations) > >> > > >> > > >> >> > > > > > > > /etc/apex/dt-site.xml (we may have to > >> change > >> > > dtcli > >> > > >> > > >> >> behaviour on > >> > > >> > > >> >> > > how > >> > > >> > > >> >> > > > > it > >> > > >> > > >> >> > > > > > > > finds those locations) > >> > > >> > > >> >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other > files > >> > (for > >> > > >> new > >> > > >> > > users > >> > > >> > > >> >> that > >> > > >> > > >> >> > > will > >> > > >> > > >> >> > > > > be > >> > > >> > > >> >> > > > > > > > created in system in future.) > >> > > >> > > >> >> > > > > > > > > >> > > >> > > >> >> > > > > > > > We may also have to modify bashrc / > >> > bashprofile > >> > > >> for > >> > > >> > > >> >> population > >> > > >> > > >> >> > > any > >> > > >> > > >> >> > > > > > > > variables that are required. > >> > > >> > > >> >> > > > > > > > > >> > > >> > > >> >> > > > > > > > @dev, please put in your comments / > >> > suggestion. > >> > > >> > > >> >> > > > > > > > > >> > > >> > > >> >> > > > > > > > > >> > > >> > > >> >> > > > > > > > > >> > > >> > > >> >> > > > > > > > Thanks, > >> > > >> > > >> >> > > > > > > > > >> > > >> > > >> >> > > > > > > > > >> > > >> > > >> >> > > > > > > > Aniruddha > >> > > >> > > >> >> > > > > > > > > >> > > >> > > >> >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas > >> Weise > >> > < > >> > > >> > > >> >> > > > > [email protected]> > >> > > >> > > >> >> > > > > > > > wrote: > >> > > >> > > >> >> > > > > > > > > >> > > >> > > >> >> > > > > > > > > --> > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay > >> > > >> Kolhatkar < > >> > > >> > > >> >> > > > > > > > [email protected] > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > wrote: > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > Hello Everyone!! > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > Continuing with packaging effort > >> (rpm+deb) > >> > > of > >> > > >> > apex, > >> > > >> > > >> here > >> > > >> > > >> >> > are > >> > > >> > > >> >> > > > some > >> > > >> > > >> >> > > > > > > > > proposals > >> > > >> > > >> >> > > > > > > > > > about package structure etc.. > >> > > >> > > >> >> > > > > > > > > > Before posting it on bbigtop mailing > >> > list, I > >> > > >> have > >> > > >> > > >> some > >> > > >> > > >> >> > > question > >> > > >> > > >> >> > > > > for > >> > > >> > > >> >> > > > > > > > apex > >> > > >> > > >> >> > > > > > > > > > community. > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > Proposed Directory structure of apex > >> > package > >> > > >> for > >> > > >> > > both > >> > > >> > > >> >> deb & > >> > > >> > > >> >> > > > rpm: > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/bin/dtcli > >> > > >> > > >> >> > > > > > > > > > > >> /usr/lib/apex/lib/apex-api-<version>.jar > >> > > >> > > >> >> > > > > > > > > > > >> > /usr/lib/apex/lib/apex-engine-<version>.jar > >> > > >> > > >> >> > > > > > > > > > > >> > > >> /usr/lib/apex/lib/apex-bufferserver-<version>.jar > >> > > >> > > >> >> > > > > > > > > > > >> > /usr/lib/apex/lib/apex-common-<version>.jar > >> > > >> > > >> >> > > > > > > > > > /usr/lib/apex/lib/<other dependent > >> jars> > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > /usr/bin/dtcli -> > /usr/lib/apex/bin/dtcli > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz > >> > > >> > > >> >> > > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt > >> > > >> > > >> >> > > > > > > > > > > >> > > >> /usr/share/doc/apex/license/<package>-LICENSE.txt > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG > >> > > >> > > >> >> > > > > > > > > > /usr/share/doc/apex/NOTICE > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > <version> = 3.3.0-incubating. > >> > > >> > > >> >> > > > > > > > > > <other dependent jars> = All the 3rd > >> party > >> > > >> jars > >> > > >> > > which > >> > > >> > > >> >> are > >> > > >> > > >> >> > > > > required > >> > > >> > > >> >> > > > > > > for > >> > > >> > > >> >> > > > > > > > > apex > >> > > >> > > >> >> > > > > > > > > > to run. Usually the dependencies are > >> > > packaged > >> > > >> as > >> > > >> > > >> part of > >> > > >> > > >> >> > > > rpm/deb > >> > > >> > > >> >> > > > > by > >> > > >> > > >> >> > > > > > > any > >> > > >> > > >> >> > > > > > > > > > software in bigtop. > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > Can you specify what those jars are in > >> > Bigtop > >> > > >> > > context? > >> > > >> > > >> >> Same > >> > > >> > > >> >> > as > >> > > >> > > >> >> > > > > > shipped > >> > > >> > > >> >> > > > > > > > with > >> > > >> > > >> >> > > > > > > > > DT community addition under lib/ ? > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > <package LICENSE> = Licenses of > >> > > corresponding > >> > > >> of > >> > > >> > > 3rd > >> > > >> > > >> >> party > >> > > >> > > >> >> > > jars > >> > > >> > > >> >> > > > > > which > >> > > >> > > >> >> > > > > > > > > needs > >> > > >> > > >> >> > > > > > > > > > to included while packaging. > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > Questions related to this: > >> > > >> > > >> >> > > > > > > > > > 1. Should we call the cli of apex as > >> > "apex" > >> > > >> > instead > >> > > >> > > >> of > >> > > >> > > >> >> > > "dtcli" > >> > > >> > > >> >> > > > in > >> > > >> > > >> >> > > > > > > > bigtop > >> > > >> > > >> >> > > > > > > > > > package? > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > I think we should keep the name until > we > >> are > >> > > >> able > >> > > >> > to > >> > > >> > > >> >> change > >> > > >> > > >> >> > it > >> > > >> > > >> >> > > in > >> > > >> > > >> >> > > > > > Apex. > >> > > >> > > >> >> > > > > > > > > Otherwise this may get confusing. > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > 2. I see that all softwares in bigtop > >> have > >> > > man > >> > > >> > page > >> > > >> > > >> for > >> > > >> > > >> >> > their > >> > > >> > > >> >> > > > > > > > > executables. > >> > > >> > > >> >> > > > > > > > > > I think we should have it too for > >> dtcli. > >> > Is > >> > > >> there > >> > > >> > > any > >> > > >> > > >> >> > > > > documentation > >> > > >> > > >> >> > > > > > > > > which I > >> > > >> > > >> >> > > > > > > > > > can convert to man page? or can I use > >> > output > >> > > >> of > >> > > >> > > >> "dtcli > >> > > >> > > >> >> > > --help"? > >> > > >> > > >> >> > > > > > > > > > 3. Do we want to call version of apex > >> in > >> > > >> Bigtop > >> > > >> > as > >> > > >> > > >> >> 3.3.0 OR > >> > > >> > > >> >> > > > > > > > > > 3.3.0-incubating? > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > Has to be -incubating > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > 4. Is it ok for apex package of > bigtop > >> to > >> > > >> depend > >> > > >> > on > >> > > >> > > >> >> 2.7.1 > >> > > >> > > >> >> > > > > version > >> > > >> > > >> >> > > > > > of > >> > > >> > > >> >> > > > > > > > > > bigtop hadoop? Any problems that we > see > >> > with > >> > > >> this > >> > > >> > > >> >> > dependency? > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > Should work without changed. I thought > we > >> > > >> certified > >> > > >> > > >> >> against > >> > > >> > > >> >> > > > 2.7.0? > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > 5. Following is the apache mirror > from > >> > which > >> > > >> > bigtop > >> > > >> > > >> will > >> > > >> > > >> >> > pick > >> > > >> > > >> >> > > > the > >> > > >> > > >> >> > > > > > > apex > >> > > >> > > >> >> > > > > > > > > > source for compilation and packaging. > >> > Please > >> > > >> > > correct > >> > > >> > > >> if > >> > > >> > > >> >> > > > > incorrect: > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > >> > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/ > >> > > >> > > >> >> > > > > > > > > > NOTE: apache.osuosl.org is the > >> mirror > >> > > >> used > >> > > >> > by > >> > > >> > > >> all > >> > > >> > > >> >> the > >> > > >> > > >> >> > > > > > softwares > >> > > >> > > >> >> > > > > > > in > >> > > >> > > >> >> > > > > > > > > > bigtop. > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > Please share your opinion. > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > Thanks, > >> > > >> > > >> >> > > > > > > > > > Chinmay. > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, > >> Chinmay > >> > > >> > Kolhatkar > >> > > >> > > < > >> > > >> > > >> >> > > > > > > > > > [email protected]> > >> > > >> > > >> >> > > > > > > > > > wrote: > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > > Yes.. Starting to work on the > >> packaging. > >> > > >> > > >> >> > > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > > I've already started discussion on > >> > bigtop > >> > > >> dev > >> > > >> > > >> mailing > >> > > >> > > >> >> > list > >> > > >> > > >> >> > > > for > >> > > >> > > >> >> > > > > > > > > > > integration. Also created a Jira > for > >> the > >> > > >> same. > >> > > >> > > For > >> > > >> > > >> >> this > >> > > >> > > >> >> > > > > > communities > >> > > >> > > >> >> > > > > > > > > > > reference, here is the bigtop Jira: > >> > > >> > BIGTOP-2313. > >> > > >> > > >> >> > > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, > >> Thomas > >> > > >> Weise < > >> > > >> > > >> >> > > > > > > [email protected] > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > > wrote: > >> > > >> > > >> >> > > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > >> Chinmay, > >> > > >> > > >> >> > > > > > > > > > >> > >> > > >> > > >> >> > > > > > > > > > >> I don't see anything under > >> > prerequisites > >> > > >> that > >> > > >> > > >> would > >> > > >> > > >> >> be a > >> > > >> > > >> >> > > > > > problem. > >> > > >> > > >> >> > > > > > > We > >> > > >> > > >> >> > > > > > > > > > >> looked > >> > > >> > > >> >> > > > > > > > > > >> at the ASF licencing compatibility > >> as > >> > > part > >> > > >> of > >> > > >> > > >> >> becoming > >> > > >> > > >> >> > an > >> > > >> > > >> >> > > > > > > incubator > >> > > >> > > >> >> > > > > > > > > > >> project. > >> > > >> > > >> >> > > > > > > > > > >> > >> > > >> > > >> >> > > > > > > > > > >> Please focus on the packaging > during > >> > the > >> > > >> next > >> > > >> > > >> weeks. > >> > > >> > > >> >> > Since > >> > > >> > > >> >> > > > the > >> > > >> > > >> >> > > > > > > work > >> > > >> > > >> >> > > > > > > > > will > >> > > >> > > >> >> > > > > > > > > > >> be > >> > > >> > > >> >> > > > > > > > > > >> part of Bigtop, related > discussions > >> and > >> > > >> JIRAs > >> > > >> > > >> should > >> > > >> > > >> >> > also > >> > > >> > > >> >> > > be > >> > > >> > > >> >> > > > > > > there. > >> > > >> > > >> >> > > > > > > > > > >> > >> > > >> > > >> >> > > > > > > > > > >> Would be good to have the > packaging > >> in > >> > > >> place > >> > > >> > by > >> > > >> > > >> end > >> > > >> > > >> >> Feb. > >> > > >> > > >> >> > > > > > > > > > >> > >> > > >> > > >> >> > > > > > > > > > >> Thanks, > >> > > >> > > >> >> > > > > > > > > > >> Thomas > >> > > >> > > >> >> > > > > > > > > > >> > >> > > >> > > >> >> > > > > > > > > > >> > >> > > >> > > >> >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, > >> > Chinmay > >> > > >> > > >> Kolhatkar < > >> > > >> > > >> >> > > > > > > > > > >> [email protected]> > >> > > >> > > >> >> > > > > > > > > > >> wrote: > >> > > >> > > >> >> > > > > > > > > > >> > >> > > >> > > >> >> > > > > > > > > > >> > Hi All, > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> > > >> >> > > > > > > > > > >> > We're planning a work on adding > >> > Apache > >> > > >> Apex > >> > > >> > > as a > >> > > >> > > >> >> > > component > >> > > >> > > >> >> > > > > to > >> > > >> > > >> >> > > > > > > > Apache > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop. > >> > > >> > > >> >> > > > > > > > > > >> > Bigtop is the packaging system > for > >> > the > >> > > >> > Apache > >> > > >> > > >> big > >> > > >> > > >> >> data > >> > > >> > > >> >> > > > > > > ecosystem. > >> > > >> > > >> >> > > > > > > > > > >> Several > >> > > >> > > >> >> > > > > > > > > > >> > Hadoop distros use it, most > >> recently > >> > > EMR. > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> > > >> >> > > > > > > > > > >> > Here is the tracking Jira task > in > >> > > >> APEXCORE > >> > > >> > for > >> > > >> > > >> the > >> > > >> > > >> >> > same: > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> > > >> https://issues.apache.org/jira/browse/APEXCORE-331 > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> > > >> >> > > > > > > > > > >> > Proposed plan of execution is as > >> > > follows: > >> > > >> > > >> >> > > > > > > > > > >> > *Step 1) Handle prerequisites* > >> > > >> > > >> >> > > > > > > > > > >> > Apache bigtop has some hard and > >> soft > >> > > >> > > expectation > >> > > >> > > >> >> for > >> > > >> > > >> >> > new > >> > > >> > > >> >> > > > > > > > components > >> > > >> > > >> >> > > > > > > > > to > >> > > >> > > >> >> > > > > > > > > > >> get > >> > > >> > > >> >> > > > > > > > > > >> > integrated into Bigtop. > >> > > >> > > >> >> > > > > > > > > > >> > Here is the list of it: > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> > > >> >> > > > > > > > > > >> > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > >> > > >> > > >> >> > > > > > > > >> > > >> > > >> >> > > > > > > >> > > >> > > >> >> > > > > > >> > > >> > > >> >> > > > > >> > > >> > > >> >> > > > >> > > >> > > >> >> > > >> > > >> > > >> >> > >> > > >> > > >> > >> > > >> > > > >> > > >> > > >> > > >> > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> > > >> >> > > > > > > > > > >> > Most of them seems to be > standard > >> ASF > >> > > >> based > >> > > >> > > >> >> > > requirements, > >> > > >> > > >> >> > > > > but > >> > > >> > > >> >> > > > > > > few > >> > > >> > > >> >> > > > > > > > > need > >> > > >> > > >> >> > > > > > > > > > >> to > >> > > >> > > >> >> > > > > > > > > > >> > be checked for: > >> > > >> > > >> >> > > > > > > > > > >> > 1. Software projects are > expected > >> to > >> > be > >> > > >> > > Licensed > >> > > >> > > >> >> under > >> > > >> > > >> >> > > > > Apache > >> > > >> > > >> >> > > > > > > > > License, > >> > > >> > > >> >> > > > > > > > > > >> > Version 2.0 (and their > >> dependencies > >> > are > >> > > >> > > expected > >> > > >> > > >> >> to be > >> > > >> > > >> >> > > > > > > compatible > >> > > >> > > >> >> > > > > > > > > with > >> > > >> > > >> >> > > > > > > > > > >> this > >> > > >> > > >> >> > > > > > > > > > >> > license) > >> > > >> > > >> >> > > > > > > > > > >> > - Apex is under ASL 2.0 but > >> need > >> > to > >> > > >> > check > >> > > >> > > if > >> > > >> > > >> >> > > > > dependencies > >> > > >> > > >> >> > > > > > of > >> > > >> > > >> >> > > > > > > > > Apex > >> > > >> > > >> >> > > > > > > > > > >> are > >> > > >> > > >> >> > > > > > > > > > >> > compatible with ASL 2.0. This I > >> guess > >> > > >> would > >> > > >> > > be a > >> > > >> > > >> >> > > > > verification > >> > > >> > > >> >> > > > > > > > check. > >> > > >> > > >> >> > > > > > > > > > >> > 2. Software projects are > expected > >> to > >> > be > >> > > >> > > >> compatible > >> > > >> > > >> >> > with > >> > > >> > > >> >> > > > all > >> > > >> > > >> >> > > > > of > >> > > >> > > >> >> > > > > > > the > >> > > >> > > >> >> > > > > > > > > > >> > supported platforms that Bigtop > >> > > >> distribution > >> > > >> > > is > >> > > >> > > >> >> > > targeting > >> > > >> > > >> >> > > > > > > > > > >> > - This needs verifying > whether > >> > our > >> > > >> > > software > >> > > >> > > >> >> runs > >> > > >> > > >> >> > > fine > >> > > >> > > >> >> > > > in > >> > > >> > > >> >> > > > > > > > > centos-6 > >> > > >> > > >> >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 > >> > > debian-8 > >> > > >> > > >> >> > opensuse-13.2. > >> > > >> > > >> >> > > > > > > > > > >> > 3. What smoke tests that should > be > >> > > added > >> > > >> for > >> > > >> > > >> >> > deployment. > >> > > >> > > >> >> > > > > > > > > > >> > 4. Identifying the test > artifacts > >> > which > >> > > >> goes > >> > > >> > > >> beyond > >> > > >> > > >> >> > > smoke > >> > > >> > > >> >> > > > > test > >> > > >> > > >> >> > > > > > > > > > >> > - These are basically the > >> > > integration > >> > > >> > > tests > >> > > >> > > >> for > >> > > >> > > >> >> > > > > > verification > >> > > >> > > >> >> > > > > > > > > after > >> > > >> > > >> >> > > > > > > > > > >> the > >> > > >> > > >> >> > > > > > > > > > >> > deployment. This is a soft > >> > requirement, > >> > > >> but > >> > > >> > > aim > >> > > >> > > >> is > >> > > >> > > >> >> to > >> > > >> > > >> >> > > > > achieve > >> > > >> > > >> >> > > > > > > this > >> > > >> > > >> >> > > > > > > > > as > >> > > >> > > >> >> > > > > > > > > > >> well > >> > > >> > > >> >> > > > > > > > > > >> > or at least have explanation why > >> not > >> > to > >> > > >> > > include. > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> > > >> >> > > > > > > > > > >> > If there are any from the link > >> which > >> > > >> > > explicitly > >> > > >> > > >> >> needs > >> > > >> > > >> >> > to > >> > > >> > > >> >> > > > be > >> > > >> > > >> >> > > > > > > > checked > >> > > >> > > >> >> > > > > > > > > > >> other > >> > > >> > > >> >> > > > > > > > > > >> > than above 4, please let us > know. > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> > > >> >> > > > > > > > > > >> > *Step 2) Adding Apex as > component > >> to > >> > > >> Bigtop* > >> > > >> > > >> >> > > > > > > > > > >> > From one of the mail archive of > >> > Bigtop, > >> > > >> it > >> > > >> > was > >> > > >> > > >> >> learnt > >> > > >> > > >> >> > > that > >> > > >> > > >> >> > > > > the > >> > > >> > > >> >> > > > > > > > > bigtop > >> > > >> > > >> >> > > > > > > > > > >> > community want to see the > >> addition of > >> > > new > >> > > >> > > >> >> components > >> > > >> > > >> >> > in > >> > > >> > > >> >> > > > > > phases. > >> > > >> > > >> >> > > > > > > > Here > >> > > >> > > >> >> > > > > > > > > > are > >> > > >> > > >> >> > > > > > > > > > >> > the phases: > >> > > >> > > >> >> > > > > > > > > > >> > 1. Packaging > >> > > >> > > >> >> > > > > > > > > > >> > - This needs creating of > >> package > >> > > i.e. > >> > > >> > rpm > >> > > >> > > & > >> > > >> > > >> deb > >> > > >> > > >> >> > > files. > >> > > >> > > >> >> > > > > > > > > > >> > - documentations/READMEs, > >> > LICENSE, > >> > > >> > > >> DISCLAMER, > >> > > >> > > >> >> > NOTES > >> > > >> > > >> >> > > > etc > >> > > >> > > >> >> > > > > if > >> > > >> > > >> >> > > > > > > any > >> > > >> > > >> >> > > > > > > > > > >> needed. > >> > > >> > > >> >> > > > > > > > > > >> > - Any documentation that > need > >> to > >> > be > >> > > >> > added > >> > > >> > > to > >> > > >> > > >> >> > > > > distribution > >> > > >> > > >> >> > > > > > of > >> > > >> > > >> >> > > > > > > > our > >> > > >> > > >> >> > > > > > > > > > >> > software. > >> > > >> > > >> >> > > > > > > > > > >> > - Any license information of > >> > > >> > dependencies > >> > > >> > > >> >> required > >> > > >> > > >> >> > > to > >> > > >> > > >> >> > > > be > >> > > >> > > >> >> > > > > > > added > >> > > >> > > >> >> > > > > > > > > to > >> > > >> > > >> >> > > > > > > > > > >> > package > >> > > >> > > >> >> > > > > > > > > > >> > 2. Smoke tests (at very least) > >> > > >> > > >> >> > > > > > > > > > >> > - Adding smoke test for > >> > packaging. > >> > > >> > > >> >> > > > > > > > > > >> > 3. Puppet recipes for automatic > >> > > >> deployment > >> > > >> > and > >> > > >> > > >> >> > > > configuration > >> > > >> > > >> >> > > > > > > > > > >> > - Add puppet recipes for > >> > automatic > >> > > >> > > >> deployment > >> > > >> > > >> >> > > > > > > > > > >> > 4. Integration tests > >> > > >> > > >> >> > > > > > > > > > >> > - For verification of > >> > deployments. > >> > > >> > > >> >> > > > > > > > > > >> > 5. license clearance: > >> > > >> > > >> >> > > > > > > > > > >> > Run 'gradle rat' to make > sure > >> all > >> > > new > >> > > >> > > stuff > >> > > >> > > >> is > >> > > >> > > >> >> > > > compliant > >> > > >> > > >> >> > > > > > > with > >> > > >> > > >> >> > > > > > > > > ASF > >> > > >> > > >> >> > > > > > > > > > >> > license requirements. If you add > >> code > >> > > >> > licenses > >> > > >> > > >> >> under > >> > > >> > > >> >> > > > > different > >> > > >> > > >> >> > > > > > > > > > licenses, > >> > > >> > > >> >> > > > > > > > > > >> > those would need to be listed in > >> the > >> > > >> NOTICE. > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> > > >> >> > > > > > > > > > >> > Please share your thoughts on > the > >> > > >> approach. > >> > > >> > > >> >> > > > > > > > > > >> > We'll start corresponding > >> > communication > >> > > >> on > >> > > >> > > >> bigtop > >> > > >> > > >> >> > > mailing > >> > > >> > > >> >> > > > > list > >> > > >> > > >> >> > > > > > > as > >> > > >> > > >> >> > > > > > > > > > well. > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> > > >> >> > > > > > > > > > >> > We have some specific > >> > > >> questions/suggestions > >> > > >> > > >> >> related to > >> > > >> > > >> >> > > > what > >> > > >> > > >> >> > > > > > > should > >> > > >> > > >> >> > > > > > > > > be > >> > > >> > > >> >> > > > > > > > > > >> the > >> > > >> > > >> >> > > > > > > > > > >> > content of the package and what > >> > should > >> > > be > >> > > >> > the > >> > > >> > > >> smoke > >> > > >> > > >> >> > > tests, > >> > > >> > > >> >> > > > > but > >> > > >> > > >> >> > > > > > > in > >> > > >> > > >> >> > > > > > > > > the > >> > > >> > > >> >> > > > > > > > > > >> > interest of not having too much > >> > content > >> > > >> > here, > >> > > >> > > >> we'll > >> > > >> > > >> >> > put > >> > > >> > > >> >> > > > the > >> > > >> > > >> >> > > > > > > > > questions > >> > > >> > > >> >> > > > > > > > > > >> as a > >> > > >> > > >> >> > > > > > > > > > >> > separate mail in this > mailthread. > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> > > >> >> > > > > > > > > > >> > Thanks, > >> > > >> > > >> >> > > > > > > > > > >> > Chinmay. > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> > > >> >> > > > > > > > > > >> > >> > > >> > > >> >> > > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > > >> > > >> > > >> >> > > > > > > > > > >> > > >> > > >> >> > > > > > > > > >> > > >> > > >> >> > > > > > > > >> > > >> > > >> >> > > > > > > >> > > >> > > >> >> > > > > > >> > > >> > > >> >> > > > > >> > > >> > > >> >> > > > >> > > >> > > >> >> > > >> > > >> > > >> >> > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > >> > >> > > > > >> > > > > >> > > > >> > > >> > > > > >
