/usr/lib/apex/bin contains executables related to apex. dtcli, even though its a symlink, for current users its still an executable. Hence for the sake of completion, added dtcli link under /usr/lib/apex/bin.
But if you think otherwise, please let me know. Thanks, Chinmay. On Mon, Feb 15, 2016 at 10:42 AM, Thomas Weise <[email protected]> wrote: > Why the dtcli link under lib? > > > On Sun, Feb 14, 2016 at 9:09 PM, Chinmay Kolhatkar <[email protected]> > wrote: > > > Hmmm... I get your point. In development mode, it tries to find the > hadoop > > executable like following: > > > > > > > > > HADOOP_SEARCH_PATH="${HADOOP_PREFIX}/bin:${HADOOP_HOME}/bin:${PATH}:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:." > > export DT_HADOOP=`PATH=${HADOOP_SEARCH_PATH} && command -v hadoop > > 2>/dev/null` > > > > I will check with bigtop community that what is the best way to ensure > > during installation that hadoop binary will be present in above search > path > > search. > > > > Also, I checked as you suggested, i.e. removed dt-env.sh from package and > > tried to run the dtcli... It works!!! > > > > Thank you for pointing that out Thomas. > > > > Now I agree with you that, we can get rid of conf/dt-env.sh.. As this was > > the only file in conf/, unless someone has anything to include in conf/ > > folder, I'll remove that too then. > > > > Also, this would mean, only dt* that remains in package are dtcli links > to > > apex script added for backward compatibility. > > i.e following: > > > > / > > `-- usr > > |-- bin > > | |-- apex -> ../lib/apex/bin/apex > > | `-- dtcli -> ../lib/apex/bin/apex > > |-- lib > > | `-- apex > > | |-- bin > > | | |-- apex // This is the real script > > | | `-- dtcli -> apex > > > > > > > > Thanks, > > Chinmay. > > > > > > > > On Mon, Feb 15, 2016 at 10:22 AM, Thomas Weise <[email protected]> > > wrote: > > > > > I'm not convinced dt-env.sh is even needed. In development mode, the > file > > > doesn't exist but it will still find Hadoop. > > > > > > The absolute path to hadoop script was introduced to deal with the > layout > > > of specific distros. You will know whether that applies to Bigtop or > not. > > > > > > Thomas > > > > > > On Sun, Feb 14, 2016 at 8:16 PM, Chinmay Kolhatkar <[email protected] > > > > > wrote: > > > > > > > Hi Thomas, > > > > > > > > There are only 2 things that are required from dt-env.sh: > > > > 1. DT_HADOOP variable which is exported by dt-env.sh This variable > has > > > > absolute path for hadoop binary > > > > 2. A find_hadoop() method which find the path of hadoop binary. > > > > > > > > JAVA_HOME is not required for this. > > > > > > > > The idea is not to change the dtcli. Current dtcli (in apex > > > > 3.3.0-incubating src code), sources dt-env.sh for above 2 > information. > > > > Hence included dt-env.sh > > > > > > > > If the community agrees, we can make dtcli more generic so that in > > future > > > > packing of bigtop, dt-env.sh can be completely excluded OR at least > > > change > > > > the name to apex-env.sh. > > > > > > > > Thanks, > > > > Chinmay. > > > > > > > > > > > > > > > > On Sun, Feb 14, 2016 at 7:55 AM, Thomas Weise < > [email protected]> > > > > wrote: > > > > > > > > > Chinmay, > > > > > > > > > > Can you please specify what settings in dt-env.sh are required in a > > > > bigtop > > > > > environment? > > > > > > > > > > Location of hadoop and JAVA_HOME should be provided by the stack > > > already? > > > > > > > > > > Thanks, > > > > > Thomas > > > > > > > > > > > > > > > On Fri, Feb 12, 2016 at 7:04 PM, Chinmay Kolhatkar < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > Here is a suggestion I have related to name change: > > > > > > > > > > > > 1. While packaging we change the name of "dtcli" script to > "apex". > > > This > > > > > is > > > > > > done so that in future when we change the name to "apex", users > of > > > > bigtop > > > > > > apex don't have to transition much. > > > > > > 2. We also keep a symlink named dtcli which points to apex > script. > > > This > > > > > is > > > > > > for backward compatibility. > > > > > > 3. We still keep conf/dt-env.sh for first integration in bigtop. > > This > > > > is > > > > > in > > > > > > the interest of not changing the content of dtcli or apex script. > > > > > > 4. In apex 3.4.0, we take care of all these naming related > changes > > > and > > > > > > update bigtop repository later to remove references to dtcli all > > > > > together. > > > > > > > > > > > > Please share your thoughts on above approach. > > > > > > > > > > > > Also, please share what could be the man page content for "apex". > > > > > > > > > > > > Thanks, > > > > > > Chinmay. > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Feb 13, 2016 at 8:17 AM, Chinmay Kolhatkar < > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > > > Yes.. It'll be build from source tar downloaded from one of the > > > > apache > > > > > > > mirror: apache.osuosl.org > > > > > > > > > > > > > > The reason for dt-env.sh is it's sourced from dtcli. This > leaves > > > us 3 > > > > > > > options: > > > > > > > If we should not have any dt-env.sh, dtcli would need a change > > > while > > > > > > > packaging. > > > > > > > > > > > > > > But if we want to use dtcli as it is, then we would need > > dt-env.sh > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Feb 13, 2016 at 8:07 AM, Thomas Weise < > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > >> Looks good overall, though there shouldn't be any dt-env.sh > > > > > > >> > > > > > > >> What will the bigtop package be built from, the source tar? > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> On Fri, Feb 12, 2016 at 6:31 PM, Chinmay Kolhatkar < > > > > > [email protected]> > > > > > > >> wrote: > > > > > > >> > > > > > > >> > Hi Thomas, > > > > > > >> > > > > > > > >> > Thanks for the feedback. > > > > > > >> > > > > > > > >> > First, we're not changing the name anywhere. We'll follow > what > > > > > > >> currently is > > > > > > >> > in source tarball for 3.3.0-incubating version. > > > > > > >> > > > > > > > >> > Secondly, I've mentioned a directory structure below which > is > > > > inline > > > > > > >> with > > > > > > >> > other existing integrations with bigtop. > > > > > > >> > > > > > > > >> > The need for each file is as follows: > > > > > > >> > 1. bin/dtcli -> CLI for apex picked from > > > engine/src/main/scripts/ > > > > of > > > > > > >> source > > > > > > >> > code apex 3.3.0-incubating. > > > > > > >> > > > > > > > >> > 2. conf/dt-env.sh -> This is source from dtcli. This > searches > > > for > > > > > > hadoop > > > > > > >> > binary path and exports an env variable for dtcli to use it. > > > > > > >> > > > > > > > >> > 3. lib/ -> Libraries for CLI. This list is extracted in a > > > similar > > > > > way > > > > > > to > > > > > > >> > how DT Community Edition finds dependency jars. > > > > > > >> > Please note that this is the first iteration list of jars. > I'm > > > > > trying > > > > > > to > > > > > > >> > narrow this down to only those which are really required. > > > > > > >> > The test that I'm running to check if dtcli runs fine with > > given > > > > > > >> dependency > > > > > > >> > is to launch+shutdown+kill+status for pi demo. > > > > > > >> > > > > > > > >> > Above is the directory structure which is required for CLI > to > > > > work. > > > > > > >> > What aniruddha and pradeep has mentioned are some additional > > > files > > > > > > which > > > > > > >> > makes the like of administrator easier. For eg. /etc/skel > > > > > > >> > > > > > > > >> > Please let me know if this is inline with what you're > > thinking. > > > > > > >> > > > > > > > >> > Thanks, > > > > > > >> > Chinmay. > > > > > > >> > > > > > > > >> > > > > > > > >> > . > > > > > > >> > |-- bin > > > > > > >> > | `-- dtcli > > > > > > >> > |-- conf > > > > > > >> > | `-- dt-env.sh > > > > > > >> > `-- lib > > > > > > >> > |-- ant-1.9.2.jar > > > > > > >> > |-- ant-launcher-1.9.2.jar > > > > > > >> > |-- apex-api-3.3.0-incubating.jar > > > > > > >> > |-- apex-bufferserver-3.3.0-incubating.jar > > > > > > >> > |-- apex-common-3.3.0-incubating.jar > > > > > > >> > |-- apex-engine.jar > > > > > > >> > |-- apex-engine-tests.jar > > > > > > >> > |-- async-http-client-1.7.20.jar > > > > > > >> > |-- bval-core-0.5.jar > > > > > > >> > |-- bval-jsr303-0.5.jar > > > > > > >> > |-- commons-beanutils-1.8.3.jar > > > > > > >> > |-- commons-codec-1.10.jar > > > > > > >> > |-- commons-lang3-3.1.jar > > > > > > >> > |-- commons-logging-1.1.3.jar > > > > > > >> > |-- grizzly-http-servlet-2.1.2.jar > > > > > > >> > |-- hadoop-common-2.2.0-tests.jar > > > > > > >> > |-- httpclient-4.3.5.jar > > > > > > >> > |-- httpcore-4.3.2.jar > > > > > > >> > |-- jackson-core-asl-1.9.2.jar > > > > > > >> > |-- jackson-mapper-asl-1.9.2.jar > > > > > > >> > |-- javax.servlet-3.1.jar > > > > > > >> > |-- javax.servlet-api-3.0.1.jar > > > > > > >> > |-- jersey-apache-client4-1.9.jar > > > > > > >> > |-- jersey-client-1.9.jar > > > > > > >> > |-- jetty-http-8.1.10.v20130312.jar > > > > > > >> > |-- jetty-io-8.1.10.v20130312.jar > > > > > > >> > |-- jetty-util-8.1.10.v20130312.jar > > > > > > >> > |-- jetty-websocket-8.1.10.v20130312.jar > > > > > > >> > |-- jline-2.11.jar > > > > > > >> > |-- kryo-2.24.0.jar > > > > > > >> > |-- mbassador-1.1.9.jar > > > > > > >> > |-- minlog-1.2.jar > > > > > > >> > |-- netlet-1.2.0.jar > > > > > > >> > |-- netty-3.6.6.Final.jar > > > > > > >> > |-- objenesis-2.1.jar > > > > > > >> > |-- validation-api-1.1.0.Final.jar > > > > > > >> > |-- xbean-asm5-shaded-4.3.jar > > > > > > >> > `-- zip4j-1.3.2.jar > > > > > > >> > > > > > > > >> > 3 directories, 40 files > > > > > > >> > > > > > > > >> > > > > > > > >> > On Sat, Feb 13, 2016 at 3:24 AM, Thomas Weise < > > > > > [email protected] > > > > > > > > > > > > > >> > wrote: > > > > > > >> > > > > > > > >> > > Chinmay, > > > > > > >> > > > > > > > > >> > > Before discussing where to put the files, let's make sure > > they > > > > are > > > > > > >> really > > > > > > >> > > needed for the operation of the CLI. As for names, > anything > > > new > > > > > > needs > > > > > > >> to > > > > > > >> > > reflect Apex in the name and should follow common > > conventions, > > > > > > >> especially > > > > > > >> > > in Bigtop where there are many existing integrations to > look > > > at. > > > > > > >> > > > > > > > > >> > > Thanks, > > > > > > >> > > Thomas > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > On Wed, Feb 10, 2016 at 8:07 AM, Chinmay Kolhatkar < > > > > > > >> > > [email protected]> > > > > > > >> > > wrote: > > > > > > >> > > > > > > > > >> > > > Really good points Pradeep and Aniruddha. > > > > > > >> > > > > > > > > > >> > > > 1. I believe we won't need to change the dtcli > considering > > > it > > > > > > works > > > > > > >> > with > > > > > > >> > > DT > > > > > > >> > > > Community edition. We can keep the directory structure > > > similar > > > > > to > > > > > > >> that. > > > > > > >> > > > dt-env.sh has variables which contains information > > required > > > > for > > > > > > >> dtcli > > > > > > >> > to > > > > > > >> > > > launch. > > > > > > >> > > > > > > > > > >> > > > 2. Let me check with bigtop community that whether they > > > > > facilitate > > > > > > >> the > > > > > > >> > > > installation of rpms/debs before user is created. In > > either > > > > > case, > > > > > > >> > current > > > > > > >> > > > dtcli creates a .dt folder in home directory. Also > before > > > > > putting > > > > > > >> > > anything > > > > > > >> > > > in /etc/skel we need to define what are the default > > contents > > > > > that > > > > > > >> > should > > > > > > >> > > go > > > > > > >> > > > to ~/.dt/ folder. If there is no defaults, probably we > > > should > > > > > not > > > > > > >> > > > explicitly add it in /etc/skel. > > > > > > >> > > > > > > > > > >> > > > 3. /etc/profile.d approach looks nice. This way > contents > > of > > > > > > >> dt-env.sh > > > > > > >> > > are > > > > > > >> > > > present as env variables. But I see a catch there. > Adding > > > > > > dt-env.sh > > > > > > >> to > > > > > > >> > > > /etc/profile.d would make all the variables available at > > > > runtime > > > > > > all > > > > > > >> > the > > > > > > >> > > > time. I feel a little skeptical about that. Maybe a > > possible > > > > > > >> collision > > > > > > >> > > can > > > > > > >> > > > occur with other application vars. > > > > > > >> > > > Moreover, current dtcli does source "../conf/dt-env.sh" > > and > > > > > > >> > > > "~/.dt/dt-env.sh". So those variables are anyway > > available. > > > > > > >> > > > > > > > > > >> > > > @Aniruddha, the Jira and PR effort is happening at > Bigtop > > > > > > >> > > Jira/repository. > > > > > > >> > > > Packaging related code usually goes there. (That's what > > all > > > > the > > > > > > >> > > components > > > > > > >> > > > in bigtop does). > > > > > > >> > > > Having said that, once a PR is created, I'll be sharing > > the > > > > link > > > > > > of > > > > > > >> PR > > > > > > >> > > > here, so that apex community as well can review it. > > > > > > >> > > > > > > > > > >> > > > Thanks, > > > > > > >> > > > Chinmay. > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > On Wed, Feb 10, 2016 at 7:57 PM, Aniruddha Thombare < > > > > > > >> > > > [email protected]> wrote: > > > > > > >> > > > > > > > > > >> > > > > +1 on suggestions and approach. > > > > > > >> > > > > We may need to iron out details about exact paths etc. > > > > > > >> > > > > Which can be done on jira / PR comments. > > > > > > >> > > > > Is that right @dev? > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > On Wed, 10 Feb 2016 7:53 pm Pradeep A. Dalvi < > > > > > > >> > [email protected]> > > > > > > >> > > > > wrote: > > > > > > >> > > > > > > > > > > >> > > > > > Inline comments... > > > > > > >> > > > > > > > > > > > >> > > > > > On Wed, Feb 10, 2016 at 2:51 PM, Chinmay Kolhatkar < > > > > > > >> > > > > > [email protected]> > > > > > > >> > > > > > wrote: > > > > > > >> > > > > > > > > > > > >> > > > > > > @Thomas, Not all the jar present in DT community > > > edition > > > > > > will > > > > > > >> be > > > > > > >> > > > > included > > > > > > >> > > > > > > there. > > > > > > >> > > > > > > > > > > > > >> > > > > > > Here is the list of jars I found from common > > > > dependencies > > > > > > >> between > > > > > > >> > > of > > > > > > >> > > > > apex > > > > > > >> > > > > > > and DT Community edition: > > > > > > >> > > > > > > netlet-1.2.0.jar > > > > > > >> > > > > > > kryo-2.24.0.jar > > > > > > >> > > > > > > jackson-core-asl-1.9.2.jar > > > > > > >> > > > > > > jackson-mapper-asl-1.9.2.jar > > > > > > >> > > > > > > async-http-client-1.7.20.jar > > > > > > >> > > > > > > netty-3.6.6.Final.jar > > > > > > >> > > > > > > validation-api-1.1.0.Final.jar > > > > > > >> > > > > > > bval-jsr303-0.5.jar > > > > > > >> > > > > > > bval-core-0.5.jar > > > > > > >> > > > > > > commons-lang3-3.1.jar > > > > > > >> > > > > > > commons-beanutils-1.8.3.jar > > > > > > >> > > > > > > httpclient-4.3.5.jar > > > > > > >> > > > > > > commons-codec-1.10.jar > > > > > > >> > > > > > > zip4j-1.3.2.jar > > > > > > >> > > > > > > jetty-websocket-8.1.10.v20130312.jar > > > > > > >> > > > > > > xbean-asm5-shaded-4.3.jar > > > > > > >> > > > > > > jersey-apache-client4-1.9.jar > > > > > > >> > > > > > > jline-2.11.jar > > > > > > >> > > > > > > ant-1.9.2.jar > > > > > > >> > > > > > > ant-launcher-1.9.2.jar > > > > > > >> > > > > > > mbassador-1.1.9.jar > > > > > > >> > > > > > > jackson-jaxrs-1.9.2.jar > > > > > > >> > > > > > > jackson-xc-1.9.2.jar > > > > > > >> > > > > > > hadoop-common-2.2.0-tests.jar > > > > > > >> > > > > > > > > > > > > >> > > > > > > Ofcourse, I'll be running some tests do check that > > > dtcli > > > > > > works > > > > > > >> > > > properly > > > > > > >> > > > > > for > > > > > > >> > > > > > > launch+shutdown+kill of apps with only these > > libraries > > > > > > >> present in > > > > > > >> > > > > > isolation > > > > > > >> > > > > > > without dependency on local m2. > > > > > > >> > > > > > > I'm believe that there are unwanted jars which are > > > used > > > > > for > > > > > > >> > compile > > > > > > >> > > > > time > > > > > > >> > > > > > > dependency and not runtime in above list which I > can > > > > drop > > > > > to > > > > > > >> keep > > > > > > >> > > > > package > > > > > > >> > > > > > > size minimal. > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > @Anniruddha, > > > > > > >> > > > > > > 1. dt-env.sh -> I'm not sure if all of the > dt-env.sh > > > is > > > > > > >> required. > > > > > > >> > > > Only > > > > > > >> > > > > > > required values I see in it are classpath. and > dtcli > > > is > > > > > > >> capable > > > > > > >> > of > > > > > > >> > > > > > building > > > > > > >> > > > > > > it on the fly. > > > > > > >> > > > > > > > > > > > > >> > > > > > > 2. dt-site.xml -> Yes, we'll have to see how dtcli > > can > > > > > find > > > > > > >> the > > > > > > >> > jar > > > > > > >> > > > > path, > > > > > > >> > > > > > > dt-env.sh and all such conf files. If we need a > > change > > > > in > > > > > > >> dtcli, > > > > > > >> > > then > > > > > > >> > > > > > > community's opinion is required for whether dtcli > > > should > > > > > > >> change > > > > > > >> > in > > > > > > >> > > > our > > > > > > >> > > > > > repo > > > > > > >> > > > > > > or a copy of that with required changes exist in > > > bigtop > > > > > repo > > > > > > >> > until > > > > > > >> > > we > > > > > > >> > > > > > make > > > > > > >> > > > > > > the dtcli generic enough. > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > Can we have these file paths set using environment > > > > variables > > > > > > >> with > > > > > > >> > > some > > > > > > >> > > > > > default values in dtcli? > > > > > > >> > > > > > Then we can set such params in dt-env.sh. > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > 3. dt-sited.xml -> How is this file different from > > > > > > >> dt-site.xml? > > > > > > >> > Not > > > > > > >> > > > > sure > > > > > > >> > > > > > if > > > > > > >> > > > > > > adding in /etc/skel would help. Files & Dirs from > > > > > /etc/skel > > > > > > >> are > > > > > > >> > > > copied > > > > > > >> > > > > to > > > > > > >> > > > > > > home of new user when useradd program is called. > But > > > for > > > > > > >> existing > > > > > > >> > > > users > > > > > > >> > > > > > > this won't be of any use. Moreover, dtcli creates > > > ~/.dt/ > > > > > on > > > > > > >> the > > > > > > >> > fly > > > > > > >> > > > if > > > > > > >> > > > > > not > > > > > > >> > > > > > > encountered for the first time. Again correct me > if > > > I'm > > > > > > wrong. > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > Yes, you are right. However this would probably be > > > > necessary > > > > > > >> step > > > > > > >> > for > > > > > > >> > > > > > rpm/deb, as they may also get installed during OS > > > install > > > > > and > > > > > > >> > before > > > > > > >> > > > user > > > > > > >> > > > > > accounts were created. > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > 4. Changing bashrc & bash_profile -> I'm not sure > > its > > > > the > > > > > > best > > > > > > >> > > idea. > > > > > > >> > > > > > We'll > > > > > > >> > > > > > > anyway put dtcli in location which is by default > > > present > > > > > in > > > > > > >> path > > > > > > >> > > i.e. > > > > > > >> > > > > > > /usr/bin. > > > > > > >> > > > > > > > > > > > >> > > > > > For other env variables specific to apex, I'll > prefer > > to > > > > use > > > > > > >> > > dt-env.sh > > > > > > >> > > > > and > > > > > > >> > > > > > > source it in dtcli rather than changing bashrc > > etc... > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > Instead, coping dt-env.sh to /etc/profile.d/ shall > be > > > the > > > > > > >> cleaner > > > > > > >> > way > > > > > > >> > > > to > > > > > > >> > > > > > achieve the same. > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > Thanks, > > > > > > >> > > > > > > Chinmay. > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > On Wed, Feb 10, 2016 at 1:07 PM, Aniruddha > Thombare > > < > > > > > > >> > > > > > > [email protected]> wrote: > > > > > > >> > > > > > > > > > > > > >> > > > > > > > Hi, > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > @Chinmay, > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > We need to consider following: > > > > > > >> > > > > > > > System wide default config files can be located > at > > > > > > following > > > > > > >> > > > > locations: > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > /etc/apex/dt-env.sh (we may have to change > dtcli > > > > > > behaviour > > > > > > >> on > > > > > > >> > > how > > > > > > >> > > > it > > > > > > >> > > > > > > finds > > > > > > >> > > > > > > > those locations) > > > > > > >> > > > > > > > /etc/apex/dt-site.xml (we may have to change > dtcli > > > > > > >> behaviour on > > > > > > >> > > how > > > > > > >> > > > > it > > > > > > >> > > > > > > > finds those locations) > > > > > > >> > > > > > > > /etc/skel/.dt/dt-sited.xml and other files (for > > new > > > > > users > > > > > > >> that > > > > > > >> > > will > > > > > > >> > > > > be > > > > > > >> > > > > > > > created in system in future.) > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > We may also have to modify bashrc / bashprofile > > for > > > > > > >> population > > > > > > >> > > any > > > > > > >> > > > > > > > variables that are required. > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > @dev, please put in your comments / suggestion. > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > Thanks, > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > Aniruddha > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > On Wed, Feb 10, 2016 at 1:16 AM, Thomas Weise < > > > > > > >> > > > > [email protected]> > > > > > > >> > > > > > > > wrote: > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > --> > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > On Tue, Feb 9, 2016 at 9:05 AM, Chinmay > > Kolhatkar > > > < > > > > > > >> > > > > > > > [email protected] > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > wrote: > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > Hello Everyone!! > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > Continuing with packaging effort (rpm+deb) > of > > > > apex, > > > > > > here > > > > > > >> > are > > > > > > >> > > > some > > > > > > >> > > > > > > > > proposals > > > > > > >> > > > > > > > > > about package structure etc.. > > > > > > >> > > > > > > > > > Before posting it on bbigtop mailing list, I > > > have > > > > > some > > > > > > >> > > question > > > > > > >> > > > > for > > > > > > >> > > > > > > > apex > > > > > > >> > > > > > > > > > community. > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > Proposed Directory structure of apex package > > for > > > > > both > > > > > > >> deb & > > > > > > >> > > > rpm: > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > /usr/lib/apex/bin/dtcli > > > > > > >> > > > > > > > > > /usr/lib/apex/lib/apex-api-<version>.jar > > > > > > >> > > > > > > > > > /usr/lib/apex/lib/apex-engine-<version>.jar > > > > > > >> > > > > > > > > > > > > /usr/lib/apex/lib/apex-bufferserver-<version>.jar > > > > > > >> > > > > > > > > > /usr/lib/apex/lib/apex-common-<version>.jar > > > > > > >> > > > > > > > > > /usr/lib/apex/lib/<other dependent jars> > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli > > > > > > >> > > > > > > > > > /usr/share/doc/man/man1/dtcli.1.gz > > > > > > >> > > > > > > > > > /usr/share/doc/apex/license/LICENSE.txt > > > > > > >> > > > > > > > > > > > > /usr/share/doc/apex/license/<package>-LICENSE.txt > > > > > > >> > > > > > > > > > /usr/share/doc/apex/CHANGELOG > > > > > > >> > > > > > > > > > /usr/share/doc/apex/NOTICE > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > <version> = 3.3.0-incubating. > > > > > > >> > > > > > > > > > <other dependent jars> = All the 3rd party > > jars > > > > > which > > > > > > >> are > > > > > > >> > > > > required > > > > > > >> > > > > > > for > > > > > > >> > > > > > > > > apex > > > > > > >> > > > > > > > > > to run. Usually the dependencies are > packaged > > as > > > > > part > > > > > > of > > > > > > >> > > > rpm/deb > > > > > > >> > > > > by > > > > > > >> > > > > > > any > > > > > > >> > > > > > > > > > software in bigtop. > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > Can you specify what those jars are in Bigtop > > > > context? > > > > > > >> Same > > > > > > >> > as > > > > > > >> > > > > > shipped > > > > > > >> > > > > > > > with > > > > > > >> > > > > > > > > DT community addition under lib/ ? > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > <package LICENSE> = Licenses of > corresponding > > of > > > > 3rd > > > > > > >> party > > > > > > >> > > jars > > > > > > >> > > > > > which > > > > > > >> > > > > > > > > needs > > > > > > >> > > > > > > > > > to included while packaging. > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > Questions related to this: > > > > > > >> > > > > > > > > > 1. Should we call the cli of apex as "apex" > > > > instead > > > > > of > > > > > > >> > > "dtcli" > > > > > > >> > > > in > > > > > > >> > > > > > > > bigtop > > > > > > >> > > > > > > > > > package? > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > I think we should keep the name until we are > > able > > > to > > > > > > >> change > > > > > > >> > it > > > > > > >> > > in > > > > > > >> > > > > > Apex. > > > > > > >> > > > > > > > > Otherwise this may get confusing. > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > 2. I see that all softwares in bigtop have > man > > > > page > > > > > > for > > > > > > >> > their > > > > > > >> > > > > > > > > executables. > > > > > > >> > > > > > > > > > I think we should have it too for dtcli. Is > > > there > > > > > any > > > > > > >> > > > > documentation > > > > > > >> > > > > > > > > which I > > > > > > >> > > > > > > > > > can convert to man page? or can I use output > > of > > > > > "dtcli > > > > > > >> > > --help"? > > > > > > >> > > > > > > > > > 3. Do we want to call version of apex in > > Bigtop > > > as > > > > > > >> 3.3.0 OR > > > > > > >> > > > > > > > > > 3.3.0-incubating? > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > Has to be -incubating > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > 4. Is it ok for apex package of bigtop to > > depend > > > > on > > > > > > >> 2.7.1 > > > > > > >> > > > > version > > > > > > >> > > > > > of > > > > > > >> > > > > > > > > > bigtop hadoop? Any problems that we see with > > > this > > > > > > >> > dependency? > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > Should work without changed. I thought we > > > certified > > > > > > >> against > > > > > > >> > > > 2.7.0? > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > 5. Following is the apache mirror from which > > > > bigtop > > > > > > will > > > > > > >> > pick > > > > > > >> > > > the > > > > > > >> > > > > > > apex > > > > > > >> > > > > > > > > > source for compilation and packaging. Please > > > > correct > > > > > > if > > > > > > >> > > > > incorrect: > > > > > > >> > > > > > > > > > > > > > > > >> > > > http://apache.osuosl.org/incubator/apex/v3.3.0-incubating/ > > > > > > >> > > > > > > > > > NOTE: apache.osuosl.org is the mirror > > used > > > by > > > > > all > > > > > > >> the > > > > > > >> > > > > > softwares > > > > > > >> > > > > > > in > > > > > > >> > > > > > > > > > bigtop. > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > Please share your opinion. > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > Thanks, > > > > > > >> > > > > > > > > > Chinmay. > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > On Mon, Feb 8, 2016 at 11:19 AM, Chinmay > > > > Kolhatkar < > > > > > > >> > > > > > > > > > [email protected]> > > > > > > >> > > > > > > > > > wrote: > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > Yes.. Starting to work on the packaging. > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > I've already started discussion on bigtop > > dev > > > > > > mailing > > > > > > >> > list > > > > > > >> > > > for > > > > > > >> > > > > > > > > > > integration. Also created a Jira for the > > same. > > > > For > > > > > > >> this > > > > > > >> > > > > > communities > > > > > > >> > > > > > > > > > > reference, here is the bigtop Jira: > > > BIGTOP-2313. > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > On Mon, Feb 8, 2016 at 1:13 AM, Thomas > > Weise < > > > > > > >> > > > > > > [email protected] > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > wrote: > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > >> Chinmay, > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> I don't see anything under prerequisites > > that > > > > > would > > > > > > >> be a > > > > > > >> > > > > > problem. > > > > > > >> > > > > > > We > > > > > > >> > > > > > > > > > >> looked > > > > > > >> > > > > > > > > > >> at the ASF licencing compatibility as > part > > of > > > > > > >> becoming > > > > > > >> > an > > > > > > >> > > > > > > incubator > > > > > > >> > > > > > > > > > >> project. > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> Please focus on the packaging during the > > next > > > > > > weeks. > > > > > > >> > Since > > > > > > >> > > > the > > > > > > >> > > > > > > work > > > > > > >> > > > > > > > > will > > > > > > >> > > > > > > > > > >> be > > > > > > >> > > > > > > > > > >> part of Bigtop, related discussions and > > JIRAs > > > > > > should > > > > > > >> > also > > > > > > >> > > be > > > > > > >> > > > > > > there. > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> Would be good to have the packaging in > > place > > > by > > > > > end > > > > > > >> Feb. > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> Thanks, > > > > > > >> > > > > > > > > > >> Thomas > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> On Thu, Feb 4, 2016 at 10:14 AM, Chinmay > > > > > Kolhatkar > > > > > > < > > > > > > >> > > > > > > > > > >> [email protected]> > > > > > > >> > > > > > > > > > >> wrote: > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > Hi All, > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > We're planning a work on adding Apache > > Apex > > > > as > > > > > a > > > > > > >> > > component > > > > > > >> > > > > to > > > > > > >> > > > > > > > Apache > > > > > > >> > > > > > > > > > >> > Bigtop. > > > > > > >> > > > > > > > > > >> > Bigtop is the packaging system for the > > > Apache > > > > > big > > > > > > >> data > > > > > > >> > > > > > > ecosystem. > > > > > > >> > > > > > > > > > >> Several > > > > > > >> > > > > > > > > > >> > Hadoop distros use it, most recently > EMR. > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > Here is the tracking Jira task in > > APEXCORE > > > > for > > > > > > the > > > > > > >> > same: > > > > > > >> > > > > > > > > > >> > > > > > > > https://issues.apache.org/jira/browse/APEXCORE-331 > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > Proposed plan of execution is as > follows: > > > > > > >> > > > > > > > > > >> > *Step 1) Handle prerequisites* > > > > > > >> > > > > > > > > > >> > Apache bigtop has some hard and soft > > > > > expectation > > > > > > >> for > > > > > > >> > new > > > > > > >> > > > > > > > components > > > > > > >> > > > > > > > > to > > > > > > >> > > > > > > > > > >> get > > > > > > >> > > > > > > > > > >> > integrated into Bigtop. > > > > > > >> > > > > > > > > > >> > Here is the list of it: > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > Most of them seems to be standard ASF > > based > > > > > > >> > > requirements, > > > > > > >> > > > > but > > > > > > >> > > > > > > few > > > > > > >> > > > > > > > > need > > > > > > >> > > > > > > > > > >> to > > > > > > >> > > > > > > > > > >> > be checked for: > > > > > > >> > > > > > > > > > >> > 1. Software projects are expected to be > > > > > Licensed > > > > > > >> under > > > > > > >> > > > > Apache > > > > > > >> > > > > > > > > License, > > > > > > >> > > > > > > > > > >> > Version 2.0 (and their dependencies are > > > > > expected > > > > > > >> to be > > > > > > >> > > > > > > compatible > > > > > > >> > > > > > > > > with > > > > > > >> > > > > > > > > > >> this > > > > > > >> > > > > > > > > > >> > license) > > > > > > >> > > > > > > > > > >> > - Apex is under ASL 2.0 but need to > > > check > > > > > if > > > > > > >> > > > > dependencies > > > > > > >> > > > > > of > > > > > > >> > > > > > > > > Apex > > > > > > >> > > > > > > > > > >> are > > > > > > >> > > > > > > > > > >> > compatible with ASL 2.0. This I guess > > would > > > > be > > > > > a > > > > > > >> > > > > verification > > > > > > >> > > > > > > > check. > > > > > > >> > > > > > > > > > >> > 2. Software projects are expected to be > > > > > > compatible > > > > > > >> > with > > > > > > >> > > > all > > > > > > >> > > > > of > > > > > > >> > > > > > > the > > > > > > >> > > > > > > > > > >> > supported platforms that Bigtop > > > distribution > > > > is > > > > > > >> > > targeting > > > > > > >> > > > > > > > > > >> > - This needs verifying whether our > > > > software > > > > > > >> runs > > > > > > >> > > fine > > > > > > >> > > > in > > > > > > >> > > > > > > > > centos-6 > > > > > > >> > > > > > > > > > >> > centos-7 fedora-20 ubuntu-14.04 > debian-8 > > > > > > >> > opensuse-13.2. > > > > > > >> > > > > > > > > > >> > 3. What smoke tests that should be > added > > > for > > > > > > >> > deployment. > > > > > > >> > > > > > > > > > >> > 4. Identifying the test artifacts which > > > goes > > > > > > beyond > > > > > > >> > > smoke > > > > > > >> > > > > test > > > > > > >> > > > > > > > > > >> > - These are basically the > integration > > > > tests > > > > > > for > > > > > > >> > > > > > verification > > > > > > >> > > > > > > > > after > > > > > > >> > > > > > > > > > >> the > > > > > > >> > > > > > > > > > >> > deployment. This is a soft requirement, > > but > > > > aim > > > > > > is > > > > > > >> to > > > > > > >> > > > > achieve > > > > > > >> > > > > > > this > > > > > > >> > > > > > > > > as > > > > > > >> > > > > > > > > > >> well > > > > > > >> > > > > > > > > > >> > or at least have explanation why not to > > > > > include. > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > If there are any from the link which > > > > explicitly > > > > > > >> needs > > > > > > >> > to > > > > > > >> > > > be > > > > > > >> > > > > > > > checked > > > > > > >> > > > > > > > > > >> other > > > > > > >> > > > > > > > > > >> > than above 4, please let us know. > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > *Step 2) Adding Apex as component to > > > Bigtop* > > > > > > >> > > > > > > > > > >> > From one of the mail archive of Bigtop, > > it > > > > was > > > > > > >> learnt > > > > > > >> > > that > > > > > > >> > > > > the > > > > > > >> > > > > > > > > bigtop > > > > > > >> > > > > > > > > > >> > community want to see the addition of > new > > > > > > >> components > > > > > > >> > in > > > > > > >> > > > > > phases. > > > > > > >> > > > > > > > Here > > > > > > >> > > > > > > > > > are > > > > > > >> > > > > > > > > > >> > the phases: > > > > > > >> > > > > > > > > > >> > 1. Packaging > > > > > > >> > > > > > > > > > >> > - This needs creating of package > i.e. > > > > rpm & > > > > > > deb > > > > > > >> > > files. > > > > > > >> > > > > > > > > > >> > - documentations/READMEs, LICENSE, > > > > > DISCLAMER, > > > > > > >> > NOTES > > > > > > >> > > > etc > > > > > > >> > > > > if > > > > > > >> > > > > > > any > > > > > > >> > > > > > > > > > >> needed. > > > > > > >> > > > > > > > > > >> > - Any documentation that need to be > > > added > > > > > to > > > > > > >> > > > > distribution > > > > > > >> > > > > > of > > > > > > >> > > > > > > > our > > > > > > >> > > > > > > > > > >> > software. > > > > > > >> > > > > > > > > > >> > - Any license information of > > > dependencies > > > > > > >> required > > > > > > >> > > to > > > > > > >> > > > be > > > > > > >> > > > > > > added > > > > > > >> > > > > > > > > to > > > > > > >> > > > > > > > > > >> > package > > > > > > >> > > > > > > > > > >> > 2. Smoke tests (at very least) > > > > > > >> > > > > > > > > > >> > - Adding smoke test for packaging. > > > > > > >> > > > > > > > > > >> > 3. Puppet recipes for automatic > > deployment > > > > and > > > > > > >> > > > configuration > > > > > > >> > > > > > > > > > >> > - Add puppet recipes for automatic > > > > > deployment > > > > > > >> > > > > > > > > > >> > 4. Integration tests > > > > > > >> > > > > > > > > > >> > - For verification of deployments. > > > > > > >> > > > > > > > > > >> > 5. license clearance: > > > > > > >> > > > > > > > > > >> > Run 'gradle rat' to make sure all > new > > > > stuff > > > > > > is > > > > > > >> > > > compliant > > > > > > >> > > > > > > with > > > > > > >> > > > > > > > > ASF > > > > > > >> > > > > > > > > > >> > license requirements. If you add code > > > > licenses > > > > > > >> under > > > > > > >> > > > > different > > > > > > >> > > > > > > > > > licenses, > > > > > > >> > > > > > > > > > >> > those would need to be listed in the > > > NOTICE. > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > Please share your thoughts on the > > approach. > > > > > > >> > > > > > > > > > >> > We'll start corresponding communication > > on > > > > > bigtop > > > > > > >> > > mailing > > > > > > >> > > > > list > > > > > > >> > > > > > > as > > > > > > >> > > > > > > > > > well. > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > We have some specific > > questions/suggestions > > > > > > >> related to > > > > > > >> > > > what > > > > > > >> > > > > > > should > > > > > > >> > > > > > > > > be > > > > > > >> > > > > > > > > > >> the > > > > > > >> > > > > > > > > > >> > content of the package and what should > be > > > the > > > > > > smoke > > > > > > >> > > tests, > > > > > > >> > > > > but > > > > > > >> > > > > > > in > > > > > > >> > > > > > > > > the > > > > > > >> > > > > > > > > > >> > interest of not having too much content > > > here, > > > > > > we'll > > > > > > >> > put > > > > > > >> > > > the > > > > > > >> > > > > > > > > questions > > > > > > >> > > > > > > > > > >> as a > > > > > > >> > > > > > > > > > >> > separate mail in this mailthread. > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > Thanks, > > > > > > >> > > > > > > > > > >> > Chinmay. > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
