Hi Davor, Sorry for keeping my emails short and undescriptive, I have only one hand for the next month due to a skateboarding accident. Based on what I've seen in other projects, it seems common to release bot the binaries (in Amaterasu those are released in the .tar file) as well as a source (which are in the -source.tar.gz file) The main question is if we do or do not need to release both.
Cheers, Yaniv On Sat, Jun 2, 2018 at 1:13 AM, Davor Bonaci <dbon...@gmail.com> wrote: > No problem with .tar.gz. But, you don't need .tar then. > > On Fri, Jun 1, 2018 at 12:04 AM, Yaniv Rodenski <ya...@shinto.io> wrote: > > > One thing to clarify, the tar.gz is the source code as I've seen this is > > released in other projects. if that is not needed I'm happy to drop it > from > > future releases. > > > > On Fri, Jun 1, 2018 at 5:01 PM, Yaniv Rodenski <ya...@shinto.io> wrote: > > > > > Thanks Davor, > > > > > > We will fix those for the next release. > > > > > > Cheers, > > > Yaniv > > > > > > On Fri, Jun 1, 2018 at 7:23 AM, Davor Bonaci <dbon...@gmail.com> > wrote: > > > > > >> Great improvement! > > >> > > >> license/notice/disclaimer: ok > > >> headers: ~fine (missing in some non-code files, it would be great to > add > > >> those -- non-blocking) > > >> binaries: there's a gradle binary, but that may be ruled as acceptable > > >> (non-blocking from me) > > >> > > >> Not sure why we need both .tar and .tar.gz. I'd get rid of one of > them. > > >> Also, shipping private keys in the distribution looks weird as well. > > >> > > >> For the first release, this is good. So, +1 to send onwards for an > > >> Incubator review, and polish a few of these things for a subsequent > > >> release. > > >> > > >> On Wed, May 30, 2018 at 7:12 PM, Yaniv Rodenski <ya...@shinto.io> > > wrote: > > >> > > >> > Hi All, > > >> > > > >> > Just a kind reminder for our mentors, we need your vote on this one > :) > > >> > > > >> > Cheers, > > >> > Yaniv > > >> > > > >> > On Wed, May 30, 2018 at 3:25 PM, Arun Manivannan <a...@arunma.com> > > >> wrote: > > >> > > > >> > > Thanks a lot, Nadav. Will get home and spend some more time on > > this. I > > >> > was > > >> > > in a rush and did this poor workaround. My VM is just 8 GB. > > >> > > > > >> > > Cheers > > >> > > Arun > > >> > > > > >> > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi < > nadavhart...@gmail.com> > > >> > wrote: > > >> > > > > >> > > > Yaniv and I just tested it. It worked flawlessly on my end (HDP > > >> docker > > >> > on > > >> > > > AWS). Both Spark-Scala and PySpark. > > >> > > > It worked on Yaniv's HDP cluster as well. > > >> > > > Worth noting: > > >> > > > 1. HDP 2.6.4 > > >> > > > 2. Cluster has total of 32GB memory available > > >> > > > 3. Each container is allocated 1G memory. > > >> > > > 4. Amaterasu.properties: > > >> > > > > > >> > > > zk=sandbox-hdp.hortonworks.com > > >> > > > version=0.2.0-incubating-rc3 > > >> > > > master=192.168.33.11 > > >> > > > user=root > > >> > > > mode=yarn > > >> > > > webserver.port=8000 > > >> > > > webserver.root=dist > > >> > > > spark.version=2.6.4.0-91 > > >> > > > yarn.queue=default > > >> > > > yarn.jarspath=hdfs:///apps/amaterasu > > >> > > > spark.home=/usr/hdp/current/spark2-client > > >> > > > > > >> > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0. > > >> > cloudera2-1.cdh5.7.0.p0. > > >> > > 171658/lib/spark2 > > >> > > > yarn.hadoop.home.dir=/etc/hadoop > > >> > > > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2. > > >> 6.4.0-91" > > >> > > > spark.opts.spark.driver.extraJavaOptions="-Dhdp. > > version=2.6.4.0-91" > > >> > > > > > >> > > > > > >> > > > Arun, please share: > > >> > > > 1. YARN memory configurations > > >> > > > 2. amaterasu.properties content > > >> > > > 3. HDP version. > > >> > > > > > >> > > > Cheers, > > >> > > > Nadav > > >> > > > > > >> > > > > > >> > > > On 30 May 2018 at 07:11, Arun Manivannan <a...@arunma.com> > wrote: > > >> > > > > > >> > > > > The pmem disabling is just temporary. I'll do a detailed > > analysis > > >> and > > >> > > get > > >> > > > > back with a proper solution. > > >> > > > > > > >> > > > > Any hints on this front is highly appreciated. > > >> > > > > > > >> > > > > Cheers > > >> > > > > Arun > > >> > > > > > > >> > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi < > > >> nadavhart...@gmail.com> > > >> > > > wrote: > > >> > > > > > > >> > > > > > Yaniv, Eyal, this might be related to the same issue you > faced > > >> with > > >> > > > HDP. > > >> > > > > > Can you confirm? > > >> > > > > > > > >> > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan < > a...@arunma.com> > > >> > wrote: > > >> > > > > > > > >> > > > > > > +1 from me > > >> > > > > > > > > >> > > > > > > Unit Tests and Build ran fine. > > >> > > > > > > > > >> > > > > > > Tested on HDP (VM) but had trouble allocating containers > > >> (didn't > > >> > > have > > >> > > > > > that > > >> > > > > > > before). Apparently Centos VMs are known to have this > > >> problem. > > >> > > > > Disabled > > >> > > > > > > physical memory check (yarn.nodemanager.pmem-check- > > enabled) > > >> and > > >> > > ran > > >> > > > > jobs > > >> > > > > > > successfully. > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan < > > >> > > > > > kirupagara...@gmail.com > > >> > > > > > > > > > >> > > > > > > wrote: > > >> > > > > > > > > >> > > > > > > > Unit tests passing and build was successful on the > branch > > >> > > > > > > > "version-0.2.0-incubating-rc3" > > >> > > > > > > > > > >> > > > > > > > +1 from me > > >> > > > > > > > > > >> > > > > > > > Cheers, > > >> > > > > > > > Kirupa > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg < > > >> whisr...@gmail.com > > >> > > > > >> > > > > wrote: > > >> > > > > > > > > > >> > > > > > > > > +1 looks good to me > > >> > > > > > > > > > > >> > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi < > > >> > > > nadavhart...@gmail.com > > >> > > > > > > > >> > > > > > > > wrote: > > >> > > > > > > > > > > >> > > > > > > > > > +1 approve. Tested multiple times and after a long > > >> round of > > >> > > > > fixing > > >> > > > > > > and > > >> > > > > > > > > > testing over and over. > > >> > > > > > > > > > > > >> > > > > > > > > > Cheers, > > >> > > > > > > > > > Nadav > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski < > > >> ya...@shinto.io> > > >> > > > wrote: > > >> > > > > > > > > > > > >> > > > > > > > > > > Hi everyone, > > >> > > > > > > > > > > > > >> > > > > > > > > > > We have fixed the legal issues, as well as a bug > > >> found by > > >> > > > > @Nadav > > >> > > > > > > > please > > >> > > > > > > > > > > review and vote on the release candidate #3 for > the > > >> > version > > >> > > > > > > > > > > 0.2.0-incubating, as follows > > >> > > > > > > > > > > > > >> > > > > > > > > > > [ ] +1, Approve the release > > >> > > > > > > > > > > [ ] -1, Do not approve the release (please provide > > >> > specific > > >> > > > > > > comments) > > >> > > > > > > > > > > > > >> > > > > > > > > > > The complete staging area is available for your > > >> review, > > >> > > which > > >> > > > > > > > includes: > > >> > > > > > > > > > > > > >> > > > > > > > > > > * JIRA release notes [1], > > >> > > > > > > > > > > * the official Apache source release to be > deployed > > to > > >> > > > > > > > dist.apache.org > > >> > > > > > > > > > > [2], > > >> > > > > > > > > > > which is signed with the key with fingerprint [3], > > >> > > > > > > > > > > * source code tag "version-0.2.0-incubating-rc3" > > [4], > > >> > > > > > > > > > > * Java artifacts were built with Gradle 3.1 and > > >> > > > OpenJDK/Oracle > > >> > > > > > JDK > > >> > > > > > > > > > > 1.8.0_151 > > >> > > > > > > > > > > > > >> > > > > > > > > > > The vote will be open for at least 72 hours. It is > > >> > adopted > > >> > > by > > >> > > > > > > > majority > > >> > > > > > > > > > > approval, with at least 3 PMC affirmative votes. > > >> > > > > > > > > > > > > >> > > > > > > > > > > Thanks, > > >> > > > > > > > > > > Yaniv > > >> > > > > > > > > > > > > >> > > > > > > > > > > [1] https://issues.apache.org/ > > >> > jira/secure/ReleaseNote.jspa > > >> > > ? > > >> > > > > > > > > > > projectId=12321521&version=12342793 > > >> > > > > > > > > > > [2] https://dist.apache.org/repos/ > > >> > > > > dist/dev/incubator/amaterasu/ > > >> > > > > > > > > 0.2.0rc3/ > > >> > > > > > > > > > > [3] > > >> > > > > > > https://dist.apache.org/repos/ > > dist/dev/incubator/amaterasu/K > > >> EYS > > >> > > > > > > > > > > [4] https://github.com/apache/incu > > >> bator-amaterasu/tags > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > -- > > >> > Yaniv Rodenski > > >> > > > >> > +61 477 778 405 > > >> > ya...@shinto.io > > >> > > > >> > > > > > > > > > > > > -- > > > Yaniv Rodenski > > > > > > +61 477 778 405 > > > ya...@shinto.io > > > > > > > > > > > -- > > Yaniv Rodenski > > > > +61 477 778 405 > > ya...@shinto.io > > > -- Yaniv Rodenski +61 477 778 405 ya...@shinto.io
