Thanks a lot, Nadav. Will get home and spend some more time on this. I was in a rush and did this poor workaround. My VM is just 8 GB.
Cheers Arun On Wed, May 30, 2018, 12:27 Nadav Har Tzvi <nadavhart...@gmail.com> wrote: > Yaniv and I just tested it. It worked flawlessly on my end (HDP docker on > AWS). Both Spark-Scala and PySpark. > It worked on Yaniv's HDP cluster as well. > Worth noting: > 1. HDP 2.6.4 > 2. Cluster has total of 32GB memory available > 3. Each container is allocated 1G memory. > 4. Amaterasu.properties: > > zk=sandbox-hdp.hortonworks.com > version=0.2.0-incubating-rc3 > master=192.168.33.11 > user=root > mode=yarn > webserver.port=8000 > webserver.root=dist > spark.version=2.6.4.0-91 > yarn.queue=default > yarn.jarspath=hdfs:///apps/amaterasu > spark.home=/usr/hdp/current/spark2-client > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2 > yarn.hadoop.home.dir=/etc/hadoop > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > > > Arun, please share: > 1. YARN memory configurations > 2. amaterasu.properties content > 3. HDP version. > > Cheers, > Nadav > > > On 30 May 2018 at 07:11, Arun Manivannan <a...@arunma.com> wrote: > > > The pmem disabling is just temporary. I'll do a detailed analysis and get > > back with a proper solution. > > > > Any hints on this front is highly appreciated. > > > > Cheers > > Arun > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi <nadavhart...@gmail.com> > wrote: > > > > > Yaniv, Eyal, this might be related to the same issue you faced with > HDP. > > > Can you confirm? > > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan <a...@arunma.com> wrote: > > > > > > > +1 from me > > > > > > > > Unit Tests and Build ran fine. > > > > > > > > Tested on HDP (VM) but had trouble allocating containers (didn't have > > > that > > > > before). Apparently Centos VMs are known to have this problem. > > Disabled > > > > physical memory check (yarn.nodemanager.pmem-check-enabled) and ran > > jobs > > > > successfully. > > > > > > > > > > > > > > > > > > > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan < > > > kirupagara...@gmail.com > > > > > > > > > wrote: > > > > > > > > > Unit tests passing and build was successful on the branch > > > > > "version-0.2.0-incubating-rc3" > > > > > > > > > > +1 from me > > > > > > > > > > Cheers, > > > > > Kirupa > > > > > > > > > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg <whisr...@gmail.com> > > wrote: > > > > > > > > > > > +1 looks good to me > > > > > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi < > nadavhart...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > +1 approve. Tested multiple times and after a long round of > > fixing > > > > and > > > > > > > testing over and over. > > > > > > > > > > > > > > Cheers, > > > > > > > Nadav > > > > > > > > > > > > > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski <ya...@shinto.io> > wrote: > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > We have fixed the legal issues, as well as a bug found by > > @Nadav > > > > > please > > > > > > > > review and vote on the release candidate #3 for the version > > > > > > > > 0.2.0-incubating, as follows > > > > > > > > > > > > > > > > [ ] +1, Approve the release > > > > > > > > [ ] -1, Do not approve the release (please provide specific > > > > comments) > > > > > > > > > > > > > > > > The complete staging area is available for your review, which > > > > > includes: > > > > > > > > > > > > > > > > * JIRA release notes [1], > > > > > > > > * the official Apache source release to be deployed to > > > > > dist.apache.org > > > > > > > > [2], > > > > > > > > which is signed with the key with fingerprint [3], > > > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4], > > > > > > > > * Java artifacts were built with Gradle 3.1 and > OpenJDK/Oracle > > > JDK > > > > > > > > 1.8.0_151 > > > > > > > > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > > > > > majority > > > > > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Yaniv > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > > > > > > > > projectId=12321521&version=12342793 > > > > > > > > [2] https://dist.apache.org/repos/ > > dist/dev/incubator/amaterasu/ > > > > > > 0.2.0rc3/ > > > > > > > > [3] > > > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS > > > > > > > > [4] https://github.com/apache/incubator-amaterasu/tags > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
