Ok, I will do an E2E test on a pyspark only flow tonight. I think that after rebasing onto master with the changes from Amaterasu-8, I will have to make some adjustments in regard to the integration between the leader and the executor. If there are any, we should be coordinated.
Cheers, Nadav On Wed, 20 Feb 2019 at 13:10, Arun Manivannan <a...@arunma.com> wrote: > Hi Nadav, > > Had a look at the datasets.yml and the bindings for it. This aligns > perfectly with the consensus that we had on the call that we had the day > before. > > I'll have it implemented for the JVM by the end of this week. > > Cheers, > Arun > > On Wed, Feb 20, 2019 at 3:12 AM Nadav Har Tzvi <nadavhart...@gmail.com> > wrote: > > > Hey everyone, > > > > I opened a draft PR for Amaterasu 45 (Python SDK, pyspark runtime, python > > runtime). Do mind that this is still a WIP. I would like you to review > this > > PR from time to time as it evolves. The sooner I get inputs, the sooner > > this feature will be out. > > > > PR: https://github.com/apache/incubator-amaterasu/pull/44 > > > > Yaniv, Eyal, Arun. Please let's now define the final MVP of this feature > so > > I can know which reviews to defer and which to attend to before this PR > is > > closed and merged. > > > > Arun, this is most important, if I ended up implementing a design that is > > vastly different than the scala runtime design, we should talk ASAP. > > > > I would like the community to give any feedback. Did I miss something? > Did > > I make some stupid mistake? Does the design look awkward? etc. > > > > Also a note on the python runtime. It is currently February 2019. As of > > January 1st 2020 Python2 is EOL. Many third party Python packages already > > dropped support entirely for Python2. Because of that, the entire SDK and > > runtimes are implemented in Python >=3.4 (Python <= 3.3.x is EOL) > > If you think this is problematic, please raise a flag. > > > > Cheers, > > Nadav > > >
