[jira] [Resolved] (AMATERASU-45) PySpark: refactor the PySpark runtime into it's own component

[ https://issues.apache.org/jira/browse/AMATERASU-45?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nadav Har Tzvi resolved AMATERASU-45. - Resolution: Fixed > PySpark: refactor the PySpark runtime into it's own compon

Re: [Discuss] datasets input file in user's repository

t is very common > to use different databases for dev/test/prod), so I think that datasets as > configurations in Amaterasu should sit under env). > > Cheers, > Yaniv > > On Tue, Mar 19, 2019 at 5:13 AM Nadav Har Tzvi > wrote: > > > Hi, > > > > Just wa

[jira] [Created] (AMATERASU-74) amaterasu-vagrant is broken

Nadav Har Tzvi created AMATERASU-74: --- Summary: amaterasu-vagrant is broken Key: AMATERASU-74 URL: https://issues.apache.org/jira/browse/AMATERASU-74 Project: AMATERASU Issue Type: Bug

Re: [DISCUSS] Review and discussions thread for Amaterasu-45 PR

the end of this week. > > Cheers, > Arun > > On Wed, Feb 20, 2019 at 3:12 AM Nadav Har Tzvi > wrote: > > > Hey everyone, > > > > I opened a draft PR for Amaterasu 45 (Python SDK, pyspark runtime, python > > runtime). Do mind that this is still a WIP. I woul

Re: [jira] [Created] (AMATERASU-52) Implement AmaContext.datastores

In my opinion it is more straightforward and uniform. I think it is also more straightforward code-wise. What do you think? Cheers, Nadav On Mon, 14 Jan 2019 at 00:57, Yaniv Rodenski wrote: > Hi Arun, > > I've added my comments to the PR, but good call, I agree @Nadav Har Tzvi

Re: [DISCUSS] podling report

Yeah, I give it a +1 Cheers, Nadav On Tue, 1 Jan 2019 at 17:12, Yaniv Rodenski wrote: > Hi All, > > I propose the following report to be submitted. > > Amaterasu > > > Apache Amaterasu is a framework providing configuration management and > > deployment for Big Data Pipelines. > > > It

[DISCUSS] Common dependencies formats

Hey everyone, In both conventions I spoke in (PyCon-IL and SDP) the participants were inquiring about why we don't use common formats to specify job dependencies (e.g. - requirements.txt in Python). I want to bring this matter for discussion. What can we do to conform better with the developers

[DISCUSS] Dependencies resolution and action level dependencies

Hey everyone, Yaniv and I were just discussing how to resolve dependencies in the new frameworks architecture and integrate the dependencies with the concrete cluster resource manager (Mesos/YARN) We rolled with the idea of each runner (or base runner) performing the dependencies resolution on

[jira] [Created] (AMATERASU-43) Make the notifier available for job code

Nadav Har Tzvi created AMATERASU-43: --- Summary: Make the notifier available for job code Key: AMATERASU-43 URL: https://issues.apache.org/jira/browse/AMATERASU-43 Project: AMATERASU Issue

[jira] [Created] (AMATERASU-42) Amaterasu needs to respect Maven classifiers

Nadav Har Tzvi created AMATERASU-42: --- Summary: Amaterasu needs to respect Maven classifiers Key: AMATERASU-42 URL: https://issues.apache.org/jira/browse/AMATERASU-42 Project: AMATERASU

[jira] [Updated] (AMATERASU-41) Support nested custom configurations in jobs environments in PySpark

[ https://issues.apache.org/jira/browse/AMATERASU-41?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nadav Har Tzvi updated AMATERASU-41: Summary: Support nested custom configurations in jobs environments in PySpark

[jira] [Created] (AMATERASU-41) Support nested custom configurations in jobs environments

Nadav Har Tzvi created AMATERASU-41: --- Summary: Support nested custom configurations in jobs environments Key: AMATERASU-41 URL: https://issues.apache.org/jira/browse/AMATERASU-41 Project: AMATERASU

[jira] [Created] (AMATERASU-40) Change Spark-Scala runner to use spark-submit

Nadav Har Tzvi created AMATERASU-40: --- Summary: Change Spark-Scala runner to use spark-submit Key: AMATERASU-40 URL: https://issues.apache.org/jira/browse/AMATERASU-40 Project: AMATERASU

[jira] [Created] (AMATERASU-39) Support SSH for Git

Nadav Har Tzvi created AMATERASU-39: --- Summary: Support SSH for Git Key: AMATERASU-39 URL: https://issues.apache.org/jira/browse/AMATERASU-39 Project: AMATERASU Issue Type: Task

[jira] [Updated] (AMATERASU-38) Failure to interpret multiline Scala code blocks

[ https://issues.apache.org/jira/browse/AMATERASU-38?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nadav Har Tzvi updated AMATERASU-38: Priority: Blocker (was: Major) > Failure to interpret multiline Scala code blo

[Discussion] Changing Amaterasu deployment strategy

k for details: https://github.com/nadav-har-tzvi/incubator-amaterasu/tree/feature/ama-cli ) 2. Create a RPM package, so users will be able to add a repository and simply "yum install amaterasu". If you have any thoughts you'd like to share in the matter, please do. Cheers, Nadav

[jira] [Updated] (AMATERASU-34) Running Amaterasu during development without a cluster

[ https://issues.apache.org/jira/browse/AMATERASU-34?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nadav Har Tzvi updated AMATERASU-34: Issue Type: New Feature (was: Task) > Running Amaterasu during development with

[jira] [Updated] (AMATERASU-34) Running Amaterasu during development without a cluster

[ https://issues.apache.org/jira/browse/AMATERASU-34?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nadav Har Tzvi updated AMATERASU-34: Issue Type: Improvement (was: New Feature) > Running Amaterasu during developm

[jira] [Created] (AMATERASU-34) Running Amaterasu during development without a cluster

Nadav Har Tzvi created AMATERASU-34: --- Summary: Running Amaterasu during development without a cluster Key: AMATERASU-34 URL: https://issues.apache.org/jira/browse/AMATERASU-34 Project: AMATERASU

[jira] [Created] (AMATERASU-32) Investigate why Amaterasu requires minimum of 2G memory to run on DC/OS

Nadav Har Tzvi created AMATERASU-32: --- Summary: Investigate why Amaterasu requires minimum of 2G memory to run on DC/OS Key: AMATERASU-32 URL: https://issues.apache.org/jira/browse/AMATERASU-32

[jira] [Created] (AMATERASU-31) Investigate why PySpark actions on EMR require minimum of 2G memory

Nadav Har Tzvi created AMATERASU-31: --- Summary: Investigate why PySpark actions on EMR require minimum of 2G memory Key: AMATERASU-31 URL: https://issues.apache.org/jira/browse/AMATERASU-31 Project

[jira] [Created] (AMATERASU-29) PySpark breaks for jobs without extra configurations

Nadav Har Tzvi created AMATERASU-29: --- Summary: PySpark breaks for jobs without extra configurations Key: AMATERASU-29 URL: https://issues.apache.org/jira/browse/AMATERASU-29 Project: AMATERASU

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

and build was successful on the branch > > "version-0.2.0-incubating-rc3" > > > > +1 from me > > > > Cheers, > > Kirupa > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg wrote: > > > > > +1 looks good to me > > &

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

+1 approve. Tested multiple times and after a long round of fixing and testing over and over. Cheers, Nadav On 29 May 2018 at 07:38, Yaniv Rodenski wrote: > Hi everyone, > > We have fixed the legal issues, as well as a bug found by @Nadav please > review and vote on the release candidate #3

Re: Speeding up Anaconda deployment on executors

I am kinda inclined towards a solution that could be performed at setup time. Let's try to explore in that direction. If we manage to nail this at setup time, it means that the runtime will be lightning fast (compared to what it is now :)) Cheers, Nadav On 28 May 2018 at 00:17, Nadav Har Tzvi

Re: AMATERASU-24

I agree with Yaniv that Frameworks should be plugins. Think about it like this, in the future, hopefully, you will be able to do something like "sudo yum install amaterasu" After install the "core" amaterasu using yum, you will be able to use the new CLI like this: "ama frameworks add " to add a

[jira] [Created] (AMATERASU-27) ama CLI doesn't take into account amaterasu.properties changes in YARN

Nadav Har Tzvi created AMATERASU-27: --- Summary: ama CLI doesn't take into account amaterasu.properties changes in YARN Key: AMATERASU-27 URL: https://issues.apache.org/jira/browse/AMATERASU-27

[jira] [Created] (AMATERASU-25) Create documentation with ReadTheDocs

Nadav Har Tzvi created AMATERASU-25: --- Summary: Create documentation with ReadTheDocs Key: AMATERASU-25 URL: https://issues.apache.org/jira/browse/AMATERASU-25 Project: AMATERASU Issue Type

Re: Project status

ssible. > >> > > > >> > > Given the above, blaming a mentor (whomever you may be referring > >to) > >> > > doesn't make sense. > >> > > > >> > > * We are ready to release version 0.2.0-incubating, the reason it > >took > >>

[jira] [Created] (AMATERASU-5) Allocated resources aren't cleaned up properly on crash/unexpected halt

Nadav Har Tzvi created AMATERASU-5: -- Summary: Allocated resources aren't cleaned up properly on crash/unexpected halt Key: AMATERASU-5 URL: https://issues.apache.org/jira/browse/AMATERASU-5 Project

Re: Moving forward

Sure, no objections here. I will open PR for ama-cli after you are done. Actually, after I am done :) > On 26 Oct 2017, at 2:48, Yaniv Rodenski wrote: > > OK so GitHub doesn't allow forking empty branches, so I think I'll just > push the shintoio master to the