[jira] [Commented] (AMATERASU-52) Implement AmaContext.datastores

[ https://issues.apache.org/jira/browse/AMATERASU-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853741#comment-16853741 ] Arun Manivannan commented on AMATERASU-52: -- [~yaniv] [~nadavha]  - This PR (53) contains

Re: [jira] [Created] (AMATERASU-52) Implement AmaContext.datastores

iform. I think it is also > more straightforward code-wise. > What do you think? > > Cheers, > Nadav > > > > On Mon, 14 Jan 2019 at 00:57, Yaniv Rodenski wrote: > > > Hi Arun, > > > > I've added my comments to the PR, but good call, I agree @Nadav Har

Re: [jira] [Created] (AMATERASU-52) Implement AmaContext.datastores

TERASU-52: > --- > > Summary: Implement AmaContext.datastores > Key: AMATERASU-52 > URL: https://issues.apache.org/jira/browse/AMATERASU-52 > Project: AMATERASU > Issue Type: Task > Reporter: Yaniv Rodenski >

Unable to read config with konf

All, As part of Amaterasu 52, I am trying to come up with a format for datasets configuration. Here's what I have datasets: hive: - key: transactions uri: /user/somepath format: parquet database: transations_daily table: transx - key: second_transactions

AMATERASU-44

Hi Yaniv and all, I am trying to take a stab at AMATERASU-44 and would like to get some background behind this issue - specifically around the current state and what are the things that ought to be considered before making the change. Cheers,

[jira] [Resolved] (AMATERASU-28) Pull Miniconda version away from compiled code

[ https://issues.apache.org/jira/browse/AMATERASU-28?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Manivannan resolved AMATERASU-28. -- Resolution: Fixed Assignee: Yaniv Rodenski (was: Arun Manivannan) > P

[jira] [Commented] (AMATERASU-28) Pull Miniconda version away from compiled code

[ https://issues.apache.org/jira/browse/AMATERASU-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645834#comment-16645834 ] Arun Manivannan commented on AMATERASU-28: -- This issue is already fixed as part of PR

Re: [DISCUSS] podling report

+1 Looks great. On Mon, Jul 9, 2018, 17:22 Nadav Har Tzvi wrote: > +1 > > On Mon, Jul 9, 2018, 14:46 Jean-Baptiste Onofré wrote: > > > +1 > > > > Regards > > JB > > > > On 09/07/2018 11:55, Eyal Ben-Ivri wrote: > > > +1 > > > > > > > > > On 6. July 2018 at 18:50:37, Davor Bonaci

Re: [VOTE] Release Apache Amaterasu (incubating) 0.2.0 (rc4)

Clean built. Testcases run fine. Ran it on HDP and HDP 2.6.5.0-292 with minor change on the leader AM. Works perfect. +1 Cheers, Arun On Sun, Jun 24, 2018 at 5:10 PM guy peleg wrote: > Downloaded the source code and the artifact, everything looks good on my > side. > +1 > > Guy > > On Sun,

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

Anyway, ideally I think we should try and have per job configurations in > the environment rather than in the amaterasu.properties > > Cheers, > Yaniv > > On Sun, 3 Jun 2018 at 1:32 am, Arun Manivannan wrote: > > > Gentlemen, > > > > Apologies for comin

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

eanwhile, I'll also update by PR for Amaterasu-24 after pulling the latest from the branch. Cheers, Arun On Wed, May 30, 2018 at 1:25 PM Arun Manivannan wrote: > Thanks a lot, Nadav. Will get home and spend some more time on this. I was > in a rush and did this poor workaround. My VM is just 8

Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3

+1 from me Unit Tests and Build ran fine. Tested on HDP (VM) but had trouble allocating containers (didn't have that before). Apparently Centos VMs are known to have this problem. Disabled physical memory check (yarn.nodemanager.pmem-check-enabled) and ran jobs successfully. On Tue, May

[jira] [Assigned] (AMATERASU-24) Refactor Spark out of Amaterasu executor to it's own project

[ https://issues.apache.org/jira/browse/AMATERASU-24?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Manivannan reassigned AMATERASU-24: Assignee: Arun Manivannan > Refactor Spark out of Amaterasu executor to it's

Re: AMATERASU-24

structure to start with: >> frameworks >> |-> spark >> |-> runner >> |-> runtime >> >> As for the shell scripts, I will leave that for @Nadav, but please have a >> look at PR #17 containing the CLI that will replace the scripts as o

Re: Amaterasu PyCon il Presentation

Looks great !! I suppose this will be accompanied by a Demo. On Sun, May 27, 2018 at 5:49 PM Yaniv Rodenski wrote: > Hi All, > > @Nadav is giving an intro to Amaterasu @ PyCon Israel next week and we > thought everyone here might like to have the deck > < >

Re: AMATERASU-24

r the shell scripts, I will leave that for @Nadav, but please have a > > look at PR #17 containing the CLI that will replace the scripts as of > > 0.2.1-incubating. > > > > Cheers, > > Yaniv > > > > On Sat, May 26, 2018 at 5:16 PM, Arun Manivannan <a...@

[jira] [Updated] (AMATERASU-28) Pull Miniconda version away from compiled code

[ https://issues.apache.org/jira/browse/AMATERASU-28?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Manivannan updated AMATERASU-28: - Summary: Pull Miniconda version away from compiled code (was: Pull Miniconda version

[jira] [Created] (AMATERASU-28) Pull Miniconda version away from compilable code

Arun Manivannan created AMATERASU-28: Summary: Pull Miniconda version away from compilable code Key: AMATERASU-28 URL: https://issues.apache.org/jira/browse/AMATERASU-28 Project: AMATERASU

[jira] [Updated] (AMATERASU-26) Pipeline tasks (sub-Yarn jobs) runs as "yarn" user instead of inhering the user in which the amaterasu job was submitted

[ https://issues.apache.org/jira/browse/AMATERASU-26?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Manivannan updated AMATERASU-26: - Description: Referring to the screenshot, the original user with which the amaterasu

[jira] [Created] (AMATERASU-26) Pipeline tasks (sub-Yarn jobs) runs as "yarn" user instead of inhering the user in which the amaterasu job was submitted

Arun Manivannan created AMATERASU-26: Summary: Pipeline tasks (sub-Yarn jobs) runs as "yarn" user instead of inhering the user in which the amaterasu job was submitted Key: AMATERASU-26

YARN deployment

Hi Yaniv and Eyal, Sorry about the hiatus. Day job has been hectic the last couple of months. I am really glad that we now have full blown YARN support. Thanks a lot !! Is there a place where I could find a rough document around how to submit jobs on YARN. If you could respond to this thread,

ama-start and JobLauncher stuck (Solution found) + Running Spark jobs across several repo

Hi, Very Good morning and Wish you all a wonderful New Year ahead. Sorry to bother you on New Year's eve but I really appreciate any hints. *1. Setting up and running the basic setup (on MacOS High Sierra) (Solution found): * I remember having done this successfully before but it was strange

Re: Initial setup of Amaterasu - Unable to run

un again, let us know if it solves the problem. > > Cheers, > Yaniv > > On Sat, 30 Sep 2017 at 7:36 pm, Arun Manivannan <a...@arunma.com> wrote: > > > Hi, > > > > I am trying to make an initial run on Amaterasu with > > https://github.com/arunma/amateras