Hi Arun, AMATERASU-44 and its python version AMATERASU-46 (which I believe @nadav started working on) are about moving away from the implicit persistence mechanism in the current Amaterasu implementation.
The base for those changes is the existence of a datastores.yaml file in the container which will have the configuration for all data stores (both for loading and persisting) this is done as part of AMATERASU-49 (and there is an open PR for that change in Mesos so you can start with that). Given that file, AmaContext.persist will accept a dataset and the name of the data store configuration and will persist the dataset. We can currently limit the implementation to a Hadoop file system (any of them) and a file format, and add support for other configuration types such as Hive, Kafka, etc. in an additional task. Cheers, Yaniv On Thu, Oct 11, 2018 at 3:10 PM Arun Manivannan <a...@arunma.com> wrote: > Hi Yaniv and all, > > I am trying to take a stab at AMATERASU-44 > <https://issues.apache.org/jira/browse/AMATERASU-44>and would like to get > some background behind this issue - specifically around the current state > and what are the things that ought to be considered before making the > change. > > > Cheers, > Arun > -- Yaniv Rodenski +61 477 778 405 ya...@shinto.io
