Hey Arun, I kinda feel like the datastores yaml is somewhat obscure. I propose the following structure.
Instead of datasets: hive: - key: transactions uri: /user/somepath format: parquet database: transations_daily table: transx - key: second_transactions uri: /seconduser/somepath format: avro database: transations_monthly table: avro_table file: - key: users uri: s3://filestore format: parquet mode: overwrite I would have datasets: - key: transactions uri: /user/somepath format: parquet database: transations_daily table: transx type: hive - key: second_transactions uri: /seconduser/somepath format: avro database: transations_monthly table: avro_table type: hive - key: users uri: s3://filestore format: parquet mode: overwrite type: file In my opinion it is more straightforward and uniform. I think it is also more straightforward code-wise. What do you think? Cheers, Nadav On Mon, 14 Jan 2019 at 00:57, Yaniv Rodenski <ya...@shinto.io> wrote: > Hi Arun, > > I've added my comments to the PR, but good call, I agree @Nadav Har Tzvi > <nadavhart...@gmail.com> should at least review as you both need to > maintain compatible APIs. > > Cheers, > Yaniv > > On Sun, Jan 13, 2019 at 10:21 PM Arun Manivannan <a...@arunma.com> wrote: > >> Hi Guy, Yaniv and Nadiv, >> >> This PR <https://github.com/apache/incubator-amaterasu/pull/39> just >> captures part of the issue - the datasets.yaml, ConfigManager and the >> testcases. The Integration with the AmaContext is yet to be done but I >> would like to get your thoughts on the implementation. >> >> Guy - Would it be okay if you could help throw some light on the syntax >> and >> the idiomatic part of Kotlin itself. Newbie here. >> >> Cheers, >> Arun >> >> On Fri, Oct 12, 2018 at 7:15 PM Yaniv Rodenski (JIRA) <j...@apache.org> >> wrote: >> >> > Yaniv Rodenski created AMATERASU-52: >> > --------------------------------------- >> > >> > Summary: Implement AmaContext.datastores >> > Key: AMATERASU-52 >> > URL: >> https://issues.apache.org/jira/browse/AMATERASU-52 >> > Project: AMATERASU >> > Issue Type: Task >> > Reporter: Yaniv Rodenski >> > Assignee: Arun Manivannan >> > Fix For: 0.2.1-incubating >> > >> > >> > AmaContext.datastores should contain the data from datastores.yaml >> > >> > >> > >> > -- >> > This message was sent by Atlassian JIRA >> > (v7.6.3#76005) >> > >> > > > -- > Yaniv Rodenski > > +61 477 778 405 > ya...@shinto.io > >