Re: [GSoC 2018] Ignite backend - Status
Hi Carlos, I have reviewed your PR [1] added some suggestions, please address them and let s get your work merged to master. This is excellent work Carlos. Thank you for all the hard work you have put into in this summer. [1] https://github.com/apache/gora/pull/134 Regards Kevin On Mon, Jul 23, 2018 at 1:35 AM carlos muñoz wrote: > Hi all. > > I am working on the changes that you guys suggested and other small > improvements. I hope I will be pushing my commits by Thursday. In the > meantime, I would like to update Gora's Wiki with details of the project > within the GSoC section [1], so I would highly appreciate you grant me > write access (my cwiki username: carlosrmng email: carlosr...@gmail.com). > > BTW, thanks so much for your feedback and support :D > > Regards, > > Carlos. > > [1] https://cwiki.apache.org/confluence/display/GORA/Google+Summer+of+Code > > > 2018-07-17 3:35 GMT-05:00 Alfonso Nishikawa : > > > > > @Lewis, @Nishadi, wow you did a very thorough review, congrats =) I was > > liking your comments, but I guess is better just to say I subscribe all > > your coments (except one, but already commented :) > > > > Loooking forward those changes. Carlos, cheer up! :) > > Thanks! > > > > Regards, > > > > Alfonso > > > > > > > > 2018-07-15 7:05 GMT-01:00 Nishadi Kirielle : > > > >> Hi Carlos, > >> > >> This is great work..!!! > >> I have gone through your code and added some comments. Please have a > look > >> at them and let me know if you need further clarifications. > >> > >> Regards, > >> Nishadi > >> > >> On Sun, Jul 8, 2018 at 1:04 AM Lewis John Mcgibbney < > >> lewis.mcgibb...@gmail.com> wrote: > >> > >> > Hi Carlos, I’ll reciew by end of weekend this is an excellent effort > >> thank > >> > you. > >> > Lewis > >> > > >> > On Sat, Jul 7, 2018 at 07:30 carlos muñoz > wrote: > >> > > >> >> Hi. > >> >> > >> >> Just a quick update of the project. > >> >> > >> >> I just have sent a PR [1] with the initial Ignite Backend > >> implementation. > >> >> All the tests are passing, but I would like to check the code in > other > >> to > >> >> improve it and catch possible errors. So I would highly appreciacte > if > >> you > >> >> review it. > >> >> Also, I would like to work on some documentation of the new backend. > >> >> Where does the Gora's website code live? and how can I edit it?. > >> >> > >> >> Best, > >> >> Carlos > >> >> > >> >> [1] https://github.com/apache/gora/pull/134 > >> >> > >> > -- > >> > > >> > *Lewis* > >> > Dr. Lewis J. McGibbney Ph.D, B.Sc > >> > *Skype*: lewis.john.mcgibbney > >> > > >> > > >> > > >> > > >> > > > > >
Re: [GSoC 2018] Ignite backend - Status
Hi all. I am working on the changes that you guys suggested and other small improvements. I hope I will be pushing my commits by Thursday. In the meantime, I would like to update Gora's Wiki with details of the project within the GSoC section [1], so I would highly appreciate you grant me write access (my cwiki username: carlosrmng email: carlosr...@gmail.com). BTW, thanks so much for your feedback and support :D Regards, Carlos. [1] https://cwiki.apache.org/confluence/display/GORA/Google+Summer+of+Code 2018-07-17 3:35 GMT-05:00 Alfonso Nishikawa : > > @Lewis, @Nishadi, wow you did a very thorough review, congrats =) I was > liking your comments, but I guess is better just to say I subscribe all > your coments (except one, but already commented :) > > Loooking forward those changes. Carlos, cheer up! :) > Thanks! > > Regards, > > Alfonso > > > > 2018-07-15 7:05 GMT-01:00 Nishadi Kirielle : > >> Hi Carlos, >> >> This is great work..!!! >> I have gone through your code and added some comments. Please have a look >> at them and let me know if you need further clarifications. >> >> Regards, >> Nishadi >> >> On Sun, Jul 8, 2018 at 1:04 AM Lewis John Mcgibbney < >> lewis.mcgibb...@gmail.com> wrote: >> >> > Hi Carlos, I’ll reciew by end of weekend this is an excellent effort >> thank >> > you. >> > Lewis >> > >> > On Sat, Jul 7, 2018 at 07:30 carlos muñoz wrote: >> > >> >> Hi. >> >> >> >> Just a quick update of the project. >> >> >> >> I just have sent a PR [1] with the initial Ignite Backend >> implementation. >> >> All the tests are passing, but I would like to check the code in other >> to >> >> improve it and catch possible errors. So I would highly appreciacte if >> you >> >> review it. >> >> Also, I would like to work on some documentation of the new backend. >> >> Where does the Gora's website code live? and how can I edit it?. >> >> >> >> Best, >> >> Carlos >> >> >> >> [1] https://github.com/apache/gora/pull/134 >> >> >> > -- >> > >> > *Lewis* >> > Dr. Lewis J. McGibbney Ph.D, B.Sc >> > *Skype*: lewis.john.mcgibbney >> > >> > >> > >> > >> > >
Re: [GSoC 2018] Ignite backend - Status
@Lewis, @Nishadi, wow you did a very thorough review, congrats =) I was liking your comments, but I guess is better just to say I subscribe all your coments (except one, but already commented :) Loooking forward those changes. Carlos, cheer up! :) Thanks! Regards, Alfonso 2018-07-15 7:05 GMT-01:00 Nishadi Kirielle : > Hi Carlos, > > This is great work..!!! > I have gone through your code and added some comments. Please have a look > at them and let me know if you need further clarifications. > > Regards, > Nishadi > > On Sun, Jul 8, 2018 at 1:04 AM Lewis John Mcgibbney < > lewis.mcgibb...@gmail.com> wrote: > > > Hi Carlos, I’ll reciew by end of weekend this is an excellent effort > thank > > you. > > Lewis > > > > On Sat, Jul 7, 2018 at 07:30 carlos muñoz wrote: > > > >> Hi. > >> > >> Just a quick update of the project. > >> > >> I just have sent a PR [1] with the initial Ignite Backend > implementation. > >> All the tests are passing, but I would like to check the code in other > to > >> improve it and catch possible errors. So I would highly appreciacte if > you > >> review it. > >> Also, I would like to work on some documentation of the new backend. > >> Where does the Gora's website code live? and how can I edit it?. > >> > >> Best, > >> Carlos > >> > >> [1] https://github.com/apache/gora/pull/134 > >> > > -- > > > > *Lewis* > > Dr. Lewis J. McGibbney Ph.D, B.Sc > > *Skype*: lewis.john.mcgibbney > > > > > > > > >
Re: [GSoC 2018] Ignite backend - Status
Hi Carlos, This is great work..!!! I have gone through your code and added some comments. Please have a look at them and let me know if you need further clarifications. Regards, Nishadi On Sun, Jul 8, 2018 at 1:04 AM Lewis John Mcgibbney < lewis.mcgibb...@gmail.com> wrote: > Hi Carlos, I’ll reciew by end of weekend this is an excellent effort thank > you. > Lewis > > On Sat, Jul 7, 2018 at 07:30 carlos muñoz wrote: > >> Hi. >> >> Just a quick update of the project. >> >> I just have sent a PR [1] with the initial Ignite Backend implementation. >> All the tests are passing, but I would like to check the code in other to >> improve it and catch possible errors. So I would highly appreciacte if you >> review it. >> Also, I would like to work on some documentation of the new backend. >> Where does the Gora's website code live? and how can I edit it?. >> >> Best, >> Carlos >> >> [1] https://github.com/apache/gora/pull/134 >> > -- > > *Lewis* > Dr. Lewis J. McGibbney Ph.D, B.Sc > *Skype*: lewis.john.mcgibbney > > > >
Re: [GSoC 2018] Ignite backend - Status
Hi Carlos, I’ll reciew by end of weekend this is an excellent effort thank you. Lewis On Sat, Jul 7, 2018 at 07:30 carlos muñoz wrote: > Hi. > > Just a quick update of the project. > > I just have sent a PR [1] with the initial Ignite Backend implementation. > All the tests are passing, but I would like to check the code in other to > improve it and catch possible errors. So I would highly appreciacte if you > review it. > Also, I would like to work on some documentation of the new backend. Where > does the Gora's website code live? and how can I edit it?. > > Best, > Carlos > > [1] https://github.com/apache/gora/pull/134 > -- *Lewis* Dr. Lewis J. McGibbney Ph.D, B.Sc *Skype*: lewis.john.mcgibbney
Re: [GSoC 2018] Ignite Backend - Status
Hi, Carlos. Nice analysis. It looks a good to me to start with the serialized approach. Personally, I tend to be too much ambitious, so I encourage to keep it simple :) Reading your analysis brings me lots of ideas (always time consuming ideas), like creating two Stores for each case with blahblahbla. So: *avoid my errors, keep it simple* ;) For sure, if later you have more time to spend, things can be improved, but no need to rush, I guess. Just my oppinion. I know nothing :P Maybe others with knowledge of other Stores think the other way around. Regards, Alfonso Nishikawa 2018-06-26 3:57 GMT-01:00 carlos muñoz : > Hi all > > As Alfonso mentioned the Ignite backend could implement the 3-union fields > either by using joined tables or storing serialized objects as binary. Here > some comments on those approaches. > > Using joined tables would definitely has an impact in performance due the > fact distributed joins [1] imply some subquery/aggregation overhead. > Moreover, the efficiency and results of such joins will depend on the > specific configuration used into the Ignite Cluster. On the other hand, > from a developing point of view, explicitly referencing persistent entities > within class mapping (JPA - like ) will introduce some complexity to the > project (fetching strategies, relations handling, etc.) and as far I > understand Apache Gore was not conceived in that way. > > Serializing objects on the other hand sticks to other backends > implementation as I noticed while analyzing Gora code. It seems to keep > code simple and IO operations efficient due the fact that embedded > persistent entities are stored as simple literals into the data stores. In > addition, Ignite has the data type BINARY [2] which suits the needs of such > fields. > > In a nutshell, I suggest taking the serialization approach and adapting > other backends implementation of 3-union fields in the Ignite backend. But > I am not really an expert on the subject so please feel free to give your > feedback. > > > [1] https://apacheignite-sql.readme.io/docs/distributed-joins > [2] https://apacheignite-sql.readme.io/docs/data-types#section-binary > > > 2018-06-06 5:05 GMT-05:00 Alfonso Nishikawa : > > > Hi, Carlos. > > > > About the mapping files and the fields 'boss' and 'webpage', yes, they > > reference two other Persistent entities. You can see the schema of > Employee > > [1], were it has a boss (referencing itself) and a webpage (with the > > WebPage schema defined there). > > > > How to handle this depends on the actual backend. In HBase we serialize > > with avro each of the fields that are Persistent entities as shown at [2] > > (we get there from [3] > [4] > [5], but this is for HBase). > > On the other hand, if I am not worng, MongoDB Store has no problem with > > hierarchical data, so it just writes it as it is. > > > > In your case, it depends on the features of your backend and what > approach > > wold be better overall. I am seeing that Ignite has SQL + key/value > > properties... how interesting :) > > I know nothing about, but I would think about the performance costs of > > joined tables, the possibility of mapping each persistent into a table, > or > > the approach like HBase serializing (as a key/value store). Or maybe > Ignite > > has some functionality for complex data. > > Just thinking about it without knowing. > > > > If I left things unclear don't doubt to ask :) > > > > Regards, > > > > Alfonso > > > > > > [1] > > https://github.com/apache/gora/blob/apache-gora-0.8/ > > gora-core/src/examples/avro/employee.json > > [2] > > https://github.com/apache/gora/blob/apache-gora-0.8/ > > gora-hbase/src/main/java/org/apache/gora/hbase/util/ > > HBaseByteInterface.java#L233 > > [3] > > https://github.com/apache/gora/blob/apache-gora-0.8/ > > gora-hbase/src/main/java/org/apache/gora/hbase/store/ > HBaseStore.java#L240 > > [4] > > https://github.com/apache/gora/blob/apache-gora-0.8/ > > gora-hbase/src/main/java/org/apache/gora/hbase/store/ > HBaseStore.java#L262 > > [5] > > https://github.com/apache/gora/blob/apache-gora-0.8/ > > gora-hbase/src/main/java/org/apache/gora/hbase/store/ > HBaseStore.java#L331 > > > > > > 2018-06-06 4:38 GMT-01:00 carlos muñoz : > > > > > Hi all, > > > > > > Currently I am working on the Ignite backend, I am using my own fork > [1]. > > > This is what I have done so far: > > > * Read mapping files. > > > * Read configuration files (Ignite parameters). > > > * Initialize Ignite DataStore through JDBC connection. > > > * Implement some trivial methods (#close(), #flush(), > > #getSchemaName()...). > > > * Set up an embedded Ignite instance within test packages. > > > > > > The code is broken because I created a DataStoreTestBase implementation > > for > > > Ignite and due the fact that most DataStore methods are not implemented > > yet > > > the tests are failing. But the mapping/configuration reading and > > connection > > > initialization are working fine. > > > > > > I noticed
Re: [GSoC 2018] Ignite Backend - Status
Hi all As Alfonso mentioned the Ignite backend could implement the 3-union fields either by using joined tables or storing serialized objects as binary. Here some comments on those approaches. Using joined tables would definitely has an impact in performance due the fact distributed joins [1] imply some subquery/aggregation overhead. Moreover, the efficiency and results of such joins will depend on the specific configuration used into the Ignite Cluster. On the other hand, from a developing point of view, explicitly referencing persistent entities within class mapping (JPA - like ) will introduce some complexity to the project (fetching strategies, relations handling, etc.) and as far I understand Apache Gore was not conceived in that way. Serializing objects on the other hand sticks to other backends implementation as I noticed while analyzing Gora code. It seems to keep code simple and IO operations efficient due the fact that embedded persistent entities are stored as simple literals into the data stores. In addition, Ignite has the data type BINARY [2] which suits the needs of such fields. In a nutshell, I suggest taking the serialization approach and adapting other backends implementation of 3-union fields in the Ignite backend. But I am not really an expert on the subject so please feel free to give your feedback. [1] https://apacheignite-sql.readme.io/docs/distributed-joins [2] https://apacheignite-sql.readme.io/docs/data-types#section-binary 2018-06-06 5:05 GMT-05:00 Alfonso Nishikawa : > Hi, Carlos. > > About the mapping files and the fields 'boss' and 'webpage', yes, they > reference two other Persistent entities. You can see the schema of Employee > [1], were it has a boss (referencing itself) and a webpage (with the > WebPage schema defined there). > > How to handle this depends on the actual backend. In HBase we serialize > with avro each of the fields that are Persistent entities as shown at [2] > (we get there from [3] > [4] > [5], but this is for HBase). > On the other hand, if I am not worng, MongoDB Store has no problem with > hierarchical data, so it just writes it as it is. > > In your case, it depends on the features of your backend and what approach > wold be better overall. I am seeing that Ignite has SQL + key/value > properties... how interesting :) > I know nothing about, but I would think about the performance costs of > joined tables, the possibility of mapping each persistent into a table, or > the approach like HBase serializing (as a key/value store). Or maybe Ignite > has some functionality for complex data. > Just thinking about it without knowing. > > If I left things unclear don't doubt to ask :) > > Regards, > > Alfonso > > > [1] > https://github.com/apache/gora/blob/apache-gora-0.8/ > gora-core/src/examples/avro/employee.json > [2] > https://github.com/apache/gora/blob/apache-gora-0.8/ > gora-hbase/src/main/java/org/apache/gora/hbase/util/ > HBaseByteInterface.java#L233 > [3] > https://github.com/apache/gora/blob/apache-gora-0.8/ > gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L240 > [4] > https://github.com/apache/gora/blob/apache-gora-0.8/ > gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L262 > [5] > https://github.com/apache/gora/blob/apache-gora-0.8/ > gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L331 > > > 2018-06-06 4:38 GMT-01:00 carlos muñoz : > > > Hi all, > > > > Currently I am working on the Ignite backend, I am using my own fork [1]. > > This is what I have done so far: > > * Read mapping files. > > * Read configuration files (Ignite parameters). > > * Initialize Ignite DataStore through JDBC connection. > > * Implement some trivial methods (#close(), #flush(), > #getSchemaName()...). > > * Set up an embedded Ignite instance within test packages. > > > > The code is broken because I created a DataStoreTestBase implementation > for > > Ignite and due the fact that most DataStore methods are not implemented > yet > > the tests are failing. But the mapping/configuration reading and > connection > > initialization are working fine. > > > > I noticed that the mapping files (gora-XX-mapping.xml) have two fields > > 'boss' and 'webpage', which I assume are references to other objects > (other > > mapped classes). I was wondering how the backend should deal with them. > Is > > there some documentation regarding that?. > > > > Regards, > > Carlos > > > > > > [1] https://github.com/carlosrmng/gora/tree/GORA-535 > > >
Re: [GSoC 2018] Ignite Backend - Status
Hi, Carlos. About the mapping files and the fields 'boss' and 'webpage', yes, they reference two other Persistent entities. You can see the schema of Employee [1], were it has a boss (referencing itself) and a webpage (with the WebPage schema defined there). How to handle this depends on the actual backend. In HBase we serialize with avro each of the fields that are Persistent entities as shown at [2] (we get there from [3] > [4] > [5], but this is for HBase). On the other hand, if I am not worng, MongoDB Store has no problem with hierarchical data, so it just writes it as it is. In your case, it depends on the features of your backend and what approach wold be better overall. I am seeing that Ignite has SQL + key/value properties... how interesting :) I know nothing about, but I would think about the performance costs of joined tables, the possibility of mapping each persistent into a table, or the approach like HBase serializing (as a key/value store). Or maybe Ignite has some functionality for complex data. Just thinking about it without knowing. If I left things unclear don't doubt to ask :) Regards, Alfonso [1] https://github.com/apache/gora/blob/apache-gora-0.8/gora-core/src/examples/avro/employee.json [2] https://github.com/apache/gora/blob/apache-gora-0.8/gora-hbase/src/main/java/org/apache/gora/hbase/util/HBaseByteInterface.java#L233 [3] https://github.com/apache/gora/blob/apache-gora-0.8/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L240 [4] https://github.com/apache/gora/blob/apache-gora-0.8/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L262 [5] https://github.com/apache/gora/blob/apache-gora-0.8/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L331 2018-06-06 4:38 GMT-01:00 carlos muñoz : > Hi all, > > Currently I am working on the Ignite backend, I am using my own fork [1]. > This is what I have done so far: > * Read mapping files. > * Read configuration files (Ignite parameters). > * Initialize Ignite DataStore through JDBC connection. > * Implement some trivial methods (#close(), #flush(), #getSchemaName()...). > * Set up an embedded Ignite instance within test packages. > > The code is broken because I created a DataStoreTestBase implementation for > Ignite and due the fact that most DataStore methods are not implemented yet > the tests are failing. But the mapping/configuration reading and connection > initialization are working fine. > > I noticed that the mapping files (gora-XX-mapping.xml) have two fields > 'boss' and 'webpage', which I assume are references to other objects (other > mapped classes). I was wondering how the backend should deal with them. Is > there some documentation regarding that?. > > Regards, > Carlos > > > [1] https://github.com/carlosrmng/gora/tree/GORA-535 >
