[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13941543#comment-13941543 ] Renato Javier Marroquín Mogrovejo commented on GORA-94: --- Committed r1579573. Going one step at the time ;) Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.DataStoreTestUtil.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13940451#comment-13940451 ] Alparslan Avcı commented on GORA-94: Hi [~renato2099], +1 for the commit! I hope we will re-think on handling nested structures and find better ways in future (escpecially for gora-0.5 :) ). This issue is not a problem for now. We can keep moving. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.DataStoreTestUtil.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13938969#comment-13938969 ] Alparslan Avcı commented on GORA-94: Hi [~renato2099], The [^GORA-94.DataStoreTestUtil.patch] works smootly, thanks. However, [GORA-246.FailingTest.patch|https://issues.apache.org/jira/secure/attachment/12635008/GORA-246.FailingTest.patch] and [GORA-245.final.patch|https://issues.apache.org/jira/secure/attachment/12635005/GORA-245.final.patch] contains changes on the same code. Would you please reorganize the patches in GORA-245 and GORA-246? Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.DataStoreTestUtil.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13939170#comment-13939170 ] Renato Javier Marroquín Mogrovejo commented on GORA-94: --- Hi [~alparslan.avci], The patch posted in GORA-246 contains just changes for gora-hbase and I have uploaded a rebased version for GORA-245. So if you could test them out, you would make my day because I could commit them without worries :) [~talat] Maybe we should open a different JIRA issue for this one (but add a relation with GORA-94), so we can keep on handling specific patches on specific issues. Wdyt? Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-ssn-of-Employee.patch, GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.DataStoreTestUtil.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13939239#comment-13939239 ] Talat UYARER commented on GORA-94: -- Hey [~renato2099], You are right. We create new issue for this. Actually I didnt know what should i do. :) Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-ssn-of-Employee.patch, GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.DataStoreTestUtil.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13939275#comment-13939275 ] Alparslan Avcı commented on GORA-94: Thanks for the updates [~renato2099]! Just two more comments; firstly I still think that we do not need to check if the _qualifier_ parameter is null or not at _case MAP_ in _HBaseStore.addPutsAndDeletes()_ method as I mentioned in GORA-246. And would you please remove the TODO's in _DataStoreTestUtil_? There is no need for them anymore. :) Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.DataStoreTestUtil.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13923846#comment-13923846 ] Renato Javier Marroquín Mogrovejo commented on GORA-94: --- Hi [~alparslan.avci], I am trying to chip into this task as well, and I wanted to start with this: 2. Revert the test algorithm for testUpdate to the implementation currently in trunk Any suggestions or caveats I should be aware of? Thanks! Renato M. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13923906#comment-13923906 ] Alparslan Avcı commented on GORA-94: Hi [~renato2099], You can just modify testUpdateWebPagePutToNullableMap() and testUpdateWebPageRemoveMapEntry() methods in DataStoreTestUtil class. The last iterations in these test methods has to be changed in order to get only odd indexed urls and headers. This will revert the test algorithm for testUpdate to the implementation currently in trunk. You can also remove TODO's in the class. They are not needed anymore. Thanks! Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13923918#comment-13923918 ] Renato Javier Marroquín Mogrovejo commented on GORA-94: --- Yes I had seen that, but I wondered if there was more than that ;) Nicely refactored man, this is much more understandable now. I think I did a clean-up of this method a while ago, but now this looks much better, great work [~alparslan.avci]!! Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13923923#comment-13923923 ] Renato Javier Marroquín Mogrovejo commented on GORA-94: --- and you meant only: - testUpdateWebPagePutToNotNullableMap - testUpdateWebPagePutToNullableMap right? Because testUpdateWebPageRemoveMapEntry seems fine to me. Wdyt? Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13923922#comment-13923922 ] Alparslan Avcı commented on GORA-94: Thanks [~renato2099] :) I hope that after implementing the tests for other Avro field type cases, it will be clearer. ;) Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13923930#comment-13923930 ] Alparslan Avcı commented on GORA-94: Yes, you are right. There is no problem with _testUpdateWebPageRemoveMapEntry()_. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13923935#comment-13923935 ] Renato Javier Marroquín Mogrovejo commented on GORA-94: --- I updated that and we still have a test failure in CassandraStore and in HBaseStore. I will look into this and commit if I fix them (: Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13923939#comment-13923939 ] Alparslan Avcı commented on GORA-94: I have already prepared a patch for fix in HBaseStore, but have not uploaded yet. If it is ok for you, you can send your changes onto this issue. And after your patch upload, I will upload mine to GORA-246. :) AFAIK, [~talat] is working for Cassandra to fix this problem. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13923945#comment-13923945 ] Talat UYARER commented on GORA-94: -- You are right [~alparslan.avci]. :) I am working on Cassandra's Test. I will be glad if you commit it. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13923947#comment-13923947 ] Renato Javier Marroquín Mogrovejo commented on GORA-94: --- Nice [~talat]! So are you done with this? I was just getting ready to hack it ;) but if not I can move on something else. So do you want me work on this? or do you want to finish this one man? Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13923975#comment-13923975 ] Talat UYARER commented on GORA-94: -- If you are available work on this, you can work [~renato2099]. I guess I will not work until 24.00 GMT+2 tonight Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913179#comment-13913179 ] Lewis John McGibbney commented on GORA-94: -- Hi [~alparslan.avci], thanks for comments. Apologies about my delay in getting back to you. bq. I think, this is the true algorithm for testing the update... Yes I agree here. A simple test at the end, in addition to checking the _odd_ indexed urls in the outlinks map would be for us to get the webpage from the datastore and check the total number of entries in the outlink map. This would mean we add the the algorithm by doing the final check from the persisted object. wdyt? bq. ...since HBaseStore does not clear the map after DirtyMapWrapper.clear() method is called. I think that this is a big that we should correct. #clear methods should remove all entires from the data structure leaving it as an empty data structure (not null). I propose that we 1. Keep your improvements for adding the following methods +DataStoreTestUtil.testUpdateWebPagePutToArray(webPageStore); +DataStoreTestUtil.testUpdateWebPagePutToNotNullableMap(webPageStore); +DataStoreTestUtil.testUpdateWebPagePutToNullableMap(webPageStore); +DataStoreTestUtil.testUpdateWebPageRemoveMapEntry(webPageStore); +DataStoreTestUtil.testUpdateWebPageRemoveField(webPageStore); 2. Revert the test algorithm for testUpdate to the implementation currently in trunk 3. Fix datastores in accordance with this. One final comment. I feel that we should maybe add test cases for +DataStoreTestUtil.testUpdateWebPagePutToNotNullableArray(webPageStore); +DataStoreTestUtil.testUpdateWebPagePutToNullableArray(webPageStore); +DataStoreTestUtil.testUpdateWebPageRemoveArrayEntry(webPageStore); This way we would really make explicit the semantics for *dirtyable* data structures. If you can comment on the stuff above then I will get to work on these implementations. Thanks for your comments. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913197#comment-13913197 ] Lewis John McGibbney commented on GORA-94: -- Joint commit made for GORA-94v11 and GORA-246v5 Committed @revision 1572170 in GORA_94 branch Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[
https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13893157#comment-13893157
]
Alparslan Avcı commented on GORA-94:
Hi [~lewismc], thanks for checking. :)
bq. Do we need to always setHeaders as follows -- webPage.setHeaders(new
HashMapCharSequence, CharSequence()); ? What happens if we just access the
headers field without initiating a new HashMap? Do we still get NPE? I though
the builder build() method would initiate default empty HashMap?
Since headers field is defined as 'nullable' and has 'null' as default value in
JSON ({name: headers, type: [null, {type: map, values: [null,
string]}],default:null}), the builder initiate this field with null value.
So, we always have to set a new HashMap object to headers field at first.
bq. Do you have any idea what is the current issue with PersistentBase#clear()
method? I am puzzled as to why it is not clearing dirty state of object field.
I had fixed this problem in GORA-94v10.patch and it seems that it is clearing
the dirty states of all fields for now. Would you please elaborate this issue?
Upgrade to Apache Avro 1.7.x
Key: GORA-94
URL: https://issues.apache.org/jira/browse/GORA-94
Project: Apache Gora
Issue Type: Improvement
Components: build process, storage
Affects Versions: 0.2
Reporter: Lewis John McGibbney
Assignee: Ed Kohlwey
Fix For: 0.4
Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch,
GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch,
GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch,
GORA_94v5.patch
I am not sure what this involves as of yet but I have a small feeling that
it's going to be some reasonably major work...
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[
https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13893228#comment-13893228
]
Lewis John McGibbney commented on GORA-94:
--
bq. the builder initiate this field with null value. So, we always have to set
a new HashMap object to headers field at first.
Duh, sorry about this one, I just didn't check the default value in the avro
schema. You are completely right with this one, thank you for clarification.
bq. I had fixed this problem in GORA-94v10.patch and it seems that it is
clearing the dirty states of all fields for now. Would you please elaborate
this issue?
Well the v11 patch includes the following TODO
{code:java}
+ webPage.getHeaders().clear(); //TODO clear method does not work
{code}
I therefore wondered if this TODO should be removed?
Also I think we should make a decision on the other 2 TODO's regarding
increments of local variable *j*. This should be trivial to implement.
Once this is done i think we are ready to get this code in to the code base as
the tests are well justified given the change in behavior of certain data
structures which we have been discussing over the last weeks.
Upgrade to Apache Avro 1.7.x
Key: GORA-94
URL: https://issues.apache.org/jira/browse/GORA-94
Project: Apache Gora
Issue Type: Improvement
Components: build process, storage
Affects Versions: 0.2
Reporter: Lewis John McGibbney
Assignee: Ed Kohlwey
Fix For: 0.4
Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch,
GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch,
GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch,
GORA_94v5.patch
I am not sure what this involves as of yet but I have a small feeling that
it's going to be some reasonably major work...
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[
https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13893322#comment-13893322
]
Alparslan Avcı commented on GORA-94:
{code:java}webPage.getHeaders().clear(); //TODO clear method does not work{code}
Actually, this clear method mentioned above is DirtyMapWrapper.clear() method.
In gora trunk, DataStoreTestUtil.testUpdateWebPage() method works as follows:
# puts _even_ indexed urls to outlinks map
# flushes the datastore
# clears outlinks map
# puts _odd_ indexed urls to outlinks map
# flushes the datastore
# asserts _the size of outlinks map_ is equal to _odd indexed urls count_.
I think, this is the true algorithm for testing the update. However, in GORA_94
branch's DataStoreTestUtil.testUpdateWebPage() method; it asserts _the size of
outlinks map_ is equal to _the total urls count_ at the last step. The
incremental of *j* variable has been somehow changed. I think, it was changed
in order to pass the tests since HBaseStore does not clear the map after
DirtyMapWrapper.clear() method is called.
Those TODO's are all about this issue. In my opinion, we have to change the
test algorithm back to the old version and fix the stores according to this.
You can also check my latest
[comment|https://issues.apache.org/jira/browse/GORA-246?focusedCommentId=13878614page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13878614]
and
[patch|https://issues.apache.org/jira/secure/attachment/12624339/GORA-246v5.patch]
in GORA-246 for more info.
Upgrade to Apache Avro 1.7.x
Key: GORA-94
URL: https://issues.apache.org/jira/browse/GORA-94
Project: Apache Gora
Issue Type: Improvement
Components: build process, storage
Affects Versions: 0.2
Reporter: Lewis John McGibbney
Assignee: Ed Kohlwey
Fix For: 0.4
Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch,
GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch,
GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch,
GORA_94v5.patch
I am not sure what this involves as of yet but I have a small feeling that
it's going to be some reasonably major work...
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13892554#comment-13892554 ] Lewis John McGibbney commented on GORA-94: -- Hi [~alparslan.avci], GORA-94v11.patch is much much better. Only thing it lacks in Javadoc ;) Some observations * Do we need to always setHeaders as follows -- webPage.setHeaders(new HashMapCharSequence, CharSequence()); ? What happens if we just access the headers field without initiating a *new* HashMap? Do we still get NPE? I though the builder build() method would initiate default empty HashMap? * Do you have any idea what is the current issue with PersistentBase#clear() method? I am puzzled as to why it is not clearing dirty state of object field. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13883382#comment-13883382 ] Lewis John McGibbney commented on GORA-94: -- Hi [~alparslan.avci], regarding your comments on 22/Jan/14 12:58 and [^GORA-94v11.patch], I feel it is clearer to either fully document these new test operations or else make them into new tests. We want the JUnit method names within DataStoreTestBase and subsequently DataStoreTesUtil to reflect the actual tests which are being carried out... not for them to become large, multipurpose, difficult to maintain tests. We want to make them easy to debug and easy to read and infer what is actually going on. If we don't succeed in doing this, we run the risk of making future upgrades more difficult as we move on. Thanks for the patch. wdyt? Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13880012#comment-13880012 ] Lewis John McGibbney commented on GORA-94: -- Hi [~alparslan.avci]. I also noticed this yesterday when I was porting GORA-119 to GORA_94 branch. I questioned why these were removed from generated data beans. I think it is fine to add these back in. Anyone else have comments? One positive for adding these accessors back in is simple, it means that client applications need to change less of their code if they upgrade thier Gora dependency for 0.4 when we release. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13880397#comment-13880397 ] Lewis John McGibbney commented on GORA-94: -- GORA-94v12.patch Committed @revision 1560819 in GORA_94 branch. Thank you [~alparslan.avci]. I'm going to port GORA-119 here now. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v11.patch, GORA-94v12.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869345#comment-13869345 ] Alparslan Avcı commented on GORA-94: Thank you for your comment about patch [~renato2099] and sorry for the late answer. You've noticed a good point, but as [~martinkl] mentioned in this (http://mail-archives.apache.org/mod_mbox/avro-user/201401.mbox/%3CCAHNLEqv%2BmzCJJPR3Y53WHJg-%2BiOOPgj3%3DxBaM0%3D3NGeamuUjsw%40mail.gmail.com%3E) thread, In Avro, types are not nullable by default (unlike most programming languages). So, you have to specify if the field will be null by default. Moreover, if a field's default value is not set and the builder pattern is used to create a new object, an AvroRuntimeException like Field type: ... pos: .. not set and has no default value will be thrown. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13867925#comment-13867925 ] Renato Javier Marroquín Mogrovejo commented on GORA-94: --- Hi [~alparslan.avci]! First of all, thank you very much for your contributions! They are awesome! (: Just one thing about your patch, even though I think these changes are already committed. You said that for not facing the NPEs you set the values to its default value, so users will always have to set the fields with at least some value i.e. if the users forget to set these values they will get the NPE, am I understanding this correctly? if so, then we should try fixing the possibility of the users not having to specify a default value if it is null. Do you know what I mean? Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[
https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13866429#comment-13866429
]
Alparslan Avcı commented on GORA-94:
And also about my patch, in order to get NULL value as default for a 'map' type
field,
type: [null, {type:map, values:string}], default:null
has to be used as type in .avsc definition. And in order to get 'empty map
object' as default for a 'map' type field;
type: {type:map, values:string}, default:{}}
has to be used.
Upgrade to Apache Avro 1.7.x
Key: GORA-94
URL: https://issues.apache.org/jira/browse/GORA-94
Project: Apache Gora
Issue Type: Improvement
Components: build process, storage
Affects Versions: 0.2
Reporter: Lewis John McGibbney
Assignee: Ed Kohlwey
Fix For: 0.4
Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch,
GORA-94.patch, GORA-94v10.patch, GORA-94v6.diff, GORA-94v7.patch,
GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch
I am not sure what this involves as of yet but I have a small feeling that
it's going to be some reasonably major work...
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13866701#comment-13866701 ] Lewis John McGibbney commented on GORA-94: -- GORA-94v10.patch Committed @revision 1556830 in GORA_94 branch. I also re-compiled the data beans in gora-core and gora-tutorial. All tests pass in gora-core and gora-hbase. Thank you very much [~alparslan.avci] for the patches and the time. If you are interested in working with gora-cassandra. Your input would be very much appreciated over on issue GORA-245. :) Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13866725#comment-13866725 ] Alparslan Avcı commented on GORA-94: Thank you for your help [~lewismc], I'm happy to contribute. :) I'm working on using GORA_94 branch on Nutch now. I think [~talat] would have some effort on GORA-245 in a few days. :) Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865706#comment-13865706 ] Talat UYARER commented on GORA-94: -- Hi [~lewismc] I think you forget share link. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v10.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13864072#comment-13864072 ] Alfonso Nishikawa commented on GORA-94: --- Hi! I find that GORA-94-v4.patch about the compiler is essentially wrong breaking back-compatibility. Tests should not be changed (if not to add more tests), and I see (not all) interfaces being changed. Pretty implementation, but I believe it miss the mark. The compiler should not affect any public part. What are your thoughts? Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812665#comment-13812665 ] yasin tamer commented on GORA-94: - if i could touch a small helps, this would have made happy me :) thanks for your motivations and supports, special thanks to Lewis ;) Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811559#comment-13811559 ] Lewis John McGibbney commented on GORA-94: -- Hi Yasin. I'm working with your patch right now. There is a problem though. I use (and so does Gora) AVN for source code management. This means that your patches cannot be applied cleanly to a local copy of the code. Can you please generate your patches with git diff --no-prefix myBeautifulPatch.patch This would be a huge favour + help for us trying to apply your patches. Thank you so much Yasin. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811645#comment-13811645 ] Lewis John McGibbney commented on GORA-94: -- v9 patch Committed @revision 1538057 in GORA_94 branch HEAD Thanks for the input Yasin. Great help :) Lets move on to HBase ;) Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA-94.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch, GORA-94v9.patch, GORA_94v5.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13805363#comment-13805363 ] Lewis John McGibbney commented on GORA-94: -- Hi Yasin, thanks for this. Great work. Glad to see that you've moved on considerably with your use of Gora. I have some comments: * The way that the patch has been created means I cannot apply it locally to my checkout of GORA_94 branch. If you are in $GORA_HOME then you can generate a patch simply by doing svn diff myBeautifulPatch.patch or if you are in git, then please do git diff --no-prefix myBeautifulPatch.patch * The patch seems to combine numerous things which are not useful at this point. For example, it includes a bunch of MANIFEST information as well as pom.xml configuration to suit. This is not required for the Avro upgrade. * Is it possible for us to upgrade ONLY Avro with this umbrella issue? Is it absolutely required that an upgrade of HBase happens at the same time? I was hoping that this would not be the case but can live with it if it is. If we can begin with the above then we can move on. Finally, did you get a chance to look at the most recent code for the GORA-94 branch? AFAIK the Avro upgrade for gora-core is nearly complete with only one failing test e.g. testSerdeMultipleWebPages Again, thanks for your contributions. People like yourself pushing this forward is exactly what we require here. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94.patch, GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA_94v5.patch, GORA-94v6.diff, GORA-94v7.patch, GORA-94v8.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804175#comment-13804175 ] Lewis John McGibbney commented on GORA-94: -- patch v7 Committed @revision 1535375 in GORA_94 branch Two failing tests are Tests in error: testSerdeMultipleWebPages(org.apache.gora.mapreduce.TestPersistentSerialization): null of map in field outlinks of org.apache.gora.examples.generated.WebPage testCountQuery(org.apache.gora.avro.mapreduce.TestDataFileAvroStoreMapReduce) Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94.patch, GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch, GORA_94v5.patch, GORA-94v6.diff, GORA-94v7.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13747297#comment-13747297 ] yasin tamer commented on GORA-94: - ok we are waiting for your commit eagerly, so do you have any time estimation for commit Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94.patch, GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13745153#comment-13745153 ] Henry Saputra commented on GORA-94: --- I believe [~lewismc] has created gora/branches/GORA_94/ branch for Avro upgrade Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94.patch, GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13745336#comment-13745336 ] Lewis John McGibbney commented on GORA-94: -- Hi, yeah there is a branch created for this work now which can be found here [0]. This is in sync with trunk and has no 94 stuff committed as of yet. I have been working on a gora-core and cassandra patch which I will push to the branch. It will then be a case of taking the time to work with the patches attached to the sub tasks as part of this issue. That fact that you guys are willing to pick up the hbase stuff is really really great. I am not able to spend half as much time on this as I want to right now... which is really frustrating. I will try and push my changes to the 94 branch ASAP. [0] http://svn.apache.org/repos/asf/gora/branches/GORA_94/ Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Improvement Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.4 Attachments: GORA-94.patch, GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13491928#comment-13491928 ] Lewis John McGibbney commented on GORA-94: -- Hi Ed. Now that Dynamodb has been committed and the changes made to the core-module it would be excellent to pick this up. If you are able, it would be excellent to get your contribution reviewed, tested and hopefully committed in the short term. wdyt? Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Sub-task Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.3 Attachments: GORA-94.patch, GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435513#comment-13435513 ] Lewis John McGibbney commented on GORA-94: -- I just noticed that the most recent (v4) patch also incorporates GORA-160. It would be nice to deal with that one first so I will progress on that basis. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Sub-task Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.3 Attachments: GORA-94.patch, GORA-94-v2.patch, GORA-94-v3.patch, GORA-94-v4.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420588#comment-13420588 ] Ed Kohlwey commented on GORA-94: I'll be maintaining this on top of the trunk in my github repository, running the tests after each merge. If you can integrate your findings as regression tests in trunk that will go a long way towards getting this merged in. I think the tombstone system could possibly be refactored out, but much of this patch depends directly on Avro. Also, much of the code in it is generated, so its not as big and scary as it seems ;). Nothing *should* have changed in terms of data compatibility; obviously there's some pretty significant API changes though. Do you guys want to do a hangout or IRC chat to discuss? Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Sub-task Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.3 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419248#comment-13419248 ] Ferdy Galema commented on GORA-94: -- Wow that is HUGE patch. I tried to apply on trunk head but there where some merge problems already. I realize that it is a bit difficult to keep such a large patch in sync, so I think it would be best if we try to review and commit this as soon as possible. I will be able to do some testing with HBase next couple of days. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Sub-task Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.3 Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[
https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419355#comment-13419355
]
Lewis John McGibbney commented on GORA-94:
--
Hi.
{bq}Wow that is HUGE patch.{bq} ;0)
This was also my major concern. I need to be honest and say that for us to pull
this one off I think it's going to need to be much more incremental. Inevitably
it affects all aspects of the entire Gora framework so I will also step up to
try and test AMAP the gora-cassandra stuff.
Upgrade to Apache Avro 1.7.x
Key: GORA-94
URL: https://issues.apache.org/jira/browse/GORA-94
Project: Apache Gora
Issue Type: Sub-task
Components: build process, storage
Affects Versions: 0.2
Reporter: Lewis John McGibbney
Assignee: Ed Kohlwey
Fix For: 0.3
Attachments: GORA-94-v2.patch, GORA-94-v3.patch, GORA-94.patch
I am not sure what this involves as of yet but I have a small feeling that
it's going to be some reasonably major work...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418159#comment-13418159 ] Ferdy Galema commented on GORA-94: -- Ok. About the CharSequence and ByteBuffer types: Is it possible to make them immutable? Perhaps by overriding the mutators and throw exceptions. I already noticed that you are returning CharSequence (which has no mutator methods in the interface), instead of Utf8. Is this not a hint to the user that it should not be casting/changing the value? ByteBuffer can already be read-only. This might change client usage (only if a client was directly changing the mutable in the first place), but perhaps this is worth the change. Anyway we can always try to solve this problem later and leave the semantics as they currently are. I think it is no problem that the format of intermediate output is changed. If there is the case that the final input/output rows have changed then this is something we just have to find out. It might depend on the store. I will test this with the HBaseStore. Thanks for the git info. I will manage to create diffs for me. However if you a have final version to be tested I think it's best to upload a patch yourself. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Sub-task Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.3 Attachments: GORA-94-v2.patch, GORA-94.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418280#comment-13418280 ] Ed Kohlwey commented on GORA-94: Ferdy, Yes, ByteBuffer (as you note) could be protected to make them immutable. I think I would just do a copy of any buffer passed in and then do a .asReadOnlyBuffer() when the getters are called. CharSequences can also be stored as strings using the toString() on the setter to guarantee that the stored copy is immutable. By default Avro will put in Utf8's but we can do some processing on the getters to make sure no users gain access them. This might be the best thing for the moment, although it will have negative implications for the garbage collector in a map/reduce type context. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Sub-task Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.3 Attachments: GORA-94-v2.patch, GORA-94.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418874#comment-13418874 ] Ed Kohlwey commented on GORA-94: I'm getting a 500 when I try to submit this on reviewboard, so here it is as a patch. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Sub-task Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.3 Attachments: GORA-94-v2.patch, GORA-94.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-94) Upgrade to Apache Avro 1.7.x
[ https://issues.apache.org/jira/browse/GORA-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13417963#comment-13417963 ] Ed Kohlwey commented on GORA-94: actually, in a few places I iterate from 1 to n to avoid looking at the dirty byte array. I'm probably going to add a constant somewhere called FIRST_UNMANAGED_FIELD_INDEX or something to make this more explicit. I've added some map wrapper classes to the persistency.impl package that track structural changes to maps. There's no good way at the moment to track changes to ByteBuffer, and I'm not totally sure of a way to track changes to CharSequences (I'm currently thinking just extend Utf8). I was considering using AspectJ but the best way to do it would probably be as a runtime bytecode weaving thing which tends to be really slow. The option of having a class that doesn't inherit from ByteBuffer is also an option, but that will probably unnecessarily frustrate users. Another option could be to use a heuristic dirty for these based on a hash thats computed whenever clearDirty() is called. Serialization level compatibility is an issue. As I mentioned, this moves storage of the dirty field information into Avro instead of writing it out separately right before the serialized Avro object. But my understanding from a discussion on the mailing list is that that was only actually written as part of the Map/reduce intermediate output. There's also an issue with existing avro objects that don't have the dirty field. In that case I guess you would need to use a resolving decoder to read them in. If you check out the code using Git, you have two branches: my branch and the one you want to diff against. Then do git diff branch1 branch2 /path/to/my/new/patch You can also use a tag name or a ref id in lieu of either of the branch names (which you can hunt down in git log) Or if you tell me what you want the diff against I'll make it and upload it. The current version still has some bugs related to the cassandra patch that just got merged into trunk, and the generated sample classes are missing from the tutorial project. Upgrade to Apache Avro 1.7.x Key: GORA-94 URL: https://issues.apache.org/jira/browse/GORA-94 Project: Apache Gora Issue Type: Sub-task Components: build process, storage Affects Versions: 0.2 Reporter: Lewis John McGibbney Assignee: Ed Kohlwey Fix For: 0.3 Attachments: GORA-94-v2.patch, GORA-94.patch I am not sure what this involves as of yet but I have a small feeling that it's going to be some reasonably major work... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
