Jenkins build is back to normal : gora-trunk #197
See https://builds.apache.org/job/gora-trunk/197/
[jira] [Closed] (GORA-105) DataStoreFactory does not properly support multiple stores
[
https://issues.apache.org/jira/browse/GORA-105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferdy Galema closed GORA-105.
-
Resolution: Fixed
committed.
DataStoreFactory does not properly support multiple stores
--
Key: GORA-105
URL: https://issues.apache.org/jira/browse/GORA-105
Project: Apache Gora
Issue Type: Bug
Components: schema, storage
Reporter: Ferdy Galema
Priority: Blocker
Fix For: 0.2
Attachments: GORA-105-v2.patch, GORA-105.patch
DataStoreFactory has a single, static properties field. This is completely
unacceptable, because that way when multiple stores are instantiated in the
same JVM, the last store instance will overwrite the default.schema
property. This causes that all the previous stores will read a misconfigured
default schema property. Beside this it may cause several other nasty future
bugs. In my opinion this is a blocker because the methods on DataStoreFactory
suggest that it can handle multiple stores, when as a matter fact it doesn't.
I will attach and commit a patch that fixes this problem. It only modifies
gora-core. All stores directly benefit from this bugfix because of
DataStoreBase. This patch fixes the following property related problems.
-It introduces a static method createProps in DataStoreFactory. This is the
equivalent of Configuration.create(). Everyone can create a new properties
object and set everything interesting on it and pass it on to whatever stores
they like, instead of ALL stores.
-It fixes the method javadoc of DataStoreBase#getSchemaName(String
mappingSchemaName, Class? persistentClass). The previous description was
simply wrong.
-It SERIALIZES the properties field of DataStoreBase instead of grabbing the
static DataStoreFactory.properties field. This has the additional benefit of
making sure that the store can be used correctly with runtime modified
properties in a mapreduce context.
-It removes the caching functionality of DataStoreFactory. Because of the
dynamic configuration in the Properties and Configuration object, it is very
difficult to implement a correct key hash for the cache. At the moment it
only uses the triple {datastoreClass, keyClass,valueClass} as a key hash.
Multiple stores cannot be properly supported when the factory uses badly
implemented hash keys. (For example, one might instantiate 2 SqlStores, both
using the exact same {datastoreClass, keyClass,valueClass} triple, but
pointing to different databases. When one is about the instantiate the second
datastore, it will faulty return the first datastore from cache). We can
always reintroduce caching functionality when we can implement a proper key.
The patch passes all tests. Will commit when there are no objections.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [GSoC 2012] Gora-DynamoDB datastore
Apologies for not getting on here as well. Been away for a bit there, but yeah, thanks Renato for coming forward with the willingness to participate and a bit about yourself etc. Would be great to see this project take off and I really think this gets the ball rolling in the right direction. Lewis On Sat, Mar 24, 2012 at 1:37 AM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: Hey Renato, Awesome to hear! Really looking forward to you becoming an active contributor and part of the project... Cheers, Chris On Mar 23, 2012, at 4:05 PM, Renato Marroquín Mogrovejo wrote: Hi everyone, My name is Renato Marroquin and I would also like to apply for GSoC this year. I have a bit of experience with MapReduce, specifically Hadoop and Pig-Latin. I have completed my masters degree on databases (specifically cloud data management) and I really want to keep on working with hadoop related technologies specially with the ones which involve the open source community (: While doing my masters I was working for a bioinformatics laboratory where we tested a couple of NoSQL solutions (HBase and Cassandra) for genomic data. But going through the code of both Cassandra and HBase they really have come a long way. We also worked creating different data flows for scientific data analysis using MapReduce and Pig-Latin. Hope to be able to become a more active member on the list, so we can make Gora a bigger project. Renato M. ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -- *Lewis*
[jira] [Commented] (GORA-105) DataStoreFactory does not properly support multiple stores
[
https://issues.apache.org/jira/browse/GORA-105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241317#comment-13241317
]
Hudson commented on GORA-105:
-
Integrated in gora-trunk #198 (See
[https://builds.apache.org/job/gora-trunk/198/])
GORA-105 DataStoreFactory does not properly support multiple stores
(Revision 1306866)
Result = SUCCESS
ferdy :
Files :
* /gora/trunk/CHANGES.txt
*
/gora/trunk/gora-core/src/main/java/org/apache/gora/store/DataStoreFactory.java
*
/gora/trunk/gora-core/src/main/java/org/apache/gora/store/impl/DataStoreBase.java
* /gora/trunk/gora-core/src/main/java/org/apache/gora/util/WritableUtils.java
* /gora/trunk/gora-core/src/test/java/org/apache/gora/GoraTestDriver.java
*
/gora/trunk/gora-core/src/test/java/org/apache/gora/avro/mapreduce/TestDataFileAvroStoreMapReduce.java
*
/gora/trunk/gora-core/src/test/java/org/apache/gora/avro/store/TestAvroStore.java
*
/gora/trunk/gora-core/src/test/java/org/apache/gora/store/TestDataStoreFactory.java
*
/gora/trunk/gora-core/src/test/java/org/apache/gora/util/TestWritableUtils.java
DataStoreFactory does not properly support multiple stores
--
Key: GORA-105
URL: https://issues.apache.org/jira/browse/GORA-105
Project: Apache Gora
Issue Type: Bug
Components: schema, storage
Reporter: Ferdy Galema
Priority: Blocker
Fix For: 0.2
Attachments: GORA-105-v2.patch, GORA-105.patch
DataStoreFactory has a single, static properties field. This is completely
unacceptable, because that way when multiple stores are instantiated in the
same JVM, the last store instance will overwrite the default.schema
property. This causes that all the previous stores will read a misconfigured
default schema property. Beside this it may cause several other nasty future
bugs. In my opinion this is a blocker because the methods on DataStoreFactory
suggest that it can handle multiple stores, when as a matter fact it doesn't.
I will attach and commit a patch that fixes this problem. It only modifies
gora-core. All stores directly benefit from this bugfix because of
DataStoreBase. This patch fixes the following property related problems.
-It introduces a static method createProps in DataStoreFactory. This is the
equivalent of Configuration.create(). Everyone can create a new properties
object and set everything interesting on it and pass it on to whatever stores
they like, instead of ALL stores.
-It fixes the method javadoc of DataStoreBase#getSchemaName(String
mappingSchemaName, Class? persistentClass). The previous description was
simply wrong.
-It SERIALIZES the properties field of DataStoreBase instead of grabbing the
static DataStoreFactory.properties field. This has the additional benefit of
making sure that the store can be used correctly with runtime modified
properties in a mapreduce context.
-It removes the caching functionality of DataStoreFactory. Because of the
dynamic configuration in the Properties and Configuration object, it is very
difficult to implement a correct key hash for the cache. At the moment it
only uses the triple {datastoreClass, keyClass,valueClass} as a key hash.
Multiple stores cannot be properly supported when the factory uses badly
implemented hash keys. (For example, one might instantiate 2 SqlStores, both
using the exact same {datastoreClass, keyClass,valueClass} triple, but
pointing to different databases. When one is about the instantiate the second
datastore, it will faulty return the first datastore from cache). We can
always reintroduce caching functionality when we can implement a proper key.
The patch passes all tests. Will commit when there are no objections.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-104) Change to correct testWordCount() in DataStoreMapReduceTestBase
[ https://issues.apache.org/jira/browse/GORA-104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241843#comment-13241843 ] Henry Saputra commented on GORA-104: +1 to comment out the test. I dont think I have ever run this test. Change to correct testWordCount() in DataStoreMapReduceTestBase Key: GORA-104 URL: https://issues.apache.org/jira/browse/GORA-104 Project: Apache Gora Issue Type: Test Components: testing Affects Versions: 0.1.1-incubating Reporter: Lewis John McGibbney Assignee: Lewis John McGibbney Priority: Minor Fix For: 0.2 Attachments: GORA-104.patch Currently (although it is deprecated) o.a.g.maoreduce.DataStoreMapReduceTestBase#testWordCount() runs the incorrect test from o.a.g.mapreduce.MapReduceTestUtils. This requires the correct test case to be specified as well as the correct configuration parameters to be put in place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
