[GitHub] gora pull request #135: Goraexplorer needed changes
GitHub user alfonsonishikawa opened a pull request: https://github.com/apache/gora/pull/135 Goraexplorer needed changes This set of commits implements the following functionalities: - Compile schemas from Strings and not only from files. - Allow to configure the mapping of a HBase backed datastore when creating the datastore - Added a new factory method to create a datastore using class' names and not depending on generics - Updated Persistent to get the related schema. - Added #clone in Persistent - Enabled the possibility to retrieve information from the native backend behind a datastore. This information is used in the [schema's wizard of GoraExplorer](https://bitbucket.org/alfonsonishikawa/goraexplorer/wiki/User%20manual#markdown-header-handling-schemas-and-mappings-with-the-wizard). - Added gora-pig module to access Gora from Pig. [More details on gora-pig](http://people.apache.org/~alfonsonishikawa/gora-109.html) Please be picky on this pull request. Any doubt, anything, ask for it. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alfonsonishikawa/gora goraexplorer Alternatively you can review and apply these changes as the patch at: https://github.com/apache/gora/pull/135.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #135 commit 6949760dcda2be83a3c408502d89dfdc962f3764 Author: Alfonso Nishikawa Muñumer Date: 2016-07-25T17:17:58Z Creación de la rama "goraexplorer" con los .pom modificados para que hagan referencia al repositorio maven.nishilua.com donde se aloja 0.7-goraexplorer-SNAPSHOT que soporta las funcionalidades de goraexplorer, como por ejemplo: - Crear un DataStore con un Configuration y Properties sin necesidad de disponer de un DataStore en tiempo de compilación -malditos generics- commit d607fdfe37c3167fb997708811effc5c94f54af3 Author: Alfonso Nishikawa Muñumer Date: 2016-07-28T14:33:07Z - Compile a schema from a String, and not only from file. - Allow to get the mapping from Configuration (key gora.mapping) as a String, and not only from file. - Instantiate a datastore given de datastore class, the key class and the persistent class by name, not relying on generics. commit b9f9f56bd29ce2fefc57acae76ad31c2f075840e Author: Alfonso Nishikawa Muñumer Date: 2016-09-19T17:29:58Z Modified Persistent to return it's schema. commit 4410c415cc322244046a2234350c8d73f8649757 Author: Alfonso Nishikawa Muñumer Date: 2016-09-22T21:12:23Z Merge branch 'master' into goraexplorer commit f3aed481b3d73f655f61e1dd35641f5a03de1415 Author: Alfonso Nishikawa Muñumer Date: 2017-02-06T20:34:54Z Merge branch 'master' into goraexplorer commit e6eab433cf5584bac1f418dbc0caa5427408a209 Author: Alfonso Nishikawa Muñumer Date: 2017-05-21T19:04:50Z BUGFIX: HBaseStore endKey is exclusive (since HBase handles it exclusive). We add a trail zero to the end key when searching to make it inclusive to conform to Gora's quey interface. commit 3b35c36fa6855512fe3a228b82caf32952271203 Author: Alfonso Nishikawa Muñumer Date: 2017-07-14T21:58:17Z Actualizando el proyecto a todos los cambios de la release 0.7 commit 055d5b48cc618b41b2a2d41d95e83fcfa232d838 Author: Alfonso Nishikawa Muñumer Date: 2017-07-17T23:52:20Z Added #clone() method to Persistent commit b58722b587a29c8d8d069ceaafe4c2f63f6cc47e Author: Alfonso Nishikawa Muñumer Date: 2017-07-18T00:08:43Z Updated pom.xmls to deploy commit 43ddc91f9e0b77cbefdda850b7aadf3261505e04 Author: Alfonso Nishikawa Muñumer Date: 2017-09-22T21:12:25Z Implemented DataStoreMetadataAnalyzer to retrieve native backend information. - Implemented only for HBase at this moment. commit e3496145d2ee1d147bcf2af176b99c0ad12d43b9 Author: Alfonso Nishikawa Muñumer Date: 2017-09-26T09:50:45Z Added missing #close() method to DataStoreMetadataAnalyzer commit 0ae4ec44259ea0e3feec3e76671420438547fdae Author: Alfonso Nishikawa Muñumer Date: 2017-09-26T11:27:56Z Updated DataStoreMetadataFactory to throw ClassNotFoundException not hiding this exceptional case. commit bf1165fac897d7d51fd6645a0eb8dfdd4eb4165c Author: Alfonso Nishikawa Muñumer Date: 2017-12-07T12:38:45Z Updated Persistent with @JsonIgnore to allow jackson serialization commit 6dc22695d51eb464df4bb27765b1af18ca757b9e Author: Alfonso Nishikawa Muñumer Date: 2017-12-20T21:06:10Z Added a constructor to BeanFactoryImpl with String parameter commit 5e5e43dad805433c533948f63599ed27706ed6cb Author: Alfonso Nishikawa Muñumer Date: 2018-04-03T17:38:58Z Merged with Gora 0.9 - breaking code, will fix in next commits commit 6d40006563f96442c039b9541355e3c18390ddab Author: Alfonso Nishikawa Muñumer Date:
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user alfonsonishikawa commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222505739 --- Diff: gora-compiler/src/main/java/org/apache/gora/compiler/GoraCompiler.java --- @@ -86,6 +88,31 @@ public static void compileSchema(File[] srcFiles, File dest, LicenseHeaders lice } } + /** --- End diff -- Schema compilation from a String (instead of a file), with a configurable templatesPath. I actually used it as `GoraCompiler.compileSchema(avroSchemaDefinition, outputDirectory, null)` ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user alfonsonishikawa commented on a diff in the pull request:
https://github.com/apache/gora/pull/135#discussion_r222508076
--- Diff:
gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java ---
@@ -80,15 +81,20 @@
* DataStore for HBase. Thread safe.
*
*/
-public class HBaseStore extends
DataStoreBase
-implements Configurable {
+public class HBaseStore extends
DataStoreBase {
public static final Logger LOG =
LoggerFactory.getLogger(HBaseStore.class);
public static final String PARSE_MAPPING_FILE_KEY =
"gora.hbase.mapping.file";
public static final String DEFAULT_MAPPING_FILE =
"gora-hbase-mapping.xml";
+ /**
+ * Key at DataStore Properties (same as gora.properties) to hold a
mapping loaded from memory, instead from filesystem like
+ * the one at PARSE_MAPPING_FILE_KEY. If this key is present, the
mapping is loaded from the value instead of gora-hbase-mapping.xml
+ */
+ public static final String XML_MAPPING_DEFINITION = "gora.mapping" ;
--- End diff --
When creating the DataStore (for HBase!) the Properties can have a key
`gora.mapping` with the XML document of the mapping. If it exists, this mapping
is loaded instead of the file one at `gora-hbase-mapping.xml`.
---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user alfonsonishikawa commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222505121 --- Diff: bin/compile-examples.sh --- @@ -36,8 +36,57 @@ DIR=$GORA_HOME/$MODULE/src/examples/avro/ OUTDIR=$GORA_HOME/$MODULE/src/examples/java GORA_BIN=$GORA_HOME/bin/gora +rm -r $OUTDIR/org/apache/gora/examples/generated/* --- End diff -- I found that the compile-examples script was vastly outdated. I updated it, but still needs to have compiled the module 'gora-compiler-cli', so, running the first time must do: ``` mvn package // wait for compile crash after gora-compiler-cli module bin/compile-examples.sh mvn package ``` ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user alfonsonishikawa commented on a diff in the pull request:
https://github.com/apache/gora/pull/135#discussion_r222506186
--- Diff:
gora-compiler/src/main/velocity/org/apache/gora/compiler/templates/record.vm ---
@@ -180,6 +180,11 @@ public class ${this.mangle($schema.getName())}#if
($schema.isError()) extends or
return new #if
($schema.getNamespace())$schema.getNamespace().#end${this.mangle($schema.getName())}.Builder(other);
}
+ @Override
--- End diff --
A clone method. I used it
[here](https://bitbucket.org/alfonsonishikawa/goraexplorer/src/12f226408e7f9686aa6b5975692edf270b94182e/src/main/java/com/nishilua/goraexplorer/tables/TableEntityServiceImpl.java#lines-118)
because `get()` reuses the instance.
---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user alfonsonishikawa commented on a diff in the pull request:
https://github.com/apache/gora/pull/135#discussion_r222506447
--- Diff:
gora-core/src/main/java/org/apache/gora/mapreduce/GoraRecordWriter.java ---
@@ -76,4 +76,26 @@ public void write(K key, T value) throws IOException,
InterruptedException {
throw new RuntimeException(e);
}
}
+
--- End diff --
Added this method to delete by key. Used in gora-pig to delete a set of
elements.
---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user alfonsonishikawa commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222508755 --- Diff: pom.xml --- @@ -757,9 +758,9 @@ gora-maven-plugin gora-mongodb gora-solr +gora-aerospike --- End diff -- gora-aerospike must be before gora-tutorial and sources-dist to be able to compile successfully for the first time. Dependency issue fixed. ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user alfonsonishikawa commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222507887 --- Diff: gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java --- @@ -80,15 +81,20 @@ * DataStore for HBase. Thread safe. * */ -public class HBaseStore extends DataStoreBase --- End diff -- I don't remember why this Configurable was deleted. Is it needed? ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user alfonsonishikawa commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222510709 --- Diff: gora-pig/src/test/java/org/apache/gora/pig/GoraStorageTest.java-disabled --- @@ -0,0 +1,352 @@ +package org.apache.gora.pig; --- End diff -- Gora-pig does not have JUnit tests! I have been struggling to launch a Hadoop+HBase cluster. It seems with Hadoop 1 + HBase 1.0.x (or alike) I got it working in Hadoop 1 with an old Gora version: JUnit started both pseudo distributed and did the job. But with HBase 1.2.6 `utility.startMiniCluster();` and `utility.startMiniMapReduceCluster();` tries to launch a Hadoop 1 cluster. And it crashes too. At the end, I developed and tested gora-pig directly with HDP 2.6. **A hand is needed to make this run** :\ Thank you. ---
[jira] [Commented] (GORA-535) Add a data store for Apache Ignite
[ https://issues.apache.org/jira/browse/GORA-535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637825#comment-16637825 ] ASF GitHub Bot commented on GORA-535: - Github user lewismc commented on the issue: https://github.com/apache/gora/pull/134 @djkevincr @carlosrmng is this ready to merge? > Add a data store for Apache Ignite > --- > > Key: GORA-535 > URL: https://issues.apache.org/jira/browse/GORA-535 > Project: Apache Gora > Issue Type: New Feature >Reporter: Nishadi Kirielle >Priority: Major > Labels: gsoc2018 > > Currently, Gora has support for persisting objects to various database models > such as Apache Hbase, Apache Cassandra and much more. [1] This project aims > to extend its capability to provide support for Apache Ignite database. > Apache Ignite is a distributed database, caching and processing platform.[2] > [1]. [http://gora.apache.org/] > [2] . [https://ignite.apache.org/] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] gora issue #134: GORA-535 Add a data store for Apache Ignite
Github user lewismc commented on the issue: https://github.com/apache/gora/pull/134 @djkevincr @carlosrmng is this ready to merge? ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222541246 --- Diff: gora-core/src/examples/java/org/apache/gora/examples/generated/package-info.java --- @@ -1,20 +0,0 @@ -/** --- End diff -- Why remove? ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222542987 --- Diff: gora-pig/src/main/java/org/apache/gora/pig/StorageConfiguration.java --- @@ -0,0 +1,140 @@ +package org.apache.gora.pig; --- End diff -- License header. ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222542461 --- Diff: gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStoreMetadataAnalyzer.java --- @@ -0,0 +1,75 @@ +package org.apache.gora.hbase.store; --- End diff -- License header please ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222542089 --- Diff: gora-goraci/src/main/java/org/apache/gora/goraci/generated/package-info.java --- @@ -1,21 +0,0 @@ -/** --- End diff -- Why remove? ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222541419 --- Diff: gora-core/src/main/java/org/apache/gora/persistency/Persistent.java --- @@ -113,4 +118,13 @@ * @return a new instance of the object */ Persistent newInstance(); + + /** + * Returns the avro's data schema + * @return the parsed schema definition + */ + public Schema getSchema(); + + public Persistent clone() throws CloneNotSupportedException; --- End diff -- Can you provide Javadoc for this? It is a primary method in a very important interface. ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222543383 --- Diff: gora-pig/src/main/java/org/apache/gora/pig/util/PersistentUtils.java --- @@ -0,0 +1,147 @@ +package org.apache.gora.pig.util; --- End diff -- License header. ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222543248 --- Diff: gora-pig/src/main/java/org/apache/gora/pig/udf/MapDeleteElements.java --- @@ -0,0 +1,67 @@ +package org.apache.gora.pig.udf; --- End diff -- License header. ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222542231 --- Diff: gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java --- @@ -80,15 +81,20 @@ * DataStore for HBase. Thread safe. * */ -public class HBaseStore extends DataStoreBase --- End diff -- It wasn't just for HBaseStore? ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222543177 --- Diff: gora-pig/src/main/java/org/apache/gora/pig/mapreduce/PigGoraOutputFormat.java --- @@ -0,0 +1,86 @@ +package org.apache.gora.pig.mapreduce; --- End diff -- License header. ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222543621 --- Diff: gora-pig/src/test/resources/gora.properties --- @@ -0,0 +1,4 @@ +gora.datastore.default=org.apache.gora.hbase.store.HBaseStore --- End diff -- License header. ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222543455 --- Diff: gora-pig/src/test/examples/README.txt --- @@ -0,0 +1,9 @@ + --- End diff -- Can you make this an .md file? ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222542930 --- Diff: gora-pig/src/main/java/org/apache/gora/pig/ResourceFieldSchemaWithIndex.java --- @@ -0,0 +1,31 @@ +package org.apache.gora.pig; --- End diff -- License header. ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222540948 --- Diff: gora-compiler/src/main/java/org/apache/gora/compiler/GoraCompiler.java --- @@ -86,6 +88,31 @@ public static void compileSchema(File[] srcFiles, File dest, LicenseHeaders lice } } + /** --- End diff -- I like this ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/135#discussion_r222541609
--- Diff:
gora-core/src/main/java/org/apache/gora/store/DataStoreFactory.java ---
@@ -251,6 +254,70 @@ D getDataStore( Class dataStoreClass, Class
keyClass,
}
}
+ /**
+ * Instantiate a new {@link DataStore}. Uses default properties. Uses
'null' schema.
+ *
+ * @param dataStoreClass The datastore implementation class as
string.
--- End diff --
Not required this is not a parameter
---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/135#discussion_r222543584
--- Diff: gora-pig/src/test/examples/example.pig ---
@@ -0,0 +1,4 @@
+set job.name 'GoraStorage test' ;
+register gora/*.jar ;
+webpage = LOAD '.' using
org.apache.gora.pig.GoraStorage('java.lang.String','admin.WebPage','baseUrl,status,content')
;
+dump webpage;
--- End diff --
We cannot store webpage.jar in the Gora source... we need some way of
building this via compile examples maybe?
---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222543479 --- Diff: gora-pig/src/test/examples/example.pig --- @@ -0,0 +1,4 @@ +set job.name 'GoraStorage test' ; --- End diff -- License header if possible. ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/135#discussion_r222543835
--- Diff: pom.xml ---
@@ -1546,6 +1553,40 @@
${orientqb.version}
+
+
+ org.apache.pig
+ pig
+ h2
+ ${pig.version}
+
+
+ org.apache.avro
+ avro
+
+
+
+
+
+ dk.brics.automaton
+ automaton
+ ${automaton.version}
+ test
+
+
+
+ org.antlr
+ antlr-runtime
+ ${antlr.version}
+ test
+
+
+
+ com.linkedin.datafu
--- End diff --
Can you not use org.apache.datafu artifact?
---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222543728 --- Diff: pom.xml --- @@ -832,6 +833,12 @@ 1.0.1 2.5 + +0.16.0 +1.11-8 --- End diff -- Is this required? What do you use it for? ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/135#discussion_r222542415
--- Diff:
gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java ---
@@ -734,14 +756,13 @@ private void setField(T persistent, Field field, List
list) {
}
@SuppressWarnings("unchecked")
- private HBaseMapping readMapping(String filename) throws IOException {
+ private HBaseMapping readMapping(InputStream mappingStream) throws
IOException {
--- End diff --
I like this. The quicker work with ByteStream the better.
---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/135#discussion_r222543239
--- Diff:
gora-pig/src/main/java/org/apache/gora/pig/mapreduce/PigGoraOutputFormat.java
---
@@ -0,0 +1,86 @@
+package org.apache.gora.pig.mapreduce;
+
+import java.io.IOException;
+
+import org.apache.gora.mapreduce.GoraRecordWriter;
+import org.apache.gora.mapreduce.NullOutputCommitter;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.pig.StorageConfiguration;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.hadoop.conf.Configurable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapreduce.JobContext;
+import org.apache.hadoop.mapreduce.OutputCommitter;
+import org.apache.hadoop.mapreduce.OutputFormat;
+import org.apache.hadoop.mapreduce.RecordWriter;
+import org.apache.hadoop.mapreduce.TaskAttemptContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class PigGoraOutputFormat extends
OutputFormat implements Configurable {
+
+ public static final Logger LOG =
LoggerFactory.getLogger(PigGoraOutputFormat.class);
+
+ protected Configuration conf ;
+
+ /**
+ * The GoraStorage configuration setted at constructor (converted from
json to bean)
+ */
+ protected StorageConfiguration storageConfiguration ;
+
+ protected DataStore dataStore ;
+
+ @Override
+ @SuppressWarnings({ "unchecked", "rawtypes" })
+ public RecordWriter getRecordWriter(TaskAttemptContext context)
+ throws IOException, InterruptedException {
+
+if ( this.dataStore == null ) {
+ this.dataStore = DataStoreFactory.getDataStore(
+ this.storageConfiguration.getKeyClass(),
+ this.storageConfiguration.getPersistentClass(),
+ this.storageConfiguration.getGoraPropertiesAsProperties(),
+ this.conf
+ ) ;
+}
+
+return new GoraRecordWriter(this.dataStore, context);
+ }
+
+
+ @Override
+ public void checkOutputSpecs(JobContext context) throws IOException,
InterruptedException {
+// Nothing
--- End diff --
throw unsupported exception?
---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/135#discussion_r222542669
--- Diff: gora-pig/pom.xml ---
@@ -0,0 +1,180 @@
+
+http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+xmlns="http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+4.0.0
+
+
+org.apache.gora
+gora
+0.9-SNAPSHOT
+../
+
+
+gora-pig
+bundle
+
+Apache Gora :: Pig
+http://gora.apache.org
+Pig -> Gora -> DataStore connector.
+2018
+
+The Apache Software Foundation
+http://www.apache.org/
+
+
+
+*
+
org.apache.gora.pig*;version="${project.version}";-noimport:=true
+
+
+
+target
+target/classes
+${project.artifactId}-${project.version}
+target/test-classes
+src/test/java
+src/main/java
+
+
+
+org.apache.maven.plugins
+maven-jar-plugin
+
+
+
+test-jar
+
+
+
+
+
+
+
+
+
+org.slf4j
+jcl-over-slf4j
+
+
+
+
+org.apache.gora
+gora-core
+compile
+
+
+
--- End diff --
Just remove commented out dependencies.
---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/135#discussion_r222541705
--- Diff:
gora-core/src/main/java/org/apache/gora/store/DataStoreFactory.java ---
@@ -412,7 +484,7 @@ public static String getMappingFile(Properties
properties, DataStore store
return mappingFilename;
}
- private static String getDefaultDataStore(Properties properties) {
+ public static String getDefaultDataStore(Properties properties) {
--- End diff --
Why the change here?
---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/135#discussion_r222541077
--- Diff:
gora-compiler/src/main/velocity/org/apache/gora/compiler/templates/record.vm ---
@@ -180,6 +180,11 @@ public class ${this.mangle($schema.getName())}#if
($schema.isError()) extends or
return new #if
($schema.getNamespace())$schema.getNamespace().#end${this.mangle($schema.getName())}.Builder(other);
}
+ @Override
--- End diff --
You should have referenced line 121 not 118.
---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222543267 --- Diff: gora-pig/src/main/java/org/apache/gora/pig/udf/MapMerge.java --- @@ -0,0 +1,33 @@ +package org.apache.gora.pig.udf; --- End diff -- License header. ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/135#discussion_r222541967
--- Diff:
gora-core/src/main/java/org/apache/gora/store/DataStoreMetadataFactory.java ---
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.store;
+
+import java.util.Properties;
+
+import org.apache.gora.store.impl.DataStoreMetadataAnalyzer;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.ReflectionUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Factory to create Metadata Analyzers for the backends.
+ * It uses the properties from gora.properties, but can be configured with
other properties.
+ *
+ * The returned Metadata Analyzer defaults to *StoreMetadataAnalyzer, but
can be configured. By default detects the backend storage
+ * taking it from the Gora Properties (which defaults to gora.properties).
+ */
+public class DataStoreMetadataFactory {
+
+ public static final Logger log =
LoggerFactory.getLogger(DataStoreMetadataFactory.class);
+
+ private DataStoreMetadataFactory() {
+ }
+
+ /**
+ * Creates a metadata analyzer given the Hadoop Configuration instance.
Uses the default gora.properties where it takes the
+ * base name from the default store to infer the metadata analyzer to
create.
+ * @throws GoraException
+ * @throws ClassNotFoundException - Exception thrown when does not
exists a suitable metadata analyzer class
+ */
+ public static DataStoreMetadataAnalyzer createAnalyzer(Configuration
configuration) throws GoraException, ClassNotFoundException {
+return createAnalyzer(configuration, DataStoreFactory.createProps());
+ }
+
+ /**
+ * Creates a metadata analyzer given the Hadoop Configuration instance
and the gora properties to use. It uses this properties
+ * to infer the metadata analyzer to create.
+ * @throws GoraException
+ * @throws ClassNotFoundException - Exception thrown when does not
exists a suitable metadata analyzer class
+ */
+ public static DataStoreMetadataAnalyzer createAnalyzer(Configuration
configuration, Properties properties) throws GoraException,
ClassNotFoundException {
+String metadataAnalyzerClassName =
DataStoreFactory.getDefaultDataStore(properties) + "MetadataAnalyzer";
--- End diff --
Aha, I now see why you want this to be made public.
---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222542624 --- Diff: gora-pig/pom.xml --- @@ -0,0 +1,180 @@ + --- End diff -- License header. ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222540708 --- Diff: bin/compile-examples.sh --- @@ -36,8 +36,57 @@ DIR=$GORA_HOME/$MODULE/src/examples/avro/ OUTDIR=$GORA_HOME/$MODULE/src/examples/java GORA_BIN=$GORA_HOME/bin/gora +rm -r $OUTDIR/org/apache/gora/examples/generated/* --- End diff -- Yep OK ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222543677 --- Diff: pom.xml --- @@ -757,9 +758,9 @@ gora-maven-plugin gora-mongodb gora-solr +gora-aerospike --- End diff -- Good catch. ---
[GitHub] gora pull request #135: Goraexplorer needed changes
Github user lewismc commented on a diff in the pull request: https://github.com/apache/gora/pull/135#discussion_r222542504 --- Diff: gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseTableMetadata.java --- @@ -0,0 +1,27 @@ +package org.apache.gora.hbase.store; --- End diff -- License header please. ---
