[jira] [Commented] (GORA-266) Lucene datastore for Gora

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796626#comment-16796626
 ] 

Hudson commented on GORA-266:
-

FAILURE: Integrated in Jenkins build gora-trunk #1758 (See 
[https://builds.apache.org/job/gora-trunk/1758/])
GORA-266 Lucene datastore for Gora - lewismc (djkevincr: rev 
3d921068b225994b20a31eef703d889cdabb05b4)
* (add) gora-lucene/src/test/conf/gora.properties
* (edit) gora-jcache/pom.xml
* (add) gora-lucene/src/main/java/org/apache/gora/lucene/query/LuceneResult.java
* (edit) gora-core/src/main/java/org/apache/gora/avro/store/AvroStore.java
* (edit) 
gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java
* (add) gora-lucene/src/main/java/org/apache/gora/lucene/package-info.java
* (add) gora-lucene/src/main/java/org/apache/gora/lucene/store/LuceneStore.java
* (edit) gora-core/src/main/java/org/apache/gora/util/IOUtils.java
* (edit) gora-solr/pom.xml
* (add) gora-lucene/src/test/conf/gora-lucene-mapping.xml
* (add) gora-lucene/pom.xml
* (edit) pom.xml
* (add) gora-lucene/src/main/java/org/apache/gora/lucene/query/LuceneQuery.java
* (add) gora-lucene/src/main/java/org/apache/gora/lucene/query/package-info.java
* (add) gora-lucene/src/main/java/org/apache/gora/lucene/store/package-info.java
* (edit) 
gora-hbase/src/test/java/org/apache/gora/hbase/store/TestHBaseStore.java
* (add) 
gora-lucene/src/main/java/org/apache/gora/lucene/store/LuceneMapping.java
* (edit) 
gora-core/src/main/java/org/apache/gora/persistency/impl/PersistentBase.java
* (add) 
gora-lucene/src/test/java/org/apache/gora/lucene/store/TestLuceneStore.java
* (add) 
gora-lucene/src/test/java/org/apache/gora/lucene/store/TestLuceneStoreDriver.java
* (add) gora-lucene/src/test/java/org/apache/gora/lucene/store/package-info.java


> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16483883#comment-16483883
 ] 

ASF GitHub Bot commented on GORA-266:
-

Github user djkevincr commented on a diff in the pull request:

https://github.com/apache/gora/pull/131#discussion_r189883057
  
--- Diff: gora-core/src/main/java/org/apache/gora/util/IOUtils.java ---
@@ -660,7 +657,8 @@ public static void writeStringArray(DataOutput out, 
String[] arr)
 buffer.limit(count);
 buffers.add(buffer);
   }
-  if(count < BUFFER_SIZE) break;
+  if(count < BUFFER_SIZE)
--- End diff --

@lewismc if ( - Seems this code is not formatted properly. May be these 
were coming from the existing code. Can you please check?


> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412541#comment-16412541
 ] 

ASF GitHub Bot commented on GORA-266:
-

Github user renato2099 commented on a diff in the pull request:

https://github.com/apache/gora/pull/131#discussion_r176906710
  
--- Diff: gora-core/src/main/java/org/apache/gora/util/IOUtils.java ---
@@ -434,7 +432,7 @@ public static void writeBoolArray(DataOutput out, 
boolean[] boolArray)
 WritableUtils.writeVInt(out, boolArray.length);
 
 byte b = 0;
-int i = 0;
+int i;
--- End diff --

right, I thought that if was inside the for loop.
Sorry for the noise on this one.


> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412297#comment-16412297
 ] 

ASF GitHub Bot commented on GORA-266:
-

Github user lewismc commented on a diff in the pull request:

https://github.com/apache/gora/pull/131#discussion_r176891528
  
--- Diff: gora-core/src/main/java/org/apache/gora/util/IOUtils.java ---
@@ -434,7 +432,7 @@ public static void writeBoolArray(DataOutput out, 
boolean[] boolArray)
 WritableUtils.writeVInt(out, boolArray.length);
 
 byte b = 0;
-int i = 0;
+int i;
--- End diff --

@renato2099 as I said the increment is used in line 447


> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411560#comment-16411560
 ] 

ASF GitHub Bot commented on GORA-266:
-

Github user renato2099 commented on a diff in the pull request:

https://github.com/apache/gora/pull/131#discussion_r176771580
  
--- Diff: gora-core/src/main/java/org/apache/gora/util/IOUtils.java ---
@@ -434,7 +432,7 @@ public static void writeBoolArray(DataOutput out, 
boolean[] boolArray)
 WritableUtils.writeVInt(out, boolArray.length);
 
 byte b = 0;
-int i = 0;
+int i;
--- End diff --

yeah that is what I meant.
for(int i=0; i Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410532#comment-16410532
 ] 

ASF GitHub Bot commented on GORA-266:
-

Github user lewismc commented on a diff in the pull request:

https://github.com/apache/gora/pull/131#discussion_r176596674
  
--- Diff: gora-core/src/main/java/org/apache/gora/util/IOUtils.java ---
@@ -434,7 +432,7 @@ public static void writeBoolArray(DataOutput out, 
boolean[] boolArray)
 WritableUtils.writeVInt(out, boolArray.length);
 
 byte b = 0;
-int i = 0;
+int i;
--- End diff --

You mean an enhanced for loop? From what I can see the variable ```i``` is 
used outside of the ```for(i=0; i Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400611#comment-16400611
 ] 

ASF GitHub Bot commented on GORA-266:
-

Github user renato2099 commented on the issue:

https://github.com/apache/gora/pull/131
  
sorry @lewismc I didn't submitted, I just did now.


> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400607#comment-16400607
 ] 

ASF GitHub Bot commented on GORA-266:
-

Github user renato2099 commented on a diff in the pull request:

https://github.com/apache/gora/pull/131#discussion_r174723758
  
--- Diff: 
gora-lucene/src/main/java/org/apache/gora/lucene/store/LuceneStore.java ---
@@ -0,0 +1,475 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.lucene.store;
+
+import org.apache.avro.Schema;
+import org.apache.avro.Schema.Type;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.lucene.query.LuceneQuery;
+import org.apache.gora.lucene.query.LuceneResult;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.FileSplitPartitionQuery;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.store.impl.FileBackedDataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.apache.gora.util.OperationNotSupportedException;
+import org.apache.hadoop.conf.Configurable;
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.analysis.standard.StandardAnalyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.Field.Store;
+import org.apache.lucene.document.StoredField;
+import org.apache.lucene.document.StringField;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.IndexWriterConfig.OpenMode;
+import org.apache.lucene.index.Term;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.search.SearcherFactory;
+import org.apache.lucene.search.SearcherManager;
+import org.apache.lucene.search.TermQuery;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.store.FSDirectory;
+import org.apache.lucene.util.BytesRef;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.w3c.dom.Element;
+import org.w3c.dom.NodeList;
+
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.FileSystems;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+
+public class LuceneStore extends 
FileBackedDataStoreBase implements Configurable {
+
+  private static final Logger LOG = LoggerFactory.getLogger( 
LuceneStore.class );
+
+  private static final String DEFAULT_MAPPING_FILE = 
"gora-lucene-mapping.xml";
+  private static final String LUCENE_VERSION_KEY = 
"gora.lucene.index.version";
+  private static final String DEFAULT_LUCENE_VERSION = "LATEST";
+  private static final String LUCENE_RAM_BUFFER_KEY = 
"gora.lucene.index.writer.rambuffer";
+  private static final String DEFAULT_LUCENE_RAMBUFFER = "16";
+
+  private LuceneMapping mapping;
+  private IndexWriter writer;
+  private SearcherManager searcherManager;
+
+  @Override
+  public void initialize(Class keyClass, Class persistentClass, 
Properties properties) {
+try {
+  super.initialize(keyClass, persistentClass, properties);
+} catch (GoraException e1) {
+  e1.printStackTrace();
+}
+
+String mappingFile = null;
+try {
+  mappingFile = DataStoreFactory.getMappingFile(
+  properties, (DataStore) this, DEFAULT_MAPPING_FILE );
+} catch 

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400604#comment-16400604
 ] 

ASF GitHub Bot commented on GORA-266:
-

Github user renato2099 commented on a diff in the pull request:

https://github.com/apache/gora/pull/131#discussion_r174718491
  
--- Diff: 
gora-lucene/src/main/java/org/apache/gora/lucene/query/LuceneResult.java ---
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.lucene.query;
+
+import com.google.common.primitives.Ints;
+import org.apache.gora.lucene.store.LuceneStore;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.query.impl.ResultBase;
+import org.apache.gora.store.DataStore;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.search.SearcherManager;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+
+public class LuceneResult extends 
ResultBase {
+
+  private ScoreDoc[] scoreDocs = null;
+  private final LuceneStore store;
+  private String[] fields;
+  private int pos = 0;
+  private final SearcherManager searcherManager;
+  private IndexSearcher searcher;
+
+  public LuceneResult(DataStore dataStore, Query query,
+  SearcherManager sm) throws IOException {
+super(dataStore, query);
+
+searcherManager = sm;
+store = (LuceneStore)dataStore;
+if (query instanceof PartitionQueryImpl) {
+  query = ((PartitionQueryImpl)query).getBaseQuery();
+}
+fields = query.getFields();
+if (fields != null) {
+  HashSet uniqFields = new HashSet<>(Arrays.asList(fields));
+  String keyFld = ((LuceneStore)dataStore).getMapping().getPrimaryKey();
--- End diff --

The store variable is already retrieved on line 52. So no need to re-cast 
the object and get the mapping again.


> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400605#comment-16400605
 ] 

ASF GitHub Bot commented on GORA-266:
-

Github user renato2099 commented on a diff in the pull request:

https://github.com/apache/gora/pull/131#discussion_r174719493
  
--- Diff: 
gora-lucene/src/main/java/org/apache/gora/lucene/query/LuceneResult.java ---
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.lucene.query;
+
+import com.google.common.primitives.Ints;
+import org.apache.gora.lucene.store.LuceneStore;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.query.impl.ResultBase;
+import org.apache.gora.store.DataStore;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.search.SearcherManager;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+
+public class LuceneResult extends 
ResultBase {
+
+  private ScoreDoc[] scoreDocs = null;
+  private final LuceneStore store;
+  private String[] fields;
+  private int pos = 0;
+  private final SearcherManager searcherManager;
+  private IndexSearcher searcher;
+
+  public LuceneResult(DataStore dataStore, Query query,
+  SearcherManager sm) throws IOException {
+super(dataStore, query);
+
+searcherManager = sm;
+store = (LuceneStore)dataStore;
+if (query instanceof PartitionQueryImpl) {
+  query = ((PartitionQueryImpl)query).getBaseQuery();
+}
+fields = query.getFields();
+if (fields != null) {
+  HashSet uniqFields = new HashSet<>(Arrays.asList(fields));
+  String keyFld = ((LuceneStore)dataStore).getMapping().getPrimaryKey();
+  uniqFields.add(keyFld); // return also primary key
+  query.setFields(fields);
+}
+else {
+  Collection c = ((LuceneStore)dataStore).getMapping().getLuceneFields();
+  String[] a = {};
+  fields = c.toArray(a);
+  query.setFields(fields);
+}
+// This is based on the limits of IndexSearcher.search(Query, int)
+// A custom Collector could go larger than Integer.MAX_VALUE
+// (NB: TotalHitCountCollector uses an int internally)
+if (limit < 1L)
+  limit = Integer.MAX_VALUE;
+
+searcher = searcherManager.acquire();
+scoreDocs = searcher.search(((LuceneQuery)query).toLuceneQuery(), Ints.checkedCast(limit)).scoreDocs;
+  }
+  
+  public ScoreDoc[] getScoreDocs() {
+return scoreDocs;
+  }
+
+  @SuppressWarnings("unchecked")
+  @Override
+  protected boolean nextInner() throws IOException {
+if (scoreDocs == null || pos >= scoreDocs.length) {
+  return false;
+}
+
+Set f = null;
+if (fields != null) {
+  f = new HashSet<>(fields.length);
+  f.addAll(Arrays.asList(fields));
+  f.add(store.getMapping().getPrimaryKey());
+}
+else {
+  Collection c = ((LuceneStore)dataStore).getMapping().getLuceneFields();
--- End diff --

The same comment as before, you should just use the "store" class variable 
to avoid more casts.


> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400601#comment-16400601
 ] 

ASF GitHub Bot commented on GORA-266:
-

Github user renato2099 commented on a diff in the pull request:

https://github.com/apache/gora/pull/131#discussion_r174715638
  
--- Diff: gora-core/src/main/java/org/apache/gora/avro/store/AvroStore.java 
---
@@ -50,8 +48,7 @@
  * An adapter DataStore for binary-compatible Avro serializations.
  * AvroDataStore supports Binary and JSON serializations.
  */
-public class AvroStore
-extends FileBackedDataStoreBase implements Configurable {
+public class AvroStore extends 
FileBackedDataStoreBase implements Configurable {
--- End diff --

nitpick:
Didn't we have a format checker to not to allow lines longer than XX 
characters?


> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400603#comment-16400603
 ] 

ASF GitHub Bot commented on GORA-266:
-

Github user renato2099 commented on a diff in the pull request:

https://github.com/apache/gora/pull/131#discussion_r174718538
  
--- Diff: 
gora-lucene/src/main/java/org/apache/gora/lucene/query/LuceneResult.java ---
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.lucene.query;
+
+import com.google.common.primitives.Ints;
+import org.apache.gora.lucene.store.LuceneStore;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.query.impl.ResultBase;
+import org.apache.gora.store.DataStore;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.search.SearcherManager;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+
+public class LuceneResult extends 
ResultBase {
+
+  private ScoreDoc[] scoreDocs = null;
+  private final LuceneStore store;
+  private String[] fields;
+  private int pos = 0;
+  private final SearcherManager searcherManager;
+  private IndexSearcher searcher;
+
+  public LuceneResult(DataStore dataStore, Query query,
+  SearcherManager sm) throws IOException {
+super(dataStore, query);
+
+searcherManager = sm;
+store = (LuceneStore)dataStore;
+if (query instanceof PartitionQueryImpl) {
+  query = ((PartitionQueryImpl)query).getBaseQuery();
+}
+fields = query.getFields();
+if (fields != null) {
+  HashSet uniqFields = new HashSet<>(Arrays.asList(fields));
+  String keyFld = ((LuceneStore)dataStore).getMapping().getPrimaryKey();
+  uniqFields.add(keyFld); // return also primary key
+  query.setFields(fields);
+}
+else {
+  Collection c = ((LuceneStore)dataStore).getMapping().getLuceneFields();
--- End diff --

same here.


> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400606#comment-16400606
 ] 

ASF GitHub Bot commented on GORA-266:
-

Github user renato2099 commented on a diff in the pull request:

https://github.com/apache/gora/pull/131#discussion_r174720104
  
--- Diff: 
gora-lucene/src/main/java/org/apache/gora/lucene/store/LuceneMapping.java ---
@@ -0,0 +1,50 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.lucene.store;
+
+import java.util.Collection;
+import java.util.HashMap;
+
+public class LuceneMapping {
+  private final HashMap mapping;
+  private String primaryKey;
--- End diff --

Clarification question:
How does the primaryKey relates to the actual mapping? I mean I guess I can 
have a Gora record containing only a primary key, but that would mean that the 
mapping is actually empty, right? Or does this scenario can never happen? :)


> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400602#comment-16400602
 ] 

ASF GitHub Bot commented on GORA-266:
-

Github user renato2099 commented on a diff in the pull request:

https://github.com/apache/gora/pull/131#discussion_r174718062
  
--- Diff: 
gora-lucene/src/main/java/org/apache/gora/lucene/query/LuceneResult.java ---
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.lucene.query;
+
+import com.google.common.primitives.Ints;
+import org.apache.gora.lucene.store.LuceneStore;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.query.impl.ResultBase;
+import org.apache.gora.store.DataStore;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.search.SearcherManager;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+
+public class LuceneResult extends 
ResultBase {
+
+  private ScoreDoc[] scoreDocs = null;
+  private final LuceneStore store;
+  private String[] fields;
+  private int pos = 0;
+  private final SearcherManager searcherManager;
+  private IndexSearcher searcher;
+
+  public LuceneResult(DataStore dataStore, Query query,
--- End diff --

I am unsure how the other datastores are layed out, but if this is a 
LuceneResult, then we know that the dataStore is a LuceneStore, so it should 
take as a parameter a LuceneStore instead of a generic one.
This will avoid doing the extra cast on line 59 and others.


> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400599#comment-16400599
 ] 

ASF GitHub Bot commented on GORA-266:
-

Github user renato2099 commented on a diff in the pull request:

https://github.com/apache/gora/pull/131#discussion_r174716112
  
--- Diff: gora-core/src/main/java/org/apache/gora/util/IOUtils.java ---
@@ -434,7 +432,7 @@ public static void writeBoolArray(DataOutput out, 
boolean[] boolArray)
 WritableUtils.writeVInt(out, boolArray.length);
 
 byte b = 0;
-int i = 0;
+int i;
--- End diff --

nitpick:
why don't just make "i" a loop variable? It is not needed outside of it 
anyways


> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400600#comment-16400600
 ] 

ASF GitHub Bot commented on GORA-266:
-

Github user renato2099 commented on a diff in the pull request:

https://github.com/apache/gora/pull/131#discussion_r174717006
  
--- Diff: gora-core/src/main/java/org/apache/gora/util/IOUtils.java ---
@@ -611,8 +609,7 @@ public static void writeStringArray(DataOutput out, 
String[] arr)
 String classKey = dataKey + "._class";
 String className = conf.get(classKey);
 try {
-  T obj = (T) DefaultStringifier.load(conf, dataKey, 
ClassLoadingUtils.loadClass(className));
-  return obj;
+  return (T) DefaultStringifier.load(conf, dataKey, 
ClassLoadingUtils.loadClass(className));
 } catch (Exception ex) {
--- End diff --

This should throw a more specific exception. DefaultStringifier.load() 
throws an IOException.
And actually this should be aligned to the latest discussion with 
@alfonsonishikawa  and @nishadi about wrapping exceptions into GoraExceptions. 
What is your input on this one guys?


> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400583#comment-16400583
 ] 

ASF GitHub Bot commented on GORA-266:
-

Github user renato2099 commented on the issue:

https://github.com/apache/gora/pull/131
  
if you look at the file committed then you should see them. Although I also 
see them directly here in the PR page.


> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400131#comment-16400131
 ] 

ASF GitHub Bot commented on GORA-266:
-

Github user renato2099 commented on the issue:

https://github.com/apache/gora/pull/131
  
Hey @lewismc 
This looks awesome! Thanks a lot for doing this! I went over the code 
quickly and I left some questions and nitpicks, feel free to update whenever 
you have time and ping me back if you want another round of reviews.


> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399670#comment-16399670
 ] 

ASF GitHub Bot commented on GORA-266:
-

GitHub user lewismc opened a pull request:

https://github.com/apache/gora/pull/131

GORA-266 Lucene datastore for Gora

This PR updates the patch at https://issues.apache.org/jira/browse/GORA-266 
and also upgrades the Lucene API's to match with 6.2.1 which is currently where 
the Solr/Lucene dependencies are pegged in parent pom.xml.

Some issues with this PR

1. tests do not pass
1. UNION types are not supported
1. getPartitions is not supported
1. There are a number of inline comments which we should address

Code review welcome, thanks

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lewismc/gora GORA-266

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/gora/pull/131.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #131


commit 8426dac00b28aeb170a1f6dc7cea2cfe05e67bcc
Author: Lewis John McGibbney 
Date:   2018-03-15T00:29:54Z

GORA-266 Lucene datastore for Gora




> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[Lewis John McGibbney (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399464#comment-16399464
 ] 

Lewis John McGibbney commented on GORA-266:
---

I'm updating this patch and will submit shortly

> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>  Labels: features
> Fix For: 0.9
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[Lewis John McGibbney (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731988#comment-14731988
 ] 

Lewis John McGibbney commented on GORA-266:
---

No AFAIK it is not suitable for supporting UNION types. We need to rebase 
against trunk and work off of it again.

> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Scott Stults
>Priority: Minor
>  Labels: features
> Fix For: 0.7
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[Furkan KAMACI (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731498#comment-14731498
 ] 

Furkan KAMACI commented on GORA-266:


Is this patch ready to merge?

> Lucene datastore for Gora
> -
>
> Key: GORA-266
> URL: https://issues.apache.org/jira/browse/GORA-266
> Project: Apache Gora
>  Issue Type: New Feature
>  Components: gora-lucene
>Reporter: Lewis John McGibbney
>Assignee: Scott Stults
>Priority: Minor
>  Labels: features
> Fix For: 0.7
>
> Attachments: GORA-266.patch
>
>
> Sometimes we don't need to set up, administer, maintain and configure a 
> running solr server(s).
> Sometime we just want to write to Lucene index files and read directly from 
> them. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[Lewis John McGibbney (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062054#comment-14062054
 ] 

Lewis John McGibbney commented on GORA-266:
---

[~sstults], this patch (not unsurprisingly) doesn't compile against master 
branch. There are to problems with IOUtils.deserialize and IOUtils.serialize in 
LuceneStore. I am not understanding why I can't compile the static method call 
on the given paratemeters... but I just can't!!!
Can you possiby take a look please? I know we will also require @Ignore 
annotations for the test suite as it looks we are not supporting UNION  types 
quite yet. We will see how it goes.
Thanks in advance for any insight.

 Lucene datastore for Gora
 -

 Key: GORA-266
 URL: https://issues.apache.org/jira/browse/GORA-266
 Project: Apache Gora
  Issue Type: New Feature
  Components: gora-lucene
Reporter: Lewis John McGibbney
Assignee: Scott Stults
Priority: Minor
  Labels: features
 Fix For: 0.5

 Attachments: GORA-266.patch


 Sometimes we don't need to set up, administer, maintain and configure a 
 running solr server(s).
 Sometime we just want to write to Lucene index files and read directly from 
 them. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[Lewis John McGibbney (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14052556#comment-14052556
 ] 

Lewis John McGibbney commented on GORA-266:
---

[~sstults], I have been lokking at Elephant Bird and I thinkit is exactly what 
we need for getPartitions... once we get this patch in work can focus on that 
as I am VERY in need of using that in a Gora setting for ETL worflows.

 Lucene datastore for Gora
 -

 Key: GORA-266
 URL: https://issues.apache.org/jira/browse/GORA-266
 Project: Apache Gora
  Issue Type: New Feature
  Components: gora-lucene
Reporter: Lewis John McGibbney
Assignee: Scott Stults
Priority: Minor
  Labels: features
 Fix For: 0.5

 Attachments: GORA-266.patch


 Sometimes we don't need to set up, administer, maintain and configure a 
 running solr server(s).
 Sometime we just want to write to Lucene index files and read directly from 
 them. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [jira] [Commented] (GORA-266) Lucene datastore for Gora

[Scott Stults]

That's great! Should I re-target the patch to current trunk, or one of the
release tags?


On Fri, Jul 4, 2014 at 12:48 PM, Lewis John McGibbney (JIRA) 
j...@apache.org wrote:


 [
 https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14052556#comment-14052556
 ]

 Lewis John McGibbney commented on GORA-266:
 ---

 [~sstults], I have been lokking at Elephant Bird and I thinkit is exactly
 what we need for getPartitions... once we get this patch in work can focus
 on that as I am VERY in need of using that in a Gora setting for ETL
 worflows.

  Lucene datastore for Gora
  -
 
  Key: GORA-266
  URL: https://issues.apache.org/jira/browse/GORA-266
  Project: Apache Gora
   Issue Type: New Feature
   Components: gora-lucene
 Reporter: Lewis John McGibbney
 Assignee: Scott Stults
 Priority: Minor
   Labels: features
  Fix For: 0.5
 
  Attachments: GORA-266.patch
 
 
  Sometimes we don't need to set up, administer, maintain and configure a
 running solr server(s).
  Sometime we just want to write to Lucene index files and read directly
 from them.



 --
 This message was sent by Atlassian JIRA
 (v6.2#6252)




-- 
Scott Stults | Founder  Solutions Architect | OpenSource Connections, LLC
| 434.409.2780
http://www.opensourceconnections.com

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[Lewis John McGibbney (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049636#comment-14049636
 ] 

Lewis John McGibbney commented on GORA-266:
---

Hey [~sstults] are you interested in getting this in to Gora trunk now? If you 
would like I could assign the issue to you.
I would like to help drive this into trunk and if possible release the new 
module alongside 0.5 release.
It's entirely up to you?
Thanks

 Lucene datastore for Gora
 -

 Key: GORA-266
 URL: https://issues.apache.org/jira/browse/GORA-266
 Project: Apache Gora
  Issue Type: New Feature
  Components: gora-lucene
Reporter: Lewis John McGibbney
Priority: Minor
  Labels: features
 Fix For: 0.5

 Attachments: GORA-266.patch


 Sometimes we don't need to set up, administer, maintain and configure a 
 running solr server(s).
 Sometime we just want to write to Lucene index files and read directly from 
 them. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[Lewis John McGibbney (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14036906#comment-14036906
 ] 

Lewis John McGibbney commented on GORA-266:
---

https://github.com/kevinweil/elephant-bird/wiki/Elephant-Bird-Lucene

 Lucene datastore for Gora
 -

 Key: GORA-266
 URL: https://issues.apache.org/jira/browse/GORA-266
 Project: Apache Gora
  Issue Type: New Feature
  Components: gora-lucene
Reporter: Lewis John McGibbney
Priority: Minor
  Labels: features
 Fix For: 0.5

 Attachments: GORA-266.patch


 Sometimes we don't need to set up, administer, maintain and configure a 
 running solr server(s).
 Sometime we just want to write to Lucene index files and read directly from 
 them. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [jira] [Commented] (GORA-266) Lucene datastore for Gora

[Scott Stults]

Thank you!

Should I apply GORA-231v2.patch and make GORA-266 dependent on GORA-231?


-Scott

On Aug 18, 2013, at 8:52 AM, Apostolos Giannakidis (JIRA) j...@apache.org 
wrote:

 
[ 
 https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13743195#comment-13743195
  ] 
 
 Apostolos Giannakidis commented on GORA-266:
 
 
 Hey Scott. Great work.
 A small improvement could be a check that the mapping file exists before 
 reading it. Check out this: https://issues.apache.org/jira/browse/GORA-231
 
 Lucene datastore for Gora
 -
 
Key: GORA-266
URL: https://issues.apache.org/jira/browse/GORA-266
Project: Apache Gora
 Issue Type: New Feature
   Reporter: Lewis John McGibbney
   Priority: Minor
 Labels: features
Fix For: 0.5
 
Attachments: GORA-266.patch
 
 
 Sometimes we don't need to set up, administer, maintain and configure a 
 running solr server(s).
 Sometime we just want to write to Lucene index files and read directly from 
 them. 
 
 --
 This message is automatically generated by JIRA.
 If you think it was sent incorrectly, please contact your JIRA administrators
 For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[Apostolos Giannakidis (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13743195#comment-13743195
 ] 

Apostolos Giannakidis commented on GORA-266:


Hey Scott. Great work.
A small improvement could be a check that the mapping file exists before 
reading it. Check out this: https://issues.apache.org/jira/browse/GORA-231

 Lucene datastore for Gora
 -

 Key: GORA-266
 URL: https://issues.apache.org/jira/browse/GORA-266
 Project: Apache Gora
  Issue Type: New Feature
Reporter: Lewis John McGibbney
Priority: Minor
  Labels: features
 Fix For: 0.5

 Attachments: GORA-266.patch


 Sometimes we don't need to set up, administer, maintain and configure a 
 running solr server(s).
 Sometime we just want to write to Lucene index files and read directly from 
 them. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[Scott Stults (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730315#comment-13730315
 ] 

Scott Stults commented on GORA-266:
---

That's a good point. We'd have to basically fork just the portion of Solr that 
deals with the config files if we don't want to pull in Solr as a dependency. 
Maybe a simple properties file with the field name and an ordered list of 
tokenizers/analyzers would be better?

 Lucene datastore for Gora
 -

 Key: GORA-266
 URL: https://issues.apache.org/jira/browse/GORA-266
 Project: Apache Gora
  Issue Type: New Feature
Reporter: Lewis John McGibbney
Priority: Minor
 Fix For: 0.5


 Sometimes we don't need to set up, administer, maintain and configure a 
 running solr server(s).
 Sometime we just want to write to Lucene index files and read directly from 
 them. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[Scott Stults (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13727518#comment-13727518
 ] 

Scott Stults commented on GORA-266:
---

What's a good way to specify the tokenization and analysis to do on each of the 
fields? We could go with a stripped-down version of schema.xml, or maybe use 
Solr's full-blown schema.xml reader. The first way would keep things really 
simple at the expense of flexibility. The opposite would be true for the second 
approach and would probably require less code.

 Lucene datastore for Gora
 -

 Key: GORA-266
 URL: https://issues.apache.org/jira/browse/GORA-266
 Project: Apache Gora
  Issue Type: New Feature
Reporter: Lewis John McGibbney
Priority: Minor
 Fix For: 0.5


 Sometimes we don't need to set up, administer, maintain and configure a 
 running solr server(s).
 Sometime we just want to write to Lucene index files and read directly from 
 them. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (GORA-266) Lucene datastore for Gora

[Lewis John McGibbney (JIRA)]

[ 
https://issues.apache.org/jira/browse/GORA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13727768#comment-13727768
 ] 

Lewis John McGibbney commented on GORA-266:
---

I need to be honest and say that I am really not sure right now Scott!
Although I need a module like this (pretty desperately) I am going to work on 
the Avro upgrade first as a priority. I marked this for 0.5 because of that. 
This Avro stuff is like a thorn in my side :0)

 Lucene datastore for Gora
 -

 Key: GORA-266
 URL: https://issues.apache.org/jira/browse/GORA-266
 Project: Apache Gora
  Issue Type: New Feature
Reporter: Lewis John McGibbney
Priority: Minor
 Fix For: 0.5


 Sometimes we don't need to set up, administer, maintain and configure a 
 running solr server(s).
 Sometime we just want to write to Lucene index files and read directly from 
 them. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira