Re: Should MongoDB know about Utf8?

[Damien Raude-Morvan]

Hi all!

2014-10-09 20:44 GMT+02:00 Lewis John Mcgibbney <lewis.mcgibb...@gmail.com>:
> Hi Folks,
[..]
> The cause is the following line (example from FetcherJob):
> filter.getOperands().add(new Utf8(batchId));
>
> During serialization MongoDB doesn't know about Utf8 type.
> My question is now... should we remove the Utf8 from Nutch, or should
> gora-mongodb handle this silently?

I've just added a comment on https://issues.apache.org/jira/browse/NUTCH-1843
As stated there, issue arises in gora-mongo filters handling and not
really from object-to-datastore mapping already handle Utf8 nicely
I was able to complete a full generate-fetch-parse Nutch cycle using
the following patch.

I'm not complety satisfied with this patch because :
* since method signature is List<Object> rawOperands, caller can send
any datatype which might lead to other types of crash
* we have to implement this transformation in each driver (which seems
error prone)
* I'm not sure of use-case of Utf8 as Operand (ie. I'm only using
String type here)

Regards,
-- 
Damien

diff --git 
a/gora-mongodb/src/main/java/org/apache/gora/mongodb/filters/DefaultFactory.java
 
b/gora-mongodb/src/main/java/org/apache/gora/mongodb/filters/DefaultFactory.java
index 54cbdfd..7c5ad95 100644
--- 
a/gora-mongodb/src/main/java/org/apache/gora/mongodb/filters/DefaultFactory.java
+++ 
b/gora-mongodb/src/main/java/org/apache/gora/mongodb/filters/DefaultFactory.java
@@ -115,7 +115,8 @@ public class DefaultFactory<K, T extends PersistentBase> 
extends
   }
 
   protected QueryBuilder appendToBuilder(final QueryBuilder builder,
-      final FilterOp filterOp, final List<Object> operands) {
+      final FilterOp filterOp, final List<Object> rawOperands) {
+    List<String> operands = convertOperandsToString(rawOperands);
     switch (filterOp) {
     case EQUALS:
       if (operands.size() == 1) {
@@ -150,4 +151,14 @@ public class DefaultFactory<K, T extends PersistentBase> 
extends
     return builder;
   }
 
+  private List<String> convertOperandsToString(List<Object> rawOperands) {
+    List<String> operands = new ArrayList<String>(rawOperands.size());
+    for (Object rawOperand : rawOperands) {
+      if (rawOperand != null) {
+        operands.add(rawOperand.toString());
+      }
+    }
+    return operands;
+  }
+
 }