Hi Julio,
I've fixed the bug, can you please replace the exiting
TemplateTransformer.java in the SOLR-469.patch and use the attached
TemplateTransformer.java file. We'll add the changes to our next
patch. Sorry for all the trouble.
On Sat, May 31, 2008 at 10:31 PM, Noble Paul നോബിള് नोब्ळ्
<[EMAIL PROTECTED]> wrote:
> julio,
> Looks like it is a bug.
> We can give u a new TemplateTransformer.java which we will incorporate
> in the next patch
> --Noble
>
> On Sat, May 31, 2008 at 12:24 AM, Julio Castillo
> <[EMAIL PROTECTED]> wrote:
>> I'm sorry Shalin, but I still get the same Null Pointer exception. This is
>> my complete dataconfig.xml (I remove the parallel entity to narrow down the
>> scope of the problem).
>> <dataConfig>
>> <document name="doc-1">
>> <entity name="vets" pk="id"
>> query="select id as idAlias,first_name,last_name FROM vets"
>> deltaQuery="SELECT id as idAlias FROM vets WHERE last_modified >
>> '${dataimporter.last_index_time}'"
>> transformer="TemplateTransformer">
>> <field column="id" name="id"
>> template="vets-${vets.idAlias}"/>
>> <field column="first_name" name="userName"/>
>> <field column="last_name" name="userName"/>
>> </entity>
>> </document>
>> </dataConfig>
>>
>> Thanks again.
>>
>> ** julio
>>
>> -----Original Message-----
>> From: Shalin Shekhar Mangar [mailto:[EMAIL PROTECTED]
>> Sent: Friday, May 30, 2008 11:38 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: How to describe 2 entities in dataConfig for the DataImporter?
>>
>> The surname is used just as an example of a field.
>>
>> The NullPointerException is because the same field "id" tries to use it's
>> own value in a template. The template cannot contain the same field on which
>> it is being applied. I'd suggest that you get the id aliased to another
>> name, for example using a query "select id as idAlias from vets" and then
>> use:
>> <field column="id" template="vets-${vets.idAlias}" />
>>
>> That should work, let me know if you face a problem.
>>
>> On Fri, May 30, 2008 at 10:40 PM, Julio Castillo <[EMAIL PROTECTED]>
>> wrote:
>>> Thanks for all the leads.
>>> I did get however a null pointer exception while implementing it:
>>>
>>> May 30, 2008 9:57:50 AM
>>> org.apache.solr.handler.dataimport.EntityProcessorBase
>>> applyTransformer
>>> WARNING: transformer threw error
>>> java.lang.NullPointerException
>>> at
>>> org.apache.solr.handler.dataimport.TemplateTransformer.transformRow(Te
>>> mplate
>>> Transformer.java:55)
>>> at
>>> org.apache.solr.handler.dataimport.EntityProcessorBase.applyTransforme
>>> r(Enti
>>> tyProcessorBase.java:186)
>>>
>>> Looking at the source code, it appears that the resolverMap is null.
>>> The resolver returned null given the entityName.
>>>
>>> Looking at the documentation, there is the reference to a eparent.surname.
>>> The example says:
>>> <field column="namedesc" template="hello${e.name},${eparent.surname}"
>>> />
>>>
>>> I'm afraid, I don't know what an eparent.surname is. This is my
>>> current dataconfig.xml configuration excerpt:
>>>
>>> <entity name="vets" pk="id" query="..." deltaQuery="..."
>>> transformer="TemplateTransformer">
>>> <field column="id" name="id" template="vets-${vets.id}"/> ...
>>>
>>> Am I missing a surname? Whatever that may be?
>>>
>>> Thanks
>>>
>>> ** julio
>>>
>>> -----Original Message-----
>>> From: Noble Paul ??????? ?????? [mailto:[EMAIL PROTECTED]
>>> Sent: Thursday, May 29, 2008 11:10 PM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: How to describe 2 entities in dataConfig for the
>> DataImporter?
>>>
>>> Sorry I forgot to mention that.
>>> http://wiki.apache.org/solr/DataImportHandler#head-a6916b30b5d7605a990
>>> fb03c4
>>> ff461b3736496a9
>>> --Noble
>>>
>>> On Fri, May 30, 2008 at 11:37 AM, Shalin Shekhar Mangar
>>> <[EMAIL PROTECTED]> wrote:
>>>> You need to enable TemplateTransformer for your entity. For example:
>>>> <entity name="owners" pk="id" query="...."
>>>> transformer="TemplateTransformer">
>>>>
>>>> On Fri, May 30, 2008 at 11:31 AM, Julio Castillo
>>>> <[EMAIL PROTECTED]> wrote:
>>>>> Noble,
>>>>> I tried the template setting for the "id" field, but I didn't notice
>>>>> any different behavior. I also didn't see where this would be reflected.
>>>>> I looked at the fields and the debug output for the dataImporter and
>>>>> couldn't see any reference to a modified id name (per the template
>>>>> instructions).
>>>>>
>>>>> The behavior in the end seemed to be the same. Did I miss anything?
>>>>> I assume that the <uniqueKey>id</uniqueKey> setting in the
>>>>> schema.xml remains the same?
>>>>>
>>>>> Thanks again
>>>>>
>>>>> ** julio
>>>>>
>>>>> -----Original Message-----
>>>>> From: Noble Paul ??????? ?????? [mailto:[EMAIL PROTECTED]
>>>>> Sent: Thursday, May 29, 2008 9:46 PM
>>>>> To: solr-user@lucene.apache.org
>>>>> Subject: Re: How to describe 2 entities in dataConfig for the
>>> DataImporter?
>>>>>
>>>>> Consider constructing the id concatenating an extra string for each
>>>>> document . You can construct that field using the TeplateTransformer.
>>>>> in the entity owners keep the id as
>>>>>
>>>>> <field column="id" name="id" template="owners-${owners.id}"/> and in
>>>>> vets <field column="id" name="id" template="vets-${vets.id}"/>
>>>>>
>>>>> or anything else which can make it unique
>>>>>
>>>>> --Noble
>>>>>
>>>>> On Fri, May 30, 2008 at 10:05 AM, Shalin Shekhar Mangar
>>>>> <[EMAIL PROTECTED]> wrote:
>>>>>> That will happen only if id is the uniqueKey in Solr and the id
>>>>>> coming from both your tables have same values. In that case, they
>>>>>> will overwrite each other. You will need a separate uniqueKey (on
>>>>>> other than id field).
>>>>>>
>>>>>> On Fri, May 30, 2008 at 6:34 AM, Julio Castillo
>>>>>> <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>>> Thanks Shalin,
>>>>>>> I tried putting everything under the same document (two different
>>>>>>> unrelated entities), and got a bit further.
>>>>>>>
>>>>>>> My problem now appears to be both of them stepping on each other
>>>>>>> due to
>>>>> "id"
>>>>>>> conflicts. Currently my id is defined in my schema as <field
>>>>>>> name="id" type="long" indexed="true" stored="true"
>>>>>>> required="true"/>
>>>>>>>
>>>>>>> Do I have to create a new "id" field?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> ** julio
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Shalin Shekhar Mangar [mailto:[EMAIL PROTECTED]
>>>>>>> Sent: Thursday, May 29, 2008 11:40 AM
>>>>>>> To: solr-user@lucene.apache.org
>>>>>>> Subject: Re: How to describe 2 entities in dataConfig for the
>>>>> DataImporter?
>>>>>>>
>>>>>>> Hi Julio,
>>>>>>>
>>>>>>> The first data-config is correct.
>>>>>>>
>>>>>>> You're running DataImportHandler in debug mode which creates only
>>>>>>> the first 10 documents by default. You can also add count=N to
>>>>>>> index only the first N documents. But this is intended only for
>>>>>>> debugging purposes. If you want to do a full-import just use
>>>>>>> /dataimport?command=full-import. By default, a full-import
>>>>>>> automatically
>>>>> cleans and commits.
>>>>>>>
>>>>>>> Hope that helps.
>>>>>>>
>>>>>>> On Thu, May 29, 2008 at 11:42 PM, Julio Castillo
>>>>>>> <[EMAIL PROTECTED]>
>>>>>>> wrote:
>>>>>>>> I have 2 dB tables unrelated to each other that I want to index.
>>>>>>>>
>>>>>>>> I have tried 2 approaches for specifying them in my
>>>>>>>> data-config.xml
>>>>> file.
>>>>>>>> None of them seem to work (it seems I can only get data for the
>>>>>>>> first one listed).
>>>>>>>>
>>>>>>>> CASE 1)
>>>>>>>> <dataConfig>
>>>>>>>> <document name="test">
>>>>>>>> <entity name="owners" pk="id" query="....">
>>>>>>>> <field column="id" name="id"/>
>>>>>>>> <field column=name" name="userName"/>
>>>>>>>> </entity>
>>>>>>>> <entity name="vets" pk="id" query="....">
>>>>>>>> <field column="id" name="id"/>
>>>>>>>> <field column=name" name="userName"/>
>>>>>>>> </entity>
>>>>>>>> </document>
>>>>>>>> </dataConfig>
>>>>>>>>
>>>>>>>> CASE 2)
>>>>>>>> <dataConfig>
>>>>>>>> <document name="test-1">
>>>>>>>> <entity name="owners" pk="id" query="....">
>>>>>>>> <field column="id" name="id"/>
>>>>>>>> <field column=name" name="userName"/>
>>>>>>>> </entity>
>>>>>>>> </document>
>>>>>>>> <document name="test-2">
>>>>>>>> <entity name="vets" pk="id" query="....">
>>>>>>>> <field column="id" name="id"/>
>>>>>>>> <field column=name" name="userName"/>
>>>>>>>> </entity>
>>>>>>>> </document>
>>>>>>>> </dataConfig>
>>>>>>>>
>>>>>>>> I ran the dataImporter as follows
>>>>>>>> http://localhost:2455/solr/dataimport?command=full-import&debug=o
>>>>>>>> n
>>>>>>>> &v
>>>>>>>> er
>>>>>>>> bose=t
>>>>>>>> rue&clean=true&commit=true
>>>>>>>>
>>>>>>>> After running Case 1 the output only shows documents under
>>>>> entity:owners.
>>>>>>>> And an empty reference to entity:vets (no data).
>>>>>>>>
>>>>>>>> After running Case 2, the output only shows documents under
>>>>> entity:owners.
>>>>>>>> No references to entity:vets.
>>>>>>>>
>>>>>>>> In either case, I have commented out the entity:owners and I do
>>>>>>>> see the entity:vets data then.
>>>>>>>>
>>>>>>>> Thanks for your assistance
>>>>>>>>
>>>>>>>> Julio Castillo
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Shalin Shekhar Mangar.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Shalin Shekhar Mangar.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --Noble Paul
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>>
>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>>
>
>
>
> --
> --Noble Paul
>
--
Regards,
Shalin Shekhar Mangar.
/**
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.solr.handler.dataimport;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.logging.Logger;
/**
* <p>
* A Transformer which can put values into a column by resolving an expression
* containing other columns
* </p>
*
* <p>
* For example:<br />
* <field column="name" template="${e.lastName}, ${e.firstName}
* ${e.middleName}" /> will produce the name by combining values from
* lastName, firstName and middleName fields as given in the template
* attribute.
* </p>
*
* @author Shalin Shekhar Mangar ([EMAIL PROTECTED])
* @since solr 1.3
*/
public class TemplateTransformer implements Transformer {
private static final Logger LOG = Logger.getLogger(TemplateTransformer.class.getName());
@SuppressWarnings("unchecked")
public Object transformRow(Map<String, Object> row, Context context) {
String entityName = context.getEntityAttribute(DataImporter.NAME);
VariableResolverImpl resolver = (VariableResolverImpl) context.getVariableResolver();
Map<String, Object> resolverMap = (Map<String, Object>)resolver.resolve(entityName);
if (resolverMap == null) resolverMap = new HashMap<String, Object>();
//Clone resolver map
Map<String, Object> resolverMapCopy = new HashMap<String, Object>();
//Add current row to the copy of resolver map
for (Map.Entry<String, Object> entry : row.entrySet())
resolverMapCopy.put(entry.getKey(), entry.getValue());
//Add this copy to the namespace of the current entity in the resolver
resolver.addNamespace(entityName, resolverMapCopy);
for (Map<String, String> map : context.getAllEntityFields()) {
String expr = map.get(TEMPLATE);
if (expr == null)
continue;
String column = map.get(DataImporter.COLUMN);
//Verify if all variables can be resolved or not
boolean resolvable = true;
List<String> variables = TemplateString.getVariables(expr);
for (String v : variables) {
if (resolver.resolve(v) == null) {
LOG.warning("Unable to resolve variable: " + v + " while parsing expression: " + expr);
resolvable = false;
}
}
if (!resolvable)
continue;
row.put(column, resolver.replaceTokens(expr));
}
//Restore the original resolver map
resolver.addNamespace(entityName, resolverMap);
return row;
}
public static final String TEMPLATE = "template";
}
