RE: [bulk]: Dates and DataImportHandler

[Mannott, Birgit]

Hi,

I have a similar problem. I try to change the timezone for the last_index_time 
by setting 

        <propertyWriter dateFormat="yyyy-MM-dd HH:mm:ss" 
type="SimplePropertiesWriter" locale="en_US" />

in the <dataConfig> section of my data-config.xml file.

But when doing this I always get a NullPointerException on Delta Import:

2017-09-15 14:04:00.825 INFO  (Thread-2938) [   x:mex_prd_dev1100-ap] 
o.a.s.h.d.DataImporter Starting Delta Import
2017-09-15 14:04:00.827 ERROR (Thread-2938) [   x:mex_prd_dev1100-ap] 
o.a.s.h.d.DataImporter Delta Import Failed
org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to 
PropertyWriter implementation:SimplePropertiesWriter
            at 
org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImporter.java:330)
            at 
org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:439)
            at 
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:476)
            at 
org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:457)
            at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
            at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:598)
            at 
org.apache.solr.handler.dataimport.SimplePropertiesWriter.init(SimplePropertiesWriter.java:100)
            at 
org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImporter.java:328)
            ... 4 more

Has anyone an idea what is wrong or missing?

Thanks,
Birgit



-----Original Message-----
From: Jamie Jackson [mailto:jamieja...@gmail.com] 
Sent: Tuesday, September 19, 2017 3:42 AM
To: solr-user@lucene.apache.org
Subject: [bulk]: Dates and DataImportHandler

Hi folks,

My DB server is on America/Chicago time. Solr (on Docker) is running on UTC. 
Dates coming from my (MariaDB) data source seem to get translated properly into 
the Solr index without me doing anything special.

However when doing delta imports using last_index_time ( 
http://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport ), I can't 
seem to get the date, which Solr provides, to be understood by the DB as being 
UTC (and translated back, accordingly). In other words, the DB thinks the Solr 
UTC date is local, so it thinks the date is ahead by six hours.

'${dataimporter.request.clean}' != 'false'

or dt > '${dataimporter.last_index_time}'

I came up with this workaround, which seems to work:

'${dataimporter.request.clean}' != 'false'

/* ${user.timezone} is UTC, and the ${custom.dataimporter.datasource.tz}
property is set to America/Chicago */

or dt > CONVERT_TZ('${dataimporter.last_index_time}','${user.timezone}','${
custom.dataimporter.datasource.tz}')

However, isn't there a way for this translation to happen more naturally?

I thought maybe I could do something like this:

<propertyWriter

dateFormat="yyyy-MM-dd HH:mm:ssZ"

type="SimplePropertiesWriter"

/>

The above did set the property as expected (with a trailiing `+0000`), but that 
didn't seem to help the DB understand/translate the date.

Thanks,
Jamie