Re: Issue with delta import
Yes it does work in that case. And one more thing is, I'm use my database as cassandra not sql. Probably, that must be the problem. -- View this message in context: http://lucene.472066.n3.nabble.com/Issue-with-delta-import-tp4347680p4350901.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Issue with delta import
It says here https://cwiki.apache.org/confluence/display/solr/Parallel+SQL+Interface#ParallelSQLInterface-SolrSQLSyntax that you need to escape the FROM. Did you try it without the 'false' != 'false' OR On 2017-08-16 04:07 AM, bhargava ravali koganti wrote: I'm getting error like this Exception while processing: test document : SolrInputDocument(fields: []):org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: SELECT * FROM person WHERE 'false' != 'false' OR last_modified > '2017-08-09 15:48:02' Processing Document # 1 at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:318) at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:279) at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:54) at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59) at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73) at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:244) at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475) at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414) at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480) at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461) Caused by: com.datastax.driver.core.exceptions.SyntaxError: line 1:27 no viable alternative at input 'false' (SELECT * FROM person WHERE ['fals]...) at com.datastax.driver.core.exceptions.SyntaxError.copy(SyntaxError.java:58) at
Re: Issue with delta import
I'm getting error like this Exception while processing: test document : SolrInputDocument(fields: []):org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: SELECT * FROM person WHERE 'false' != 'false' OR last_modified > '2017-08-09 15:48:02' Processing Document # 1 at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:318) at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:279) at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:54) at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59) at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73) at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:244) at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475) at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414) at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480) at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461) Caused by: com.datastax.driver.core.exceptions.SyntaxError: line 1:27 no viable alternative at input 'false' (SELECT * FROM person WHERE ['fals]...) at com.datastax.driver.core.exceptions.SyntaxError.copy(SyntaxError.java:58) at com.datastax.driver.core.exceptions.SyntaxError.copy(SyntaxError.java:24) at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37) at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:68) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:43) at com.dbschema.CassandraPreparedStatement.executeQuery(CassandraPreparedStatement.java:73) at com.dbschema.CassandraPreparedStatement.execute(CassandraPreparedStatement.java:114) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.executeStatement(JdbcDataSource.java:338) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:313) ... 12 more Caused by: com.datastax.driver.core.exceptions.SyntaxError: line 1:27 no viable alternative at input 'false' (SELECT * FROM person WHERE ['fals]...) at com.datastax.driver.core.Responses$Error.asException(Responses.java:144) at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:179) at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:186) at com.datastax.driver.core.RequestHandler.access$2500(RequestHandler.java:50) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:817) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:651) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1077) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1000) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83) at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:157) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254) at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83) at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:157) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83) at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153) at
Re: Issue with delta import
Does this way even handle the deletes? -- View this message in context: http://lucene.472066.n3.nabble.com/Issue-with-delta-import-tp4347680p4350726.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Issue with delta import
yes. -- View this message in context: http://lucene.472066.n3.nabble.com/Issue-with-delta-import-tp4347680p4350734.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Issue with delta import
refer this : http://lucene.472066.n3.nabble.com/Number-of-requests-spike-up-when-i-do-the-delta-Import-td4338162.html#a4339168 -- View this message in context: http://lucene.472066.n3.nabble.com/Issue-with-delta-import-tp4347680p4350157.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Issue with delta import
Tried it had no impact. -- View this message in context: http://lucene.472066.n3.nabble.com/Issue-with-delta-import-tp4347680p4349577.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Issue with delta import
can you please try ${dih.last_index_time} instead of
${dataimporter.last_index_time}.
On Wed, Jul 26, 2017 at 2:33 PM, bhargava ravali koganti <
ravali@gmail.com> wrote:
> Hi,
>
> I'm trying to integrate Solr and Cassandra. I"m facing problem with delta
> import. For every 10 minutes I'm running deltaquery using cron job. If any
> changes in the data based on last index time, it has to fetch the data(as
> far as my knowledge), however, it keeps fetching the whole data
> irrespective of changes.
>
> My problem:
> https://stackoverflow.com/questions/45304803/deltaimport-fetches-all-the-
> data
>
> Looking forward to hear from you.
>
> Thanks,
> Bhargava Ravali Koganti
>
--
Thanks,
Sujay P Bawaskar
M:+91-77091 53669
Re: Issue with delta import (not finding data in a column)
Hello,
I was doing some more testing but I could not find a definitive reason for
this behavior. The following is my transformer:
public MapString, Object transformRow(MapString, Object row, Context
context) {
ListMapString, String fields = context.getAllEntityFields();
for (MapString, String field : fields)
{
// Check if this field has blob=true specified in the
data-config.xml
String blob = field.get(blob);
if (true.equals(blob))
{
String columnName = field.get(column);
// Get the field's value from the current row
Blob data = (Blob) row.get(columnName);
// Transform the blob and store back into the same column
if (data != null) {
row.put(columnName, process(data));
}
else
{
log.error(Blob is null.);
}
}
}
return row;
}
Note: The function process is the function that actually takes care of the
whole transformation.
What I noticed is that the row variable only has the ID, probably due to
this:
deltaQuery=select ID from TABLE1 where (LASTMODIFIED
to_date('${dataimporter.last_index_time}', '-mm-dd HH24:MI:SS'))
However, even if I change it to a select * statement, I get everything
except the column that contains the blob (it is returned as null).
Something tells me that the data-config may be incorrect. I cannot explain
how this works for full-imports and not delta-imports.
I hope that I explained this issue properly. I am really stuck on this. Any
help would be highly appreciated.
--
ahammad wrote:
I have a Solr core that retrieves data from an Oracle DB. The DB table has
a few columns, one of which is a Blob that represents a PDF document. In
order to retrieve the actual content of the PDF file, I wrote a Blob
transformer that converts the Blob into the PDF file, and subsequently
reads it using PDFBox. The blob is contained in a DB column called
DOCUMENT, and the data goes into a Solr field called fileContent, which is
required.
This works fine when doing full imports, but it fails for delta imports. I
debugged my transformer, and it appears that when it attempts to fetch the
blob stored in the column, it gets nothing back (i.e. null). Because the
data is essentially null, it cannot retrieve anything, and cannot store
anything into Solr. As a result, the document does not get imported. I am
not sure what the problem is, because this only occurs with delta imports.
Here is my data-config file:
dataConfig
dataSource driver=oracle.jdbc.driver.OracleDriver url=address
user=user password=pass/
document name=table1
entity name=TABLE1 pk=ID query=select * from TABLE1
deltaImportQuery=select * from TABLE1 where ID
='${dataimporter.delta.ID}'
deltaQuery=select ID from TABLE1 where (LASTMODIFIED
to_date('${dataimporter.last_index_time}', '-mm-dd HH24:MI:SS'))
transformer=BlobTransformer
field column=ID name=id /
field column=TITLE name=title /
field column=FILENAME name=filename /
field column=DOCUMENT name=fileContent
blob=true/
field column=LASTMODIFIED
name=lastModified /
/entity
/document
/dataConfig
Thanks.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Issue-with-delta-import-not-finding-data-in-a-column-tp788993p812511.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Issue with delta import (not finding data in a column)
Are u reusing the context object? It may help if u can paste the relevant
part of ur code
On 10 May 2010 19:03, ahammad ahmed.ham...@gmail.com wrote:
I have a Solr core that retrieves data from an Oracle DB. The DB table has a
few columns, one of which is a Blob that represents a PDF document. In order
to retrieve the actual content of the PDF file, I wrote a Blob transformer
that converts the Blob into the PDF file, and subsequently reads it using
PDFBox. The blob is contained in a DB column called DOCUMENT, and the data
goes into a Solr field called fileContent, which is required.
This works fine when doing full imports, but it fails for delta imports. I
debugged my transformer, and it appears that when it attempts to fetch the
blob stored in the column, it gets nothing back (i.e. null). Because the
data is essentially null, it cannot retrieve anything, and cannot store
anything into Solr. As a result, the document does not get imported. I am
not sure what the problem is, because this only occurs with delta imports.
Here is my data-config file:
dataConfig
dataSource driver=oracle.jdbc.driver.OracleDriver url=address
user=user password=pass/
document name=table1
entity name=TABLE1 pk=ID query=select * from TABLE1
deltaImportQuery=select * from TABLE1 where ID
='${dataimporter.delta.ID}'
deltaQuery=select ID from TABLE1 where (LASTMODIFIED
to_date('${dataimporter.last_index_time}', '-mm-dd HH24:MI:SS'))
transformer=BlobTransformer
field column=ID name=id /
field column=TITLE name=title /
field column=FILENAME name=filename /
field column=DOCUMENT name=fileContent
blob=true/
field column=LASTMODIFIED
name=lastModified /
/entity
/document
/dataConfig
Thanks.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Issue-with-delta-import-not-finding-data-in-a-column-tp788993p788993.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Issue with delta import (not finding data in a column)
Hello, I am not reusing the context object. The remaining part of the code takes in a Blob object, converts it to a FileInputStream, and reads the contents using PDFBox. It does not deal with anything related to Solr. The Transformer doesn't even execute the remaining part of the code. It doesn't get that far. Let me know if you need any more information. Thanks -- View this message in context: http://lucene.472066.n3.nabble.com/Issue-with-delta-import-not-finding-data-in-a-column-tp788993p812818.html Sent from the Solr - User mailing list archive at Nabble.com.
