M, Thank you for your suggestion I kept just one file to index. It is very small file with 3 lines and 3 characters on each line I see the content in flume log file and an entry in Solr but not sure why it is not going to Solr
-----Original Message----- From: Markus Jelsma [mailto:markus.jel...@openindex.io] Sent: Tuesday, February 07, 2017 6:04 PM To: solr-user@lucene.apache.org Subject: RE: Flume SolrSink No Error but not Indexing Hi - are the channels filling up? If not, events can be silently dropped by custom interceptors, but you don 't seem to have any. We're not using that sink you have, it might also silenently drop events. At least check you channel size, it goes up in most cases, meaning all your events are still stored and ready to be shipped to the sink. M. -----Original message----- > From:Anatharaman, Srinatha (Contractor) > <srinatha_ananthara...@comcast.com> > Sent: Tuesday 7th February 2017 22:07 > To: solr-user@lucene.apache.org > Subject: Flume SolrSink No Error but not Indexing > > Hi, > > > I am indexing text document using Flume, I do not see any error or > warning message but data is not getting ingested to Solr Log level for > both Solr and Flume is set to TRACE, ALL > > Flume version : 1.5.2.2.3 > Solr Version : 5.5 > Config files are as below > Flume Config : > agent.sources = SpoolDirSrc > agent.channels = FileChannel > agent.sinks = SolrSink > > # Configure Source > agent.sources.SpoolDirSrc.channels = fileChannel > agent.sources.SpoolDirSrc.type = spooldir > agent.sources.SpoolDirSrc.spoolDir = /home/flume/source_emails > agent.sources.SpoolDirSrc.basenameHeader = true > agent.sources.SpoolDirSrc.fileHeader = true > #agent.sources.src1.fileSuffix = .COMPLETED > agent.sources.SpoolDirSrc.deserializer = > org.apache.flume.sink.solr.morphline.BlobDeserializer$Builder > # Use a channel that buffers events in memory > agent.channels.FileChannel.type = file > agent.channels.FileChannel.capacity = 10000 > #agent.channels.FileChannel.transactionCapacity = 10000 # Configure > Solr Sink agent.sinks.SolrSink.type = > org.apache.flume.sink.solr.morphline.MorphlineSolrSink > agent.sinks.SolrSink.morphlineFile = /etc/flume/conf/morphline.conf > agent.sinks.SolrSink.batchsize = 1000 > agent.sinks.SolrSink.batchDurationMillis = 2500 > agent.sinks.SolrSink.channel = fileChannel > agent.sinks.SolrSink.morphlineId = morphline1 > agent.sources.SpoolDirSrc.channels = FileChannel > agent.sinks.SolrSink.channel = FileChannel > > Morphline Config > solrLocator: { > collection : gsearch > #zkHost : "127.0.0.1:9983" > zkHost : "codesolr-as-r3p:21810,codesolr-as-r3p:21811,codesolr-as-r3p:21812" > } > morphlines : > [ > { > id : morphline1 > importCommands : ["org.kitesdk.**", "org.apache.solr.**"] > commands : > [ > { detectMimeType { includeDefaultMimeTypes : true } } > { > solrCell { > solrLocator : ${solrLocator} > captureAttr : true > lowernames : true > capture : [_attachment_body, _attachment_mimetype, basename, > content, content_encoding, content_type, file, meta] > parsers : [ { parser : org.apache.tika.parser.txt.TXTParser } ] > } > } > { generateUUID { field : id } } > { sanitizeUnknownSolrFields { solrLocator : ${solrLocator} } } > { logDebug { format : "output record: {}", args : ["@{}"] } } > { loadSolr: { solrLocator : ${solrLocator} } } > ] > } > ] > > Please help me what could be the issue Regards, ~Sri > >
