Hi, We are using an Elasticsearch sink and have seen a file channel filling with messages that will never be delivered as the format of the message is incompatible with Elasticsearch itself.
Example message from Flume logs: 24 Jan 2014 08:14:55,173 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver event. Exception follows. org.elasticsearch.indices.InvalidIndexNameException: [UpperCase-2014-01-23] Invalid index name [UpperCase-2014-01-23], must be lowercase In this case the index name comes from a header so we have a workaround using a multiplexing channel selector to detect and re-route messages based on headers of this format. To clean up the channel this time we removed the data and checkpoint directories, which is not ideal as we probably lost other messages in doing this. We are wary of similar situations occurring in future for messages that we can't detect and divert in advance and so have a few questions: - What would be the recommended handling of this situation? - Is it possible to clear just these messages from the channel or does the whole channel have to be deleted ? - Is there a way that we can divert these messages to another channel (dead letter / invalid message style) ? Noting that they are not known to be problematic until after an attempt is made to deliver them from the sink - What happens to other messages in a batch with a bad message ? Will they also be stuck forever or will they be taken in another batch ? Thanks, Paul. ---------------------------- http://www.bbc.co.uk This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated. If you have received it in error, please delete it from your system. Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately. Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this. ---------------------
