Michael Noll created KAFKA-1519:
-----------------------------------
Summary: Console consumer: expose configuration option to
enable/disable writing the line separator
Key: KAFKA-1519
URL: https://issues.apache.org/jira/browse/KAFKA-1519
Project: Kafka
Issue Type: Bug
Components: consumer
Affects Versions: 0.8.1.1
Reporter: Michael Noll
Assignee: Neha Narkhede
Priority: Minor
The current console consumer includes a {{DefaultMessageFormatter}}, which
exposes a few user-configurable options which can be set on the command line
via --property, e.g. "--property line.separator=XYZ".
Unfortunately, the current implementation does not allow the user to completely
disable writing any such line separator. However, this functionality would be
helpful to enable users to capture data "as is" from a Kafka topic to snapshot
file. Capturing data "as is" -- without an artificial line separator -- is
particularly nice for data in a binary format (including Avro).
*No workaround*
A potential workaround would be to pass an empty string as the property value
of "line.separator", but this doesn't work in the current implementation.
The following variants throw an "Invalid parser arguments" exception:
{code}
--property line.separator= # "nothing"
--property line.separator="" # double quotes
--property line.separator='' # single quotes
{code}
Escape tricks via {{\}} backslash don't work either.
If there actually is a workaround please let me know.
*How to fix*
We can introduce a "print.line" option to enable/disable writing
"line.separator" similar to how the code already uses "print.key" to
enable/disable writing "key.separator".
This change is trivial. To preserve backwards compatibility, the "print.line"
option would be set to true by default (unlike the "print.key" option, which
defaults to false).
*Alternatives*
Apart from modifying the built-in {{DefaultMessageFormatter}}, users could of
course implement their own custom {{MessageFormatter}}. But given that it's a)
a trivial change to the {{DefaultMessageFormatter}} and b) a nice user feature
I'd say changing the built-in {{DefaultMessageFormatter}} would be the better
approach. This way, Kafka would support writing data as-is to a file out of
the box.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
