[jira] [Commented] (FLUME-2781) A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used by a Flume source
[
https://issues.apache.org/jira/browse/FLUME-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220676#comment-15220676
]
Roshan Naik commented on FLUME-2781:
Thanks for confirming [~gherreros]
I got a confused when i saw this stmt in some of the automated comments here
in this jira.
{quote}"Kafka Channel with parseAsFlumeEvent=true should write data as is, not
as flume events." .{quote}
which i believe is exactly the opposite of what is intended.
> A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used
> by a Flume source
> -
>
> Key: FLUME-2781
> URL: https://issues.apache.org/jira/browse/FLUME-2781
> Project: Flume
> Issue Type: Improvement
>Affects Versions: v1.6.0
>Reporter: Gonzalo Herreros
>Assignee: Gonzalo Herreros
> Labels: easyfix, patch
> Fix For: v1.7.0
>
> Attachments: FLUME-2781.patch
>
>
> When a Kafka channel is configured as parseAsFlumeEvent=false, the channel
> will read events from the topic as text instead of serialized Avro Flume
> events.
> This is useful so Flume can read from an existing Kafka topic, where other
> Kafka clients publish as text.
> However, if you use a Flume source on that channel, it will still write the
> events as Avro so it will create an inconsistency and those events will fail
> to be read correctly.
> Also, this would allow a Flume source to write to a Kafka channel and any
> Kafka subscriber to listen to Flume events passing through without binary
> dependencies.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2781) A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used by a Flume source
[ https://issues.apache.org/jira/browse/FLUME-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219538#comment-15219538 ] Gonzalo Herreros commented on FLUME-2781: - That's correct. You can use KafkaConsoleConsumer to verify the difference. With the flag set to true (default), each line will have a couple of strange characters before each line (it's avro) while when set to false you only get the event body as it is. > A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used > by a Flume source > - > > Key: FLUME-2781 > URL: https://issues.apache.org/jira/browse/FLUME-2781 > Project: Flume > Issue Type: Improvement >Affects Versions: v1.6.0 >Reporter: Gonzalo Herreros >Assignee: Gonzalo Herreros > Labels: easyfix, patch > Fix For: v1.7.0 > > Attachments: FLUME-2781.patch > > > When a Kafka channel is configured as parseAsFlumeEvent=false, the channel > will read events from the topic as text instead of serialized Avro Flume > events. > This is useful so Flume can read from an existing Kafka topic, where other > Kafka clients publish as text. > However, if you use a Flume source on that channel, it will still write the > events as Avro so it will create an inconsistency and those events will fail > to be read correctly. > Also, this would allow a Flume source to write to a Kafka channel and any > Kafka subscriber to listen to Flume events passing through without binary > dependencies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2781) A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used by a Flume source
[ https://issues.apache.org/jira/browse/FLUME-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219177#comment-15219177 ] Roshan Naik commented on FLUME-2781: [~gherreros] this is a very useful feature! Don't see any note in the UserGuide about this... so needed clarifcation... Setting parseAsFlumeEvent=*false* will cause events to be written as-is into the Kafka topic without the FlumeEvent wrapper right ? > A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used > by a Flume source > - > > Key: FLUME-2781 > URL: https://issues.apache.org/jira/browse/FLUME-2781 > Project: Flume > Issue Type: Improvement >Affects Versions: v1.6.0 >Reporter: Gonzalo Herreros >Assignee: Gonzalo Herreros > Labels: easyfix, patch > Fix For: v1.7.0 > > Attachments: FLUME-2781.patch > > > When a Kafka channel is configured as parseAsFlumeEvent=false, the channel > will read events from the topic as text instead of serialized Avro Flume > events. > This is useful so Flume can read from an existing Kafka topic, where other > Kafka clients publish as text. > However, if you use a Flume source on that channel, it will still write the > events as Avro so it will create an inconsistency and those events will fail > to be read correctly. > Also, this would allow a Flume source to write to a Kafka channel and any > Kafka subscriber to listen to Flume events passing through without binary > dependencies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2781) A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used by a Flume source
[ https://issues.apache.org/jira/browse/FLUME-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949805#comment-14949805 ] Hudson commented on FLUME-2781: --- UNSTABLE: Integrated in Flume-trunk-hbase-1 #129 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/129/]) FLUME-2781. Kafka Channel with parseAsFlumeEvent=true should write data (hshreedharan: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=67ed62aa18df3675b68369d0d00c8f0dcbdfb970]) * flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java * flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java > A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used > by a Flume source > - > > Key: FLUME-2781 > URL: https://issues.apache.org/jira/browse/FLUME-2781 > Project: Flume > Issue Type: Improvement >Affects Versions: v1.6.0 >Reporter: Gonzalo Herreros >Assignee: Gonzalo Herreros > Labels: easyfix, patch > Attachments: FLUME-2781.patch > > > When a Kafka channel is configured as parseAsFlumeEvent=false, the channel > will read events from the topic as text instead of serialized Avro Flume > events. > This is useful so Flume can read from an existing Kafka topic, where other > Kafka clients publish as text. > However, if you use a Flume source on that channel, it will still write the > events as Avro so it will create an inconsistency and those events will fail > to be read correctly. > Also, this would allow a Flume source to write to a Kafka channel and any > Kafka subscriber to listen to Flume events passing through without binary > dependencies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2781) A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used by a Flume source
[ https://issues.apache.org/jira/browse/FLUME-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949733#comment-14949733 ] ASF subversion and git services commented on FLUME-2781: Commit 67ed62aa18df3675b68369d0d00c8f0dcbdfb970 in flume's branch refs/heads/trunk from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=67ed62a ] FLUME-2781. Kafka Channel with parseAsFlumeEvent=true should write data as is, not as flume events. (Gonzalo Herreros via Hari) > A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used > by a Flume source > - > > Key: FLUME-2781 > URL: https://issues.apache.org/jira/browse/FLUME-2781 > Project: Flume > Issue Type: Improvement >Affects Versions: v1.6.0 >Reporter: Gonzalo Herreros >Assignee: Gonzalo Herreros > Labels: easyfix, patch > Attachments: FLUME-2781.patch > > > When a Kafka channel is configured as parseAsFlumeEvent=false, the channel > will read events from the topic as text instead of serialized Avro Flume > events. > This is useful so Flume can read from an existing Kafka topic, where other > Kafka clients publish as text. > However, if you use a Flume source on that channel, it will still write the > events as Avro so it will create an inconsistency and those events will fail > to be read correctly. > Also, this would allow a Flume source to write to a Kafka channel and any > Kafka subscriber to listen to Flume events passing through without binary > dependencies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2781) A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used by a Flume source
[ https://issues.apache.org/jira/browse/FLUME-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949729#comment-14949729 ] ASF subversion and git services commented on FLUME-2781: Commit faad35801b24b9f0ca34d8b86f28dded468d73b8 in flume's branch refs/heads/flume-1.7 from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=faad358 ] FLUME-2781. Kafka Channel with parseAsFlumeEvent=true should write data as is, not as flume events. (Gonzalo Herreros via Hari) > A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used > by a Flume source > - > > Key: FLUME-2781 > URL: https://issues.apache.org/jira/browse/FLUME-2781 > Project: Flume > Issue Type: Improvement >Affects Versions: v1.6.0 >Reporter: Gonzalo Herreros > Labels: easyfix, patch > Attachments: FLUME-2781.patch > > > When a Kafka channel is configured as parseAsFlumeEvent=false, the channel > will read events from the topic as text instead of serialized Avro Flume > events. > This is useful so Flume can read from an existing Kafka topic, where other > Kafka clients publish as text. > However, if you use a Flume source on that channel, it will still write the > events as Avro so it will create an inconsistency and those events will fail > to be read correctly. > Also, this would allow a Flume source to write to a Kafka channel and any > Kafka subscriber to listen to Flume events passing through without binary > dependencies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2781) A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used by a Flume source
[ https://issues.apache.org/jira/browse/FLUME-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949671#comment-14949671 ] Hari Shreedharan commented on FLUME-2781: - +1. LGTM. I am running the tests now. Will commit once the tests pass. > A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used > by a Flume source > - > > Key: FLUME-2781 > URL: https://issues.apache.org/jira/browse/FLUME-2781 > Project: Flume > Issue Type: Improvement >Affects Versions: v1.6.0 >Reporter: Gonzalo Herreros > Labels: easyfix, patch > Attachments: FLUME-2781.patch > > > When a Kafka channel is configured as parseAsFlumeEvent=false, the channel > will read events from the topic as text instead of serialized Avro Flume > events. > This is useful so Flume can read from an existing Kafka topic, where other > Kafka clients publish as text. > However, if you use a Flume source on that channel, it will still write the > events as Avro so it will create an inconsistency and those events will fail > to be read correctly. > Also, this would allow a Flume source to write to a Kafka channel and any > Kafka subscriber to listen to Flume events passing through without binary > dependencies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2781) A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used by a Flume source
[
https://issues.apache.org/jira/browse/FLUME-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735095#comment-14735095
]
Hari Shreedharan commented on FLUME-2781:
-
This looks good. There a few formatting issues - we use 2 space indents
everywhere - the patch uses a mix of 2 and 4 spaces. This code:
{code}
if (parseAsFlumeEvent){
{code}
should be:
{code}
if (parseAsFlumeEvent) {
{code}
> A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used
> by a Flume source
> -
>
> Key: FLUME-2781
> URL: https://issues.apache.org/jira/browse/FLUME-2781
> Project: Flume
> Issue Type: Improvement
>Affects Versions: 1.6
>Reporter: Gonzalo Herreros
> Attachments: FLUME-2781.patch
>
>
> When a Kafka channel is configured as parseAsFlumeEvent=false, the channel
> will read events from the topic as text instead of serialized Avro Flume
> events.
> This is useful so Flume can read from an existing Kafka topic, where other
> Kafka clients publish as text.
> However, if you use a Flume source on that channel, it will still write the
> events as Avro so it will create an inconsistency and those events will fail
> to be read correctly.
> Also, this would allow a Flume source to write to a Kafka channel and any
> Kafka subscriber to listen to Flume events passing through without binary
> dependencies.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2781) A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used by a Flume source
[ https://issues.apache.org/jira/browse/FLUME-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14725365#comment-14725365 ] ASF GitHub Bot commented on FLUME-2781: --- GitHub user gherreros opened a pull request: https://github.com/apache/flume/pull/24 FLUME-2781 proposed implementation Implemented a tiny change so putting to the channel is consistent with take, so if parseAsFlumeEvent is false it uses the event body text instead of an Avro Flume event object. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gherreros/flume trunk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/24.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #24 commit 2c02cfa484460056ec4d040dc2ec568e19fa916b Author: gherreros Date: 2015-09-01T13:07:55Z FLUME-2781 When the Kafka channel is configured as parseAsFlumeEvent=false, any writes to the topic are done as plain text (so the events can be read correctly) > A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used > by a Flume source > - > > Key: FLUME-2781 > URL: https://issues.apache.org/jira/browse/FLUME-2781 > Project: Flume > Issue Type: Improvement >Reporter: Gonzalo Herreros > > When a Kafka channel is configured as parseAsFlumeEvent=false, the channel > will read events from the topic as text instead of serialized Avro Flume > events. > This is useful so Flume can read from an existing Kafka topic, where other > Kafka clients publish as text. > However, if you use a Flume source on that channel, it will still write the > events as Avro so it will create an inconsistency and those events will fail > to be read correctly. > Also, this would allow a Flume source to write to a Kafka channel and any > Kafka subscriber to listen to Flume events passing through without binary > dependencies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
