Log4j appender sends events as Avro only if the original event itself was in Avro. What are you trying to do?
Thanks, Hari On Wed, Apr 8, 2015 at 3:17 PM, Kaushik Shreekanth <[email protected]> wrote: > Hi, > I'm using Flume log4j Appender to send generic records from my application to > a flume avro source. However at the sink, the data is seen as plain text. I'm > not sure why it was not converted to avro format. > This is my log4j2 flume appender config: > <?xml version="1.0" encoding="UTF-8"?> > <Configuration status="WARN"> > <Appenders> > <Console name="ConsoleAppender" target="SYSTEM_OUT"> > <PatternLayout pattern="%m%n" /> > </Console> > <Flume name="FlumeAppender" compress="false" type="Avro"> > <Agent host="localhost" port="41414"/> > </Flume> > </Appenders> > <Loggers> > <Root level="debug"> > <AppenderRef ref="ConsoleAppender" /> > <AppenderRef ref="FlumeAppender" /> > </Root> > </Loggers> > </Configuration> > Dependencies in my application: > <dependency> > <groupId>org.apache.logging.log4j</groupId> > <artifactId>log4j-api</artifactId> > <version>2.2</version> > </dependency> > <dependency> > <groupId>org.apache.logging.log4j</groupId> > <artifactId>log4j-core</artifactId> > <version>2.2</version> > </dependency> > <dependency> > <groupId>org.apache.logging.log4j</groupId> > <artifactId>log4j-flume-ng</artifactId> > <version>2.2</version> > </dependency> > <dependency> > <groupId>org.apache.avro</groupId> > <artifactId>avro</artifactId> > <version>1.7.7</version> > </dependency> > <dependency> > <groupId>com.fasterxml.jackson.core</groupId> > <artifactId>jackson-databind</artifactId> > <version>2.5.1</version> > </dependency> > <dependency> > <groupId>com.fasterxml.jackson.core</groupId> > <artifactId>jackson-core</artifactId> > <version>2.5.1</version> > </dependency> > <dependency> > <groupId>org.slf4j</groupId> > <artifactId>slf4j-simple</artifactId> > <version>1.7.12</version> > </dependency> > This is my flume.conf: > # Define a memory channel called ch1 on agent1 > agent1.channels.ch1.type = memory > > # Define an Avro source called avro-source1 on agent1 and tell it > # to bind to 0.0.0.0:41414. Connect it to channel ch1. > agent1.sources.avro-source1.type = avro > agent1.sources.avro-source1.bind = 0.0.0.0 > agent1.sources.avro-source1.port = 41414 > # Define a logger sink that simply logs all events it receives > # and connect it to the other end of the same channel. > agent1.sinks.file-roll-sink.type = file_roll > agent1.sinks.file-roll-sink.sink.directory = /Users/ds/flume_events > agent1.sinks.file-roll-sink.sink.rollInterval = 0 > # Finally, now that we've defined all of our components, tell > # agent1 which ones we want to activate. > agent1.channels = ch1 > agent1.sources = avro-source1 > agent1.sinks = file-roll-sink > #chain the different components together > agent1.sinks.file-roll-sink.channel = ch1 > agent1.sources.avro-source1.channels = ch1 > My java code that logs events: > public static void main(String[] args) throws IOException { > Logger logger = LogManager.getLogger(); > URL url = FlumeTest.class.getClassLoader().getResource("user.avsc"); > Schema schema = new Schema.Parser().parse(new File(url.getFile())); > for (i = 0; i <= 10; i++) { > GenericRecord user1 = new GenericData.Record(schema); > user1.put("name", "abc" + String.valueOf(i)); > user1.put("id", i); > logger.info(user1); > } > } > My sink output is seen as: > {"name": "abc0", "id": 0, "favorite_color": null} > {"name": "abc1", "id": 1, "favorite_color": null} > {"name": "abc2", "id": 2, "favorite_color": null} > {"name": "abc3", "id": 3, "favorite_color": null} > {"name": "abc4", "id": 4, "favorite_color": null} > {"name": "abc5", "id": 5, "favorite_color": null} > "name": "abc6", "id": 6, "favorite_color": null} > {"name": "abc7", "id": 7, "favorite_color": null} > {"name": "abc8", "id": 8, "favorite_color": null} > {"name": "abc9", "id": 9, "favorite_color": null} > {"name": "abc10", "id": 10, "favorite_color": null} > As mentioned, the final output is in plain text and not Avro, can someone > please let me know what I'm doing wrong? > Thanks. >
