Oh, not quite as easy as that. I hit the Hadoop dependency in the FILE_CHANNEL, as mentioned back in November (http://mail-archives.apache.org/mod_mbox/flume-user/201211.mbox/%3CE3E225C9E0CD4AC28A90C4E463D717E1%40cloudera.com%3E)
Now I need to work out how to resolve that on a Win7 platform. -- Thom DeCarlo [email protected] (o) 703-983-2817 (m) 757-880-6559 -----Original Message----- From: DeCarlo, Thom [mailto:[email protected]] Sent: Thursday, December 06, 2012 3:48 PM To: [email protected] Subject: RE: AVRO_EVENT problem Thanks for the info. I will skip the jdbc channel and stick with the file channel. -- Thom DeCarlo -----Original Message----- From: Brock Noland [mailto:[email protected]] Sent: Thursday, December 06, 2012 3:42 PM To: [email protected] Subject: Re: AVRO_EVENT problem Hi, I have created https://issues.apache.org/jira/browse/FLUME-1771 for this issue. Regarding JDBCChannel. Personally I would use FileChannel. It should be just as reliable and it's deployed at scale in a large variety of use cases. Brock On Thu, Dec 6, 2012 at 2:37 PM, DeCarlo, Thom <[email protected]> wrote: > Interesting. It looks like the JDBC Channel causes problems for the > file_roller+avro_event_serializer sink. Everything works right after > switching that sink to use the Memory channel. Also, the logger sink is > perfectly happy to use the jdbc channel. > > After thinking about this some more (and looking at the file output using the > avro_event serializer), maybe it doesn't make sense to use that serializer > the way I've attempted. Unless the file is read later by an AVRO reader the > file won't be useful because it contains non-printing characters, which > probably mean something to AVRO, but not much at all to vi. > > I'm going to try using the Netcat source->JDBC channel->AVRO sink, which will > send to an AVRO source->JDBC channel->file_roller sink and see if the jdbc > will work in there. (My target system is going to need high reliability > message passing and the jdbc channel sounds like it is the most robust of the > available channels.) > -- > Thom DeCarlo > > > -----Original Message----- > From: Brock Noland [mailto:[email protected]] > Sent: Thursday, December 06, 2012 2:52 PM > To: [email protected] > Subject: Re: AVRO_EVENT problem > > OK, I don't really understand how Avro is working here, but I think > you should try FileChannel or maybe MemoryChannel for simplicity to > see if that works. IE, I think the problem is JDBCChannel. Can you let > me know how it turns out? > > On Thu, Dec 6, 2012 at 1:45 PM, Brock Noland <[email protected]> wrote: >> Hmm, looks like I missed something..Not familiar with this code. >> >> On Thu, Dec 6, 2012 at 1:29 PM, DeCarlo, Thom <[email protected]> wrote: >>> Hmmm... How does one define/check/set the schema? I lifted my configuration >>> straight from the on-line documentation and it only mentions a schema in >>> the JDBC Channel section. >>> >>> -- >>> Thom DeCarlo >>> >>> >>> -----Original Message----- >>> From: Brock Noland [mailto:[email protected]] >>> Sent: Thursday, December 06, 2012 1:57 PM >>> To: [email protected] >>> Subject: Re: AVRO_EVENT problem >>> >>> It seems to me like the object you are trying to write doesn't match >>> the Schema? I'd check your convert() method. >>> >>> On Thu, Dec 6, 2012 at 9:09 AM, DeCarlo, Thom <[email protected]> wrote: >>>> Oh, right... forgot the version. I'm using the latest (as of yesterday) >>>> from the git repository. It gets built as 1.4-SNAPSHOT. >>>> >>>> -- >>>> Thom DeCarlo >>>> >>>> >>>> -----Original Message----- >>>> From: Brock Noland [mailto:[email protected]] >>>> Sent: Thursday, December 06, 2012 10:06 AM >>>> To: [email protected] >>>> Subject: Re: AVRO_EVENT problem >>>> >>>> Hi, >>>> >>>> Hopefully someone will be able to answer the AVRO issue, in order to >>>> help them, what version of Flume are you running? >>>> >>>> Brock >>>> >>>> On Thu, Dec 6, 2012 at 8:59 AM, DeCarlo, Thom <[email protected]> wrote: >>>>> Hi, >>>>> I'm just getting started with flume, so I apologize if this is an already >>>>> known problem. >>>>> >>>>> I'm trying to set up a FILE_ROLL sink that uses the AVRO_EVENT >>>>> serializer. But, when I start the agent I get an exception thrown from >>>>> within the AVRO DataFileWriter class. It is failing when trying to cast a >>>>> java.util.ArrayList to java.util.Map. >>>>> >>>>> Can anyone tell me how this is supposed to work? >>>>> >>>>> OBTW, I'm running on Windows 7 (64-bit), and I've tried both the 32- and >>>>> 64-bit versions of both Java 1.6.0_37 and 1.7.0_09, with the same results. >>>>> >>>>> Thanks, >>>>> Thom >>>>> -- >>>>> The console output looks like this: >>>>> >>>>> 2012-12-04 14:24:25,111 (lifecycleSupervisor-1-4) [INFO - >>>>> org.apache.flume.sink.RollingFileSink.start(RollingFileSink.java:135)] >>>>> RollingFileSink fileSink-1 started. >>>>> 2012-12-04 14:24:25,145 (lifecycleSupervisor-1-2) [INFO - >>>>> org.apache.flume.source.NetcatSource.start(NetcatSource.java:164)] >>>>> Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:33333] >>>>> 2012-12-04 14:24:25,593 (SinkRunner-PollingRunner-DefaultSinkProcessor) >>>>> [WARN - >>>>> org.apache.flume.channel.jdbc.impl.JdbcTransactionImpl.rollback(JdbcTransactionImpl.java:102)] >>>>> Marking transaction for rollback >>>>> 2012-12-04 14:24:25,594 (SinkRunner-PollingRunner-DefaultSinkProcessor) >>>>> [INFO - >>>>> org.apache.flume.channel.jdbc.impl.JdbcTransactionImpl.close(JdbcTransactionImpl.java:118)] >>>>> Attempting transaction roll-back >>>>> 2012-12-04 14:24:25,604 (SinkRunner-PollingRunner-DefaultSinkProcessor) >>>>> [ERROR - >>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] >>>>> Unable to deliver event. Exception follows. >>>>> org.apache.flume.EventDeliveryException: Failed to process transaction >>>>> at >>>>> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:218) >>>>> at >>>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) >>>>> at >>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) >>>>> at java.lang.Thread.run(Thread.java:722) >>>>> Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException: >>>>> java.lang.ClassCastException: java.util.ArrayList cannot be cast to >>>>> java.util.Map >>>>> at >>>>> org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:263) >>>>> at >>>>> org.apache.flume.serialization.AbstractAvroEventSerializer.write(AbstractAvroEventSerializer.java:108) >>>>> at >>>>> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:195) >>>>> ... 3 more >>>>> Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be >>>>> cast to java.util.Map >>>>> at >>>>> org.apache.avro.generic.GenericDatumWriter.getMapSize(GenericDatumWriter.java:174) >>>>> at >>>>> org.apache.avro.generic.GenericDatumWriter.writeMap(GenericDatumWriter.java:159) >>>>> at >>>>> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:68) >>>>> at >>>>> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:104) >>>>> at >>>>> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105) >>>>> at >>>>> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65) >>>>> at >>>>> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:104) >>>>> at >>>>> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57) >>>>> at >>>>> org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:257) >>>>> ... 5 more >>>>> >>>>> -- >>>>> My config file looks like this: >>>>> >>>>> ######################################## >>>>> # Sources >>>>> ######################################## >>>>> agent1.sources = source1 >>>>> >>>>> # Describe/configure source1 >>>>> agent1.sources.source1.type = netcat >>>>> agent1.sources.source1.bind = localhost >>>>> agent1.sources.source1.port = 33333 >>>>> agent1.sources.source1.interceptors = inter1 inter2 >>>>> agent1.sources.source1.interceptors.inter1.type = timestamp >>>>> agent1.sources.source1.interceptors.inter2.type = host >>>>> agent1.sources.source1.interceptors.inter2.hostHeader = hostname >>>>> >>>>> ######################################## >>>>> # Channels >>>>> ######################################## >>>>> agent1.channels = channel1 jdbcChannel-1 >>>>> >>>>> # Use a channel which buffers events in memory >>>>> agent1.channels.channel1.type = memory >>>>> agent1.channels.channel1.capacity = 1000 >>>>> agent1.channels.channel1.transactionCapacity = 100 >>>>> >>>>> # Add a channel which uses JDBC to buffer messages >>>>> agent1.channels.jdbcChannel-1.type = jdbc >>>>> >>>>> ######################################## >>>>> # Sinks >>>>> ######################################## >>>>> agent1.sinks = sink1 fileSink-1 >>>>> >>>>> # Describe sink1, a simple console Logger Sink >>>>> agent1.sinks.sink1.type = logger >>>>> >>>>> #Describe fileSink-1, which writes to a file >>>>> agent1.sinks.fileSink-1.type = FILE_ROLL >>>>> agent1.sinks.fileSink-1.sink.directory = c:/flume/logs >>>>> agent1.sinks.fileSink-1.sink.rollInterval = 3600 >>>>> #agent1.sinks.fileSink-1.sink.serializer = >>>>> org.apache.flume.serialization.HeaderAndBodyTextEventSerializer$Builder >>>>> agent1.sinks.fileSink-1.sink.serializer = AVRO_EVENT >>>>> agent1.sinks.fileSink-1.sink.serializer.compressionCodec = snappy >>>>> >>>>> ######################################## >>>>> # Bindings >>>>> ######################################## >>>>> # Bind the source and sink to the channel >>>>> agent1.sources.source1.channels = channel1 jdbcChannel-1 >>>>> agent1.sinks.sink1.channel = channel1 >>>>> agent1.sinks.fileSink-1.channel = jdbcChannel-1 >>>>> >>>>> >>>>> -- >>>>> Thom DeCarlo >>>>> >>>> >>>> >>>> >>>> -- >>>> Apache MRUnit - Unit testing MapReduce - >>>> http://incubator.apache.org/mrunit/ >>> >>> >>> >>> -- >>> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/ >> >> >> >> -- >> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/ > > > > -- > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/ -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
