Ben, There is a bug in KafkaETLContext.java in where the iterator is created multiple times. You ll have this issue only when kafka.request.limit is set to -1 in your test.properties.
Line 120 Iterator<ByteBufferMessageSet> iter = _response.iterator(); This is what is causing the loops . In my setup , I fixed it by having a _respIterator attribute in the class and assigning it to _response.iterator() just once in the fetchMore() method. I use the class attribute in the rest of the places. Hope this helps. Sam On Sep 1, 2011, at 9:18 AM, Ben Ciceron wrote: > hello, > > i am able to run the job now but it loops indefinitely. > > can someone send a valid sample of 1.dat as generated by > kafka.etl.impl.DataGenerator in the README, please ? > mine seems awfully short (see below). > >>> 3. 1.dat is a sequence file, so if it isn't compressed, you should be >>> able to see its contents in plain text. The contents will essentially list >>> the kafka server url, the partition number and the topic as well as the >>> offset. >> >> >> mine has only 1 line (some encrypted in the middle of it is shown as is): >> SEQkafka.etl.KafkaETLKey"org.apache.hadoop.io.BytesWritable$?ړ??tl?3aC+tcp://<ip_of_my_hostA>:9092 >> SimpleTestEvent 0 -1 Sam William sa...@stumbleupon.com
