Re: fsync behaviour

2014-08-21 Thread Manjunath Shivakumar
Thanks Jun. If the log flush interval is more than 1, will the unflushed messages be visible to the consumers before being written to disk. Thanks, Manju On 21 Aug 2014 06:03, Jun Rao jun...@gmail.com wrote: 1) if you configure the log flush interval to be 1 message, every produce request will

Re: Data inputs for Kafka.

2014-08-21 Thread siddharth ubale
Hi , Thanks for the quick follow up Philip. Also , can you let me know whether the kafka implementation can be as versatile as ActiveMQ with regards to connectivity? I mean to say a web-app generating transactional data in a data base from java webapp, ruby web app or .net web app can kafka

Re: Data inputs for Kafka.

2014-08-21 Thread Sharninder
there are kafka producers/consumers in a lot of languages. There is an Ecosystem page somewhere in the wiki. Take a look at that. Finally, I think you should experiment with both activemq and kafka and then take a decision. Don't try to find a solution and fit your problem to it, should be the

Kafka Mirroring Issue

2014-08-21 Thread Ashutosh Prasad Panigrahi
Hello All, When i start the mirroring process, events are written only to partition 1 on the target cluster and no data is written to partition 0. I am stuck on this and wondering what am i doing wrong . My production setup looks something like this 1. Datacenter1 :3 kafka nodes/3

Re: Data inputs for Kafka.

2014-08-21 Thread siddharth ubale
Hi , In continuity to this thread, can you let me know if i will have to write a kafka producer in C# if i have to get data from c# application onto the kafka bus or it will be handled by the native java producer API?? and like wise for any system generating data which is using any other

RE: fsync behaviour

2014-08-21 Thread Manjunath Shivakumar
Or to re frame the question when replication is enabled. Will the message be available to consumers as soon as the message has been replicated to an in-sync follower, even before it is flushed to the disk. Thanks, Manju From: Manjunath Shivakumar Sent: 21 August

Re: Data inputs for Kafka.

2014-08-21 Thread Sharninder
You can write your producer in c# ( https://github.com/kafka-dev/kafka/blob/master/clients/csharp/README.md), java, python or a number of other languages. This of Kafka as a separate system that is waiting to receive data in a particular format. The language used to send that data doesn't matter.

Re: Data inputs for Kafka.

2014-08-21 Thread siddharth ubale
Thnks sharninder!!! You kinda nailed it for me ;-) On Thu, Aug 21, 2014 at 6:15 PM, Sharninder sharnin...@gmail.com wrote: You can write your producer in c# ( https://github.com/kafka-dev/kafka/blob/master/clients/csharp/README.md), java, python or a number of other languages. This of

Re: fsync behaviour

2014-08-21 Thread Jun Rao
That's right. A message is exposed to the consumer as soon as it reaches all in-sync replicas. Whether it's flushed to disk or not depends on the configuration of flush interval. Thanks, Jun On Thu, Aug 21, 2014 at 3:37 AM, Manjunath Shivakumar manjunath.shivaku...@betfair.com wrote: Or to

Re: kafka 0.8.1.1 log.retention.minutes NOT being honored

2014-08-21 Thread François Langelier
Thanks Jun, I don't see changes in the documentation :( Just to be sure, We can either use *log.retention.hours* (giving the number of *hours* you want the retention to last) or *log.retention.minutes* (giving the number of *minutes* you want the retention to last) for all the topics (in

Re: Kafka Mirroring Issue

2014-08-21 Thread Guozhang Wang
Hi Ashutosh, Maybe this faq can help you? https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified ? Guozhang On Thu, Aug 21, 2014 at 12:09 AM, Ashutosh Prasad Panigrahi ashutosh1...@gmail.com wrote: Hello All,

Re: kafka 0.8.1.1 log.retention.minutes NOT being honored

2014-08-21 Thread Jun Rao
http://kafka.apache.org/documentation.html#configuration now has the following entry. log.retention.{minutes,hours} Thanks, Jun On Thu, Aug 21, 2014 at 7:44 AM, François Langelier f.langel...@gmail.com wrote: Thanks Jun, I don't see changes in the documentation :( Just to be sure, We