Kafka stream only consume n messages

2018-08-07 Thread Kai Jiang
Hi community, Context: We are using Kafka stream to write an application. Generally, we did transformations on messages from one topic to another topic (not using join). In order to sneaking into output results a little bit before flowing it to destination topics, we want to tweak out a debug

Re: Error persisting into KeyValueStore

2018-08-07 Thread Matthias J. Sax
You might hit a RockDB issues as reported here: https://issues.apache.org/jira/browse/KAFKA-6327 -Matthias On 8/6/18 11:07 PM, Siva Ram wrote: > Hi, > > We switched to a virtual machine (from physical node) and we are observing > the following exception occurs and all our the stream application

Re: Kafka on Windows

2018-08-07 Thread jan
@Jacob Sheck: It was 18 months ago so I don't recall but I was told clearly windows ain't supported when I reported problems. I don't know if my problems were down to my inexperience or my use of windows, but I basically assuming I was controlling for one variable (n00bish inexperience) but was in

Re: Unsupported operand types when calling send

2018-08-07 Thread Skip Montanaro
> I'm porting some Kafka-using code from Python 2.7 to 3.6. My values > are Python dicts mapping strings to strings which are serialized with > json.dumps(). My keys are simple strings. I haven't been defining a > key serializer function. I'm using kafka_2.11-1.0.0 and kafka-python > 1.3.4 installe

Re: Apache Kafka Process showing high CPU (100 to 200+) usage in Linux when Idle

2018-08-07 Thread Abhijith Sreenivasan
Controller is not running in this node. We found that one of the new producer added was publishing only to a specific partition which was in this node, that explains the high utilization only in this node. The publisher was running in an Async thread invoking the send() asynchronously but the messa

Unsupported operand types when calling send

2018-08-07 Thread Skip Montanaro
I'm porting some Kafka-using code from Python 2.7 to 3.6. My values are Python dicts mapping strings to strings which are serialized with json.dumps(). My keys are simple strings. I haven't been defining a key serializer function. I'm using kafka_2.11-1.0.0 and kafka-python 1.3.4 installed from con

Re: Kafka on Windows

2018-08-07 Thread M. Manna
By fully broken, i mean not designed and tested to work on Windows. On Tue, 7 Aug 2018, 16:34 M. Manna, wrote: > Log Cleaner functionality is fully broken... If you haven't tried that > already. > > > > On 7 Aug 2018 4:24 pm, "Jacob Sheck" wrote: > > While I agree that it is less frustrating to

Re: Kafka on Windows

2018-08-07 Thread M. Manna
Log Cleaner functionality is fully broken... If you haven't tried that already. On 7 Aug 2018 4:24 pm, "Jacob Sheck" wrote: While I agree that it is less frustrating to run Kafka on Linux, I am interested to hear what specific issues you are running into on windows? On Tue, Aug 7, 2018 at 9:

Re: Kafka on Windows

2018-08-07 Thread Jacob Sheck
While I agree that it is less frustrating to run Kafka on Linux, I am interested to hear what specific issues you are running into on windows? On Tue, Aug 7, 2018 at 9:42 AM jan wrote: > This is an excellent suggestion and I intend to do so henceforth > (thanks!), but it would be an adjunct to m

Re: Kafka on Windows

2018-08-07 Thread jan
This is an excellent suggestion and I intend to do so henceforth (thanks!), but it would be an adjunct to my request rather than the answer; it still needs to be made clear in the docs/faq that you *can't* use windows directly. jan On 07/08/2018, Rahul Singh wrote: > I would recommend using Dock

Re: Kafka on Windows

2018-08-07 Thread Rahul Singh
I would recommend using Docker — it would end up being run on a Linux kernel VM on windows and is easier to get started on with a bit of learning curve for Docker. Less time wasted overall and at least at that point you would know Docker. Rahul On Aug 7, 2018, 4:50 AM -0400, jan , wrote: > I tr

Re: Kafka on Windows

2018-08-07 Thread jan
I tried using it just for learning a while back and wasted 3 days because it's not supported on windows. Even basic stuff didn't work. I did read the docs first! I think I've seen other people on this list have questions about/problems for exactly the same reason, and that could be a lot of time s

Re: Kafka on Windows

2018-08-07 Thread M. Manna
The answer is - Absolutely not. If you don’t have Linux rack, or Kubernetes deployment -it will not work on Windows as guaranteed. I know this because I have tried to make it work for the past 1 year. File handling always fails and crashes the cluster on Windows. Thanks, On Tue, 7 Aug 2018 at