Re: Poll: Producer/Consumer impl/language you use?
I agree with Stephen, it would be really unfortunate to see the Scala api go away. On Wed, Jan 28, 2015 at 11:57 AM, Stephen Boesch java...@gmail.com wrote: The scala API going away would be a minus. As Koert mentioned we could use the java api but it is less .. well .. functional. Kafka is included in the Spark examples and external modules and is popular as a component of ecosystems on Spark (for which scala is the primary language). 2015-01-28 8:51 GMT-08:00 Otis Gospodnetic otis.gospodne...@gmail.com: Hi, I don't have a good excuse here. :( I thought about including Scala, but for some reason didn't do it. I see 12-13% of people chose Other. Do you think that is because I didn't include Scala? Also, is the Scala API reeally going away? Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Tue, Jan 20, 2015 at 4:43 PM, Koert Kuipers ko...@tresata.com wrote: no scala? although scala can indeed use the java api, its ugly we prefer to use the scala api (which i believe will go away unfortunately) On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, I was wondering which implementations/languages people use for their Kafka Producer/Consumers not everyone is using the Java APIs. So here's a 1-question poll: http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/ Will share the results in about a week when we have enough votes. Thanks! Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/
Re: Poll: Producer/Consumer impl/language you use?
Yeah Joe is exactly right. Let's not confuse scala apis with the existing Scala clients There are a ton of downsides to those clients. They aren't going away any time in the forceable future, so don't stress, but I think we can kind of deprecate them and try to shame people into upgrading. For the Java producer I actually think that API is no worse in Scala than the existing Scala api, so I'm not sure if there is much we can do to improve it for scala, but if there is there would be no harm in adding a scala wrapper. For the new Java consumer there are some Java-isms in the client (e.g. a couple methods take java maps as arguments). I actually think the existing scala implicit conversions for java collections might be totally sufficient but someone would have to try. If not we can add a scala wrapper. I actually think there is room for lots of wrappers that experiment with different styles of data access, especially for the consumer. The reactive people have a bunch of stream related things, Joe mentioned scalaz streams, etc. Actually the iterator api in the existing consumer was an attempt to make stream processing easy (since you can apply some of the scala collections things to the resulting iterator) but I think it wasn't thought all the way through. I think having this simpler, more flexible base api will let people experiment with this stuff in any way they want. -Jay On Wed, Jan 28, 2015 at 10:45 AM, Joe Stein joe.st...@stealth.ly wrote: I kind of look at the Storm, Spark, Samza, etc integrations as producers/consumers too. Not sure if that maybe was getting lumped also into other. I think Jason's 90/10 80/20 70/30 would be found to be typical. As far as the Scala API goes, I think we should have a wrapper around the shiny new Java Consumer. Folks I know use things like scalaz streams https://github.com/scalaz/scalaz-stream which the new consumer can work nicely with I think. It would be great if we could come up with a new Scala layer on top of the Java consumer that we release in the project. One of my engineers is taking a look at that now unless someone is already working on that? We are using the partition static assignment in the new consumer and just using Mesos for handling re-balance for us. When he gets further along and if it makes sense we will shoot a KIP around people can chat about it on dev. - Joe Stein On Wed, Jan 28, 2015 at 10:33 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Good point, Jason. Not sure how we could account for that easily. But maybe that is at least a partial explanation of the Java % being under 50% when Java in general is more popular than that... Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Wed, Jan 28, 2015 at 1:22 PM, Jason Rosenberg j...@squareup.com wrote: I think the results could be a bit skewed, in cases where an organization uses multiple languages, but not equally. In our case, we overwhelmingly use java clients (90%). But we also have ruby and Go clients too. But in the poll, these come out as equally used client languages. Jason On Wed, Jan 28, 2015 at 12:05 PM, David McNelis dmcne...@emergingthreats.net wrote: I agree with Stephen, it would be really unfortunate to see the Scala api go away. On Wed, Jan 28, 2015 at 11:57 AM, Stephen Boesch java...@gmail.com wrote: The scala API going away would be a minus. As Koert mentioned we could use the java api but it is less .. well .. functional. Kafka is included in the Spark examples and external modules and is popular as a component of ecosystems on Spark (for which scala is the primary language). 2015-01-28 8:51 GMT-08:00 Otis Gospodnetic otis.gospodne...@gmail.com : Hi, I don't have a good excuse here. :( I thought about including Scala, but for some reason didn't do it. I see 12-13% of people chose Other. Do you think that is because I didn't include Scala? Also, is the Scala API reeally going away? Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Tue, Jan 20, 2015 at 4:43 PM, Koert Kuipers ko...@tresata.com wrote: no scala? although scala can indeed use the java api, its ugly we prefer to use the scala api (which i believe will go away unfortunately) On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, I was wondering which implementations/languages people use for their Kafka Producer/Consumers not everyone is using the Java APIs. So here's a 1-question
Re: Poll: Producer/Consumer impl/language you use?
Hi, I don't have a good excuse here. :( I thought about including Scala, but for some reason didn't do it. I see 12-13% of people chose Other. Do you think that is because I didn't include Scala? Also, is the Scala API reeally going away? Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Tue, Jan 20, 2015 at 4:43 PM, Koert Kuipers ko...@tresata.com wrote: no scala? although scala can indeed use the java api, its ugly we prefer to use the scala api (which i believe will go away unfortunately) On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, I was wondering which implementations/languages people use for their Kafka Producer/Consumers not everyone is using the Java APIs. So here's a 1-question poll: http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/ Will share the results in about a week when we have enough votes. Thanks! Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/
Re: Poll: Producer/Consumer impl/language you use?
I kind of look at the Storm, Spark, Samza, etc integrations as producers/consumers too. Not sure if that maybe was getting lumped also into other. I think Jason's 90/10 80/20 70/30 would be found to be typical. As far as the Scala API goes, I think we should have a wrapper around the shiny new Java Consumer. Folks I know use things like scalaz streams https://github.com/scalaz/scalaz-stream which the new consumer can work nicely with I think. It would be great if we could come up with a new Scala layer on top of the Java consumer that we release in the project. One of my engineers is taking a look at that now unless someone is already working on that? We are using the partition static assignment in the new consumer and just using Mesos for handling re-balance for us. When he gets further along and if it makes sense we will shoot a KIP around people can chat about it on dev. - Joe Stein On Wed, Jan 28, 2015 at 10:33 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Good point, Jason. Not sure how we could account for that easily. But maybe that is at least a partial explanation of the Java % being under 50% when Java in general is more popular than that... Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Wed, Jan 28, 2015 at 1:22 PM, Jason Rosenberg j...@squareup.com wrote: I think the results could be a bit skewed, in cases where an organization uses multiple languages, but not equally. In our case, we overwhelmingly use java clients (90%). But we also have ruby and Go clients too. But in the poll, these come out as equally used client languages. Jason On Wed, Jan 28, 2015 at 12:05 PM, David McNelis dmcne...@emergingthreats.net wrote: I agree with Stephen, it would be really unfortunate to see the Scala api go away. On Wed, Jan 28, 2015 at 11:57 AM, Stephen Boesch java...@gmail.com wrote: The scala API going away would be a minus. As Koert mentioned we could use the java api but it is less .. well .. functional. Kafka is included in the Spark examples and external modules and is popular as a component of ecosystems on Spark (for which scala is the primary language). 2015-01-28 8:51 GMT-08:00 Otis Gospodnetic otis.gospodne...@gmail.com : Hi, I don't have a good excuse here. :( I thought about including Scala, but for some reason didn't do it. I see 12-13% of people chose Other. Do you think that is because I didn't include Scala? Also, is the Scala API reeally going away? Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Tue, Jan 20, 2015 at 4:43 PM, Koert Kuipers ko...@tresata.com wrote: no scala? although scala can indeed use the java api, its ugly we prefer to use the scala api (which i believe will go away unfortunately) On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, I was wondering which implementations/languages people use for their Kafka Producer/Consumers not everyone is using the Java APIs. So here's a 1-question poll: http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/ Will share the results in about a week when we have enough votes. Thanks! Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/
Re: Poll: Producer/Consumer impl/language you use?
Good point, Jason. Not sure how we could account for that easily. But maybe that is at least a partial explanation of the Java % being under 50% when Java in general is more popular than that... Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Wed, Jan 28, 2015 at 1:22 PM, Jason Rosenberg j...@squareup.com wrote: I think the results could be a bit skewed, in cases where an organization uses multiple languages, but not equally. In our case, we overwhelmingly use java clients (90%). But we also have ruby and Go clients too. But in the poll, these come out as equally used client languages. Jason On Wed, Jan 28, 2015 at 12:05 PM, David McNelis dmcne...@emergingthreats.net wrote: I agree with Stephen, it would be really unfortunate to see the Scala api go away. On Wed, Jan 28, 2015 at 11:57 AM, Stephen Boesch java...@gmail.com wrote: The scala API going away would be a minus. As Koert mentioned we could use the java api but it is less .. well .. functional. Kafka is included in the Spark examples and external modules and is popular as a component of ecosystems on Spark (for which scala is the primary language). 2015-01-28 8:51 GMT-08:00 Otis Gospodnetic otis.gospodne...@gmail.com : Hi, I don't have a good excuse here. :( I thought about including Scala, but for some reason didn't do it. I see 12-13% of people chose Other. Do you think that is because I didn't include Scala? Also, is the Scala API reeally going away? Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Tue, Jan 20, 2015 at 4:43 PM, Koert Kuipers ko...@tresata.com wrote: no scala? although scala can indeed use the java api, its ugly we prefer to use the scala api (which i believe will go away unfortunately) On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, I was wondering which implementations/languages people use for their Kafka Producer/Consumers not everyone is using the Java APIs. So here's a 1-question poll: http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/ Will share the results in about a week when we have enough votes. Thanks! Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/
Re: Poll: Producer/Consumer impl/language you use?
I think the results could be a bit skewed, in cases where an organization uses multiple languages, but not equally. In our case, we overwhelmingly use java clients (90%). But we also have ruby and Go clients too. But in the poll, these come out as equally used client languages. Jason On Wed, Jan 28, 2015 at 12:05 PM, David McNelis dmcne...@emergingthreats.net wrote: I agree with Stephen, it would be really unfortunate to see the Scala api go away. On Wed, Jan 28, 2015 at 11:57 AM, Stephen Boesch java...@gmail.com wrote: The scala API going away would be a minus. As Koert mentioned we could use the java api but it is less .. well .. functional. Kafka is included in the Spark examples and external modules and is popular as a component of ecosystems on Spark (for which scala is the primary language). 2015-01-28 8:51 GMT-08:00 Otis Gospodnetic otis.gospodne...@gmail.com: Hi, I don't have a good excuse here. :( I thought about including Scala, but for some reason didn't do it. I see 12-13% of people chose Other. Do you think that is because I didn't include Scala? Also, is the Scala API reeally going away? Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Tue, Jan 20, 2015 at 4:43 PM, Koert Kuipers ko...@tresata.com wrote: no scala? although scala can indeed use the java api, its ugly we prefer to use the scala api (which i believe will go away unfortunately) On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, I was wondering which implementations/languages people use for their Kafka Producer/Consumers not everyone is using the Java APIs. So here's a 1-question poll: http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/ Will share the results in about a week when we have enough votes. Thanks! Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/
Re: Poll: Producer/Consumer impl/language you use?
The scala API going away would be a minus. As Koert mentioned we could use the java api but it is less .. well .. functional. Kafka is included in the Spark examples and external modules and is popular as a component of ecosystems on Spark (for which scala is the primary language). 2015-01-28 8:51 GMT-08:00 Otis Gospodnetic otis.gospodne...@gmail.com: Hi, I don't have a good excuse here. :( I thought about including Scala, but for some reason didn't do it. I see 12-13% of people chose Other. Do you think that is because I didn't include Scala? Also, is the Scala API reeally going away? Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Tue, Jan 20, 2015 at 4:43 PM, Koert Kuipers ko...@tresata.com wrote: no scala? although scala can indeed use the java api, its ugly we prefer to use the scala api (which i believe will go away unfortunately) On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, I was wondering which implementations/languages people use for their Kafka Producer/Consumers not everyone is using the Java APIs. So here's a 1-question poll: http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/ Will share the results in about a week when we have enough votes. Thanks! Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/
Re: Poll: Producer/Consumer impl/language you use?
no scala? although scala can indeed use the java api, its ugly we prefer to use the scala api (which i believe will go away unfortunately) On Tue, Jan 20, 2015 at 2:52 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, I was wondering which implementations/languages people use for their Kafka Producer/Consumers not everyone is using the Java APIs. So here's a 1-question poll: http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/ Will share the results in about a week when we have enough votes. Thanks! Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/
