Hey Guozhang,
I just took a quick look at the KIP, is it very similar to mirror maker
with message handler?
Thanks,
Jiangjie (Becket) Qin
On Thu, Jul 23, 2015 at 10:25 PM, Ewen Cheslack-Postava e...@confluent.io
wrote:
Just some notes on the KIP doc itself:
* It'd be useful to clarify at
Ewen:
* I think trivial filtering and aggregation on a single stream usually work
fine with this model.
The way I see this, the process() API is an abstraction for
message-at-a-time computations. In the future, you could imagine providing
a simple DSL layer on top of the process() API that
[
https://issues.apache.org/jira/browse/KAFKA-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640080#comment-14640080
]
Bo Wang commented on KAFKA-1482:
When I execute unit tests reported the same fault！
@Becket - for compressed batches, I think this just works out given the KIP
as described. Without the change you're referring to, it still only makes
sense to batch messages with this KIP if all the expected offsets are
sequential (else some messages are guaranteed to fail). I think that
probably
[
https://issues.apache.org/jira/browse/KAFKA-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640181#comment-14640181
]
ASF GitHub Bot commented on KAFKA-2358:
---
GitHub user sslavic opened a pull request:
GitHub user sslavic opened a pull request:
https://github.com/apache/kafka/pull/96
KAFKA-2358 KafkaConsumer.partitionsFor returns empty list for non-existing
topic
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sslavic/kafka
Hi Jay,
On Fri, Jul 24, 2015 at 2:00 AM, Jay Kreps j...@confluent.io wrote:
- Wait for java 8 to become common so we don't have to translate to java 7
...
Even ignoring those items I don't think we would dare take this on right
now because there are so many other projects in flight but it
Hi, Guozhang,
Thanks for starting this. I took a quick look and had the following
thoughts to share:
- In the proposed KafkaProcessor API, there is no interface like Collector
that allows users to send messages to. Why is that? Is the idea to
initialize the producer once and re-use it in the
Stevo Slavic created KAFKA-2358:
---
Summary: KafkaConsumer.partitionsFor should never return null
Key: KAFKA-2358
URL: https://issues.apache.org/jira/browse/KAFKA-2358
Project: Kafka
Issue Type:
[
https://issues.apache.org/jira/browse/KAFKA-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stevo Slavic updated KAFKA-2358:
Description:
{{KafkaConsumer.partitionsFor}} method by it's signature returns a
[
https://issues.apache.org/jira/browse/KAFKA-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stevo Slavic updated KAFKA-2358:
Summary: Cluster collection returning methods should never return null
(was:
Noting here that the period '.' also causes potentially confusing behavior
when using regex whitelists or blacklists. It can be easily worked around
but users need to be aware of escaping the period.
If I create two topics 'a.c' and 'abc' and start the following consumer,
both topics will be
Ewen,
This is quite useful. I used the command that is suggested in the ASFBot
comment:
git pull https://github.com/[contribuer-name]/kafka KAFKA-
to apply the patch locally. But this would actually better I think, just
that it requires installing hub first. Maybe we can add this to the
[
https://issues.apache.org/jira/browse/KAFKA-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-2348:
-
Resolution: Fixed
Fix Version/s: 0.9.0
Status: Resolved (was: Patch Available)
Stevo Slavic created KAFKA-2359:
---
Summary: New consumer - partitions auto assigned only on poll
Key: KAFKA-2359
URL: https://issues.apache.org/jira/browse/KAFKA-2359
Project: Kafka
Issue Type:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34554/#review92917
---
Ship it!
Thanks for the patch. Looks good. +1 after addressing a
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36652/#review92920
---
Thanks for the patch. A few comments below.
[
https://issues.apache.org/jira/browse/KAFKA-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640624#comment-14640624
]
Jun Rao commented on KAFKA-251:
---
Since we are developing the new java consumer, not sure if
+1 for consistency.
On Fri, Jul 24, 2015 at 8:51 AM, Neha Narkhede n...@confluent.io wrote:
Personally, I prefer the consistency and 2 spaces for indentation. Although
I'm in agreement with Jay about the ambition to eventually move the server
code to Java, I'd hate to make that a blocker for
[
https://issues.apache.org/jira/browse/KAFKA-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640654#comment-14640654
]
Gwen Shapira commented on KAFKA-2100:
-
Regarding how often it happens: I believe its
[
https://issues.apache.org/jira/browse/KAFKA-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640694#comment-14640694
]
ASF GitHub Bot commented on KAFKA-2348:
---
Github user asfgit closed the pull request
Github user asfgit closed the pull request at:
https://github.com/apache/kafka/pull/87
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
[
https://issues.apache.org/jira/browse/KAFKA-1175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640608#comment-14640608
]
Joris Peeters commented on KAFKA-1175:
--
I agree that this would be a very useful
[
https://issues.apache.org/jira/browse/KAFKA-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641024#comment-14641024
]
Jason Gustafson commented on KAFKA-2055:
[~lvfangmin] Looks reasonable to me. Want
[
https://issues.apache.org/jira/browse/KAFKA-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641027#comment-14641027
]
Fangmin Lv commented on KAFKA-2055:
---
Sure, update it now.
Github user lvfangmin closed the pull request at:
https://github.com/apache/kafka/pull/60
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
[
https://issues.apache.org/jira/browse/KAFKA-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641033#comment-14641033
]
ASF GitHub Bot commented on KAFKA-2055:
---
Github user lvfangmin closed the pull
Personally, I prefer the consistency and 2 spaces for indentation. Although
I'm in agreement with Jay about the ambition to eventually move the server
code to Java, I'd hate to make that a blocker for discussing a simple
change like standardizing on java/scala indentation :-)
On Fri, Jul 24, 2015
[
https://issues.apache.org/jira/browse/KAFKA-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641034#comment-14641034
]
ASF GitHub Bot commented on KAFKA-2055:
---
GitHub user lvfangmin opened a pull
GitHub user lvfangmin opened a pull request:
https://github.com/apache/kafka/pull/98
KAFKA-2055; Fix transient ConsumerBounceTest.testSeekAndCommitWithBroâ¦
â¦kerFailures failure;
You can merge this pull request into a Git repository by running:
$ git pull
Jay, I understand that. Context can provide more information without
breaking the compatibility if needed. Also I am not sure ConsumerRecord is
the right abstraction of data for stream processing. After transformation
or join, what is the topic and the offset? It is odd to use ConsumerRecord.
We
[
https://issues.apache.org/jira/browse/KAFKA-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640833#comment-14640833
]
Fangmin Lv commented on KAFKA-2055:
---
Hi Jason, sorry for lately reply, if you don't
Hi Jiangjie,
Inlined.
On Thu, Jul 23, 2015 at 11:32 PM, Jiangjie Qin j...@linkedin.com.invalid
wrote:
Hey Guozhang,
I just took a quick look at the KIP, is it very similar to mirror maker
with message handler?
I think the processor client would supporting a superset of functionalities
The goal of this KIP is to provide a lightweight/embeddable streaming
framework, and allows Kafka users to start using stream processing easily. DSL
is not covered in this KIP. But, DSL is a very attractive option to have.
In the proposed KafkaProcessor API, there is no interface like Collector
[
https://issues.apache.org/jira/browse/KAFKA-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640806#comment-14640806
]
Jason Gustafson commented on KAFKA-2055:
[~lvfangmin] I'm going to go ahead and
Hey Yi,
For your other two points:
- This definitely doesn't cover any kind of SQL or anything like this.
- The prototype we started with just had process() as a method but Yasuhiro
had some ideas of adding additional filter/aggregate convenience methods.
We should discuss how this would fit
[
https://issues.apache.org/jira/browse/KAFKA-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640730#comment-14640730
]
Jay Kreps commented on KAFKA-2359:
--
Interesting. This behavior was intentional in that
Ismael,
Makes sense. I think there is a good chance that it is just our ignorance
of scala tools. I really do like having compile time enforced formatting
and dependency checking as we have for java. But we really put no effort
into trying to improve the scala developer experience so it may be an
Hi Yi,
Inlined.
On Fri, Jul 24, 2015 at 12:57 AM, Yi Pan nickpa...@gmail.com wrote:
Hi, Guozhang,
Thanks for starting this. I took a quick look and had the following
thoughts to share:
- In the proposed KafkaProcessor API, there is no interface like Collector
that allows users to send
@guozhang re: library vs framework - Nothing critical missing. But I think
KIPs will serve as a sort of changelog since ones like this
https://archive.apache.org/dist/kafka/0.8.2.0/RELEASE_NOTES.html are not
all that helpful to end users. A KIP like this is a major new addition, so
laying out all
See https://builds.apache.org/job/KafkaPreCommit/163/changes
Changes:
[wangguoz] KAFKA-2348; Drop support for Scala 2.9
--
Started by an SCM change
Building remotely on ubuntu3 (Ubuntu ubuntu legacy-ubuntu) in workspace
I'm with Neha on this one. I don't have a strong preference on 2 vs 4 but I
do think that consistency is more important. It makes writing code a bit
easier especially since patches are increasingly likely to touch both Java
and Scala code and it's nice to not think about formatting certain files
[
https://issues.apache.org/jira/browse/KAFKA-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640841#comment-14640841
]
Jason Gustafson commented on KAFKA-2055:
[~lvfangmin] Sure, no problem.
This sounds like an interesting problem. What is your consumerTimeout set
to?
Thanks,
Mayuresh
On Tue, Jul 21, 2015 at 6:12 PM, Pranay Agarwal agarwalpran...@gmail.com
wrote:
Thanks Mayuresh,
Can I at least control the rebalance of consumers? Currently consumes die
after specific partition
Hi Ewen,
Replies inlined.
On Thu, Jul 23, 2015 at 10:25 PM, Ewen Cheslack-Postava e...@confluent.io
wrote:
Just some notes on the KIP doc itself:
* It'd be useful to clarify at what point the plain consumer + custom code
+ producer breaks down. I think trivial filtering and aggregation on a
Agree that the normal KIP process is awkward for larger changes like this.
I'm a +1 on trying out this new process for the processor client, see how
it works out and then make that a process for future large changes of this
nature.
On Fri, Jul 24, 2015 at 10:03 AM, Jay Kreps j...@confluent.io
On Fri, Jul 24, 2015 at 2:00 AM, Jay Kreps j...@confluent.io wrote:
I do agree that working with a mixture of scala and java is a pain in the
butt. What about considering the more extreme idea of just moving the
remaining server-side scala into java? I like Scala, but the tooling and
[
https://issues.apache.org/jira/browse/KAFKA-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-2348:
---
Fix Version/s: (was: 0.9.0)
0.8.3
Drop support for Scala 2.9
[
https://issues.apache.org/jira/browse/KAFKA-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641118#comment-14641118
]
Guozhang Wang commented on KAFKA-2100:
--
I like option #2 as well, I think it is
[
https://issues.apache.org/jira/browse/KAFKA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641246#comment-14641246
]
Guozhang Wang commented on KAFKA-2349:
--
[~ijuma] Looks good to me overall. One thing
[
https://issues.apache.org/jira/browse/KAFKA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641250#comment-14641250
]
Ismael Juma commented on KAFKA-2349:
Yes, that sounds good to me. Agreed that it would
[
https://issues.apache.org/jira/browse/KAFKA-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640941#comment-14640941
]
David Jacot commented on KAFKA-2100:
I think that UNKNOWN must be kept in the Errors
[
https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640958#comment-14640958
]
Yasuhiro Matsuda commented on KAFKA-2350:
-
Throwing exceptions makes sense.
In
[
https://issues.apache.org/jira/browse/KAFKA-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640943#comment-14640943
]
Fangmin Lv commented on KAFKA-2055:
---
[~guozhang], I've just settled down in a new
[
https://issues.apache.org/jira/browse/KAFKA-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640955#comment-14640955
]
Gwen Shapira commented on KAFKA-2100:
-
We have KAFKA-1929 for re-using the Errors
[
https://issues.apache.org/jira/browse/KAFKA-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fangmin Lv updated KAFKA-2055:
--
Attachment: KAFKA-2055.patch
Update patch to use TestUtils.waitUntilTrue instead of sleep.
[
https://issues.apache.org/jira/browse/KAFKA-2351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641009#comment-14641009
]
Mayuresh Gharat commented on KAFKA-2351:
Hi [~junrao],
Thanks for the comments on
To follow on to one of Yi's points about taking ConsumerRecord vs
topic/key/value. One thing we have found is that for user-facing APIs
considering future API evolution is really important. If you do
topic/key/value and then realize you need offset added you end up having to
break everyones code.
[
https://issues.apache.org/jira/browse/KAFKA-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Lin updated KAFKA-2066:
Assignee: (was: Dong Lin)
Replace FetchRequest / FetchResponse with their
[
https://issues.apache.org/jira/browse/KAFKA-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aditya A Auradkar updated KAFKA-2205:
-
Attachment: KAFKA-2205_2015-07-24_18:11:34.patch
Generalize TopicConfigManager to handle
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34554/
---
(Updated July 25, 2015, 1:12 a.m.)
Review request for kafka, Joel Koshy and
Hi,
I started seeing these errors in the logs continuously when I try to bring
the High Level Consumer up. Please help.
ZookeeperConsumerConnector [INFO] [XXX], waiting for the partition
ownership to be deleted: 1
ZookeeperConsumerConnector [INFO] [XXX], end rebalancing
consumer
Agree. All I found on this was KAFKA-1325, which is more about
inconsistency than real use-case.
Anyway, I'd argue that if we added it in 0.8.2.0, we can't take it out now :)
On Fri, Jul 24, 2015 at 7:04 PM, Aditya Auradkar
aaurad...@linkedin.com.invalid wrote:
Is there actually a use case
Hmm.
This creates confusion for people who might be new to kafka or have never
used those properties.
Does deprecating the older ones make sense here?
Thanks,
Mayuresh
On Fri, Jul 24, 2015 at 7:15 PM, Gwen Shapira gshap...@cloudera.com wrote:
Agree. All I found on this was KAFKA-1325, which
[
https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641354#comment-14641354
]
Sriharsha Chintalapani commented on KAFKA-1690:
---
[~lindong] I guess that
Hi Guys,
I am trying to send logs from solr directly to kafka using log4j. I have made
changes in my log4j.properties file, and am not sure where is the problem.
Also, do I need to include a kafka jar file? Can somebody help me on this?
The warning and error I
[
https://issues.apache.org/jira/browse/KAFKA-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640448#comment-14640448
]
David Jacot commented on KAFKA-2100:
I have looked at how Errors are used in the java
[
https://issues.apache.org/jira/browse/KAFKA-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Jacot updated KAFKA-2100:
---
Attachment: KAFKA-2100-1.patch
Client Error doesn't preserve or display original server error code
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34554/
---
(Updated July 25, 2015, 1:11 a.m.)
Review request for kafka, Joel Koshy and
[
https://issues.apache.org/jira/browse/KAFKA-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641311#comment-14641311
]
Aditya A Auradkar commented on KAFKA-2205:
--
Updated reviewboard
[
https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641327#comment-14641327
]
Dong Lin commented on KAFKA-1690:
-
[~stp_abhi] Is the patch available for test? It will be
Backward compatibility, I think.
At least the ms one is fairly new, and I think we left the others to
avoid break configuration during upgrade.
On Fri, Jul 24, 2015 at 6:34 PM, Mayuresh Gharat
gharatmayures...@gmail.com wrote:
I was thinking why we have 3 different configs for the same property
Oh ok.
Is there a plan that we should deprecate the older ones. This is because as
I said some users might misconfigure it like by using both the
log.retention.ms and log.retention.minutes by mistake.
We can probably set a warning that only one of this should be used.
Also looking at the
Cool. Yeah +1 on make it consistent.
Thanks,
Mayuresh
On Fri, Jul 24, 2015 at 7:29 PM, Grant Henke ghe...@cloudera.com wrote:
They take precedence in order of granularity (ms minutes hours). Below
is the relevant snippet of code.
I vote for keeping all time related configs in
This also came up recently when Geoff and I were discussing KAFKA-2257,
which needs to add a timeout option to a tool that KAFKA-2276 is
introducing. For that timeout, the question is whether we should strive for
consistency (use ms), the unit that is most likely convenient (use s), or
finest
On 24 Jul 2015 18:03, Jay Kreps j...@confluent.io wrote:
Does this make sense to people? If so let's try it and if we like it
better
we can formally make that the process for this kind of big thing.
Yes, sounds good to me.
Best,
Ismael
[
https://issues.apache.org/jira/browse/KAFKA-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641313#comment-14641313
]
Aditya Auradkar commented on KAFKA-2205:
[~junrao] One more time :). Addressed
I was thinking why we have 3 different configs for the same property (log
retention) :
log.retention.ms
log.retention.minutes
log.retention.hours
Why don't we only use the Milliseconds?
There are other properties as well like log Jitter, LogRollTime which raise
the same question in my mind.
Is there actually a use case where we need log.retention.ms? In most
cases, people would want to retain their logs for at least a few minutes
I'd think.
Aditya
On Fri, Jul 24, 2015 at 6:49 PM, Gwen Shapira gshap...@cloudera.com wrote:
Backward compatibility, I think.
At least the ms one is
They take precedence in order of granularity (ms minutes hours). Below
is the relevant snippet of code.
I vote for keeping all time related configs in milliseconds and deprecating
the others. It gives you the most control in a single config. The minutes
and hours ones offer no new functionality
[
https://issues.apache.org/jira/browse/KAFKA-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-2321:
---
Attachment: (was: KAFKA-2321.patch)
Introduce CONTRIBUTING.md
-
[
https://issues.apache.org/jira/browse/KAFKA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640360#comment-14640360
]
Ismael Juma commented on KAFKA-2349:
Vote has passed, so this is ready for review (and
Hi Ewen,
On Fri, Jul 24, 2015 at 5:32 AM, Ewen Cheslack-Postava e...@confluent.io
wrote:
One other thing that might be worth mentioning in The Review Process
section is the hub tool: https://hub.github.com/ I only ever use one
feature, but it's immensely useful if you're reviewing PRs on
Thank you for voting. 72 hours have elapsed and the vote has passed with 4
binding +1s. There were also 8 non-binding +1s.
I removed the In Progress warning from:
https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes
It would be good for a committer to review (and
GitHub user ijuma opened a pull request:
https://github.com/apache/kafka/pull/97
KAFKA-2321; Introduce CONTRIBUTING.md
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ijuma/kafka kafka-2321
Alternatively you can review and
[
https://issues.apache.org/jira/browse/KAFKA-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640363#comment-14640363
]
ASF GitHub Bot commented on KAFKA-2321:
---
GitHub user ijuma opened a pull request:
86 matches
Mail list logo
