+1 on resolving some of these open issues separately. It is a pain to
maintain, review, discuss and commit these type of large patches. Typically
the set of improvements to really stabilize new features like this go well
beyond one large initial patch anyway.
On Thu, Aug 13, 2015 at 10:24 PM,
Copycat enables streaming data in and out of Kafka. Connector writers need
to define the serde of the data as it is different per system. Metadata
should be entirely hidden by the copycat framework and isn't something
users or connector implementors need to serialize differently as long as we
[
https://issues.apache.org/jira/browse/KAFKA-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14696551#comment-14696551
]
Onur Karaman commented on KAFKA-2397:
-
Sooo... ship it?!
leave group request
[
https://issues.apache.org/jira/browse/KAFKA-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14696602#comment-14696602
]
Ewen Cheslack-Postava commented on KAFKA-2397:
--
[~becket_qin] I was more
Yes, I think that makes sense. As I see it, the tradeoffs are:
1. Complexity - adding these APIs increases the number of things connector
developers need to implement just to get started. In a lot of cases, the
first version of a connector might literally only have some connection
string as a
Yes, I think the rename (other than being fun) will prevent some confusion
with existing coordinator.
On Aug 14, 2015 1:09 AM, Ewen Cheslack-Postava e...@confluent.io wrote:
For obvious reasons, I definitely won't object, but I will probably quickly
have a follow-up patch to add a very basic
Will be nice to include Kafka-2308 and fix two critical snappy issues in
the maintenance release.
Gwen
On Aug 14, 2015 6:16 AM, Grant Henke ghe...@cloudera.com wrote:
Just to clarify. Will KAFKA-2189 be the only patch in the release?
On Fri, Aug 14, 2015 at 7:35 AM, Manikumar Reddy
[
https://issues.apache.org/jira/browse/KAFKA-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697002#comment-14697002
]
Flavio Junqueira commented on KAFKA-1387:
-
I'm actually really sorry that this
For obvious reasons, I definitely won't object, but I will probably quickly
have a follow-up patch to add a very basic system test for Copycat. I was
trying to get it in with this patch since I think it is a good idea to
encourage people to write system tests with major additions like this, but
I
On Thu, Aug 13, 2015 at 11:07 PM, Neha Narkhede n...@confluent.io wrote:
Becket,
As you say, the metadata hash addresses the concern you originally raised
about large topic subscriptions. Can you please list other problems you are
raising more clearly? It is more helpful to know problems
The JDBC connector I started implementing just handles this manually, and
isn't much code (and could be made into a simple utility):
https://github.com/confluentinc/copycat-jdbc/blob/master/src/main/java/io/confluent/copycat/jdbc/JdbcSourceTask.java#L152
Given the current APIs, sources can just
I'm not sure the existing discussion is clear about how the format of
offset data is decided. One possibility is that we choose one fixed format
and that is what we use internally to store offsets no matter what
serializer you choose. This would be similar to how the __offsets topic is
currently
[
https://issues.apache.org/jira/browse/KAFKA-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697189#comment-14697189
]
Andrew Olson commented on KAFKA-2196:
-
Doesn't this [1] code also need to be updated?
[
https://issues.apache.org/jira/browse/KAFKA-2434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Olson reassigned KAFKA-2434:
---
Assignee: Andrew Olson
remove roundrobin identical topic constraint in consumer coordinator
Andrew Olson created KAFKA-2434:
---
Summary: remove roundrobin identical topic constraint in consumer
coordinator (old API)
Key: KAFKA-2434
URL: https://issues.apache.org/jira/browse/KAFKA-2434
Project:
[
https://issues.apache.org/jira/browse/KAFKA-2434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697192#comment-14697192
]
Andrew Olson commented on KAFKA-2434:
-
I will submit a patch for this change shortly.
[
https://issues.apache.org/jira/browse/KAFKA-1477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sriharsha Chintalapani resolved KAFKA-1477.
---
Resolution: Duplicate
add authentication layer and initial JKS x509
Andrew Olson created KAFKA-2435:
---
Summary: More optimally balanced partition assignment strategy
Key: KAFKA-2435
URL: https://issues.apache.org/jira/browse/KAFKA-2435
Project: Kafka
Issue
[
https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697567#comment-14697567
]
Ismael Juma commented on KAFKA-2367:
I wasn't aware that it was so difficult to
[
https://issues.apache.org/jira/browse/KAFKA-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697588#comment-14697588
]
Parth Brahmbhatt commented on KAFKA-1695:
-
[~fpj] When you have time can you
[
https://issues.apache.org/jira/browse/KAFKA-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-2402:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Fixed as part of KAFKA-2406
On Fri, Aug 14, 2015 at 10:59 AM, Jiangjie Qin j...@linkedin.com.invalid
wrote:
Neha and Ewen,
About the metadata change frequency. I guess it really depends on how
frequent the metadata change might occur. If we run Kafka as a service, I
can see that happens from time to time. As I can
[
https://issues.apache.org/jira/browse/KAFKA-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697662#comment-14697662
]
Andrew Olson commented on KAFKA-2172:
-
[~jjkoshy] I've implemented a new assignment
[
https://issues.apache.org/jira/browse/KAFKA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697692#comment-14697692
]
Jason Gustafson commented on KAFKA-2388:
[~onurkaraman] I submitted a WIP patch
[
https://issues.apache.org/jira/browse/KAFKA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gustafson reopened KAFKA-1782:
It looks like this patch broke some of the tests (e.g. ConsumerTest). We
probably need to go
[
https://issues.apache.org/jira/browse/KAFKA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697688#comment-14697688
]
ASF GitHub Bot commented on KAFKA-2388:
---
GitHub user hachikuji opened a pull
GitHub user hachikuji opened a pull request:
https://github.com/apache/kafka/pull/139
KAFKA-2388 [WIP]; refactor KafkaConsumer subscribe API
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hachikuji/kafka KAFKA-2388
[
https://issues.apache.org/jira/browse/KAFKA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gustafson reassigned KAFKA-2388:
--
Assignee: Jason Gustafson (was: Onur Karaman)
subscribe(topic)/unsubscribe(topic)
Github user becketqin closed the pull request at:
https://github.com/apache/kafka/pull/108
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
I thought batch was dead? :-)
Yeah I think this would be really useful. Kafka kind of allows you to unify
batch and streams since you produce or consume your stream on your own
schedule so you would want the ingress/egress to work the same.
Ewen, rather than sleeping, I think the use case is
GitHub user jaikiran opened a pull request:
https://github.com/apache/kafka/pull/138
Log the real exception which triggered a reconnect
The commit here improves the logging in SimpleConsumer to log the real
reason why a reconnect was attempted. Relates to
[
https://issues.apache.org/jira/browse/KAFKA-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697401#comment-14697401
]
ASF GitHub Bot commented on KAFKA-2221:
---
GitHub user jaikiran opened a pull request:
Neha and Ewen,
About the metadata change frequency. I guess it really depends on how
frequent the metadata change might occur. If we run Kafka as a service, I
can see that happens from time to time. As I can imagine people will create
some topic, test and maybe delete the topic in some automated
On Fri, Aug 14, 2015 at 10:57 AM, Jay Kreps j...@confluent.io wrote:
I thought batch was dead? :-)
Yeah I think this would be really useful. Kafka kind of allows you to unify
batch and streams since you produce or consume your stream on your own
schedule so you would want the ingress/egress
Ewen, I have no problem with adding an explicit version. As you mentioned,
I was thinking that each protocol would manage that in its name field, but
the explicit field is probably clearer for implementors.
Becket, I was also a little concerned that the consumers would have some
difficulty
[
https://issues.apache.org/jira/browse/KAFKA-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697427#comment-14697427
]
Abhishek Nigam commented on KAFKA-1387:
---
Thanks a lot for digging into this. Not
[
https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697499#comment-14697499
]
Jay Kreps commented on KAFKA-2367:
--
I'm more negative on using Avro. Here's my thinking.
[
https://issues.apache.org/jira/browse/KAFKA-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697520#comment-14697520
]
ASF GitHub Bot commented on KAFKA-2402:
---
Github user becketqin closed the pull
[
https://issues.apache.org/jira/browse/KAFKA-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697907#comment-14697907
]
Guozhang Wang commented on KAFKA-1387:
--
Thanks [~fpj], this is very helpful.
Just to
[
https://issues.apache.org/jira/browse/KAFKA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on KAFKA-2388 started by Jason Gustafson.
--
subscribe(topic)/unsubscribe(topic) should either take a callback to allow
GitHub user ewencp opened a pull request:
https://github.com/apache/kafka/pull/140
KAFKA-1782: Follow up - add missing @Test annotations.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ewencp/kafka kafka-1782-followup
[
https://issues.apache.org/jira/browse/KAFKA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697899#comment-14697899
]
ASF GitHub Bot commented on KAFKA-1782:
---
GitHub user ewencp opened a pull request:
[
https://issues.apache.org/jira/browse/KAFKA-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697920#comment-14697920
]
ASF GitHub Bot commented on KAFKA-2366:
---
Github user asfgit closed the pull request
[
https://issues.apache.org/jira/browse/KAFKA-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697923#comment-14697923
]
Gwen Shapira commented on KAFKA-2366:
-
Merged to trunk. Thank you for your
[
https://issues.apache.org/jira/browse/KAFKA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewen Cheslack-Postava updated KAFKA-1782:
-
Status: Patch Available (was: Reopened)
Junit3 Misusage
---
[
https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697942#comment-14697942
]
Gwen Shapira commented on KAFKA-2367:
-
The fact that Avro brings in bazillion things
So if you're just subscribing to one or a couple of topics, why not just
compute the hash by filtering out everything but the topics you are
subscribed to? The problem there is if you ever add/remove subscriptions
and want to support rolling upgrades. If the group was subscribed to topic
A,
[
https://issues.apache.org/jira/browse/KAFKA-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-2366:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Issue resolved by pull request
Github user asfgit closed the pull request at:
https://github.com/apache/kafka/pull/99
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
On Aug. 14, 2015, 1:57 a.m., Jun Rao wrote:
clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java,
lines 69-77
https://reviews.apache.org/r/33049/diff/25/?file=1039169#file1039169line69
This is probably not the right place to throw QuotaViolationException.
Jason,
The way this is handled today (in the old consumer) is by grouping several
rebalance events into one rebalance operation (similar to how we handle log
flushes using group commit). The idea is to batch several rebalance
events while one rebalance operation is in progress.
This is naturally
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/#review95496
---
Ship it!
Minor edits that I will take care of on check-in.
[
https://issues.apache.org/jira/browse/KAFKA-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698000#comment-14698000
]
Aditya A Auradkar commented on KAFKA-2084:
--
Updated reviewboard
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---
(Updated Aug. 15, 2015, 12:43 a.m.)
Review request for kafka, Joel Koshy and
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---
(Updated Aug. 15, 2015, 12:43 a.m.)
Review request for kafka, Joel Koshy and
I think metadata churn is an interesting problem and it would be nice if we
had some approach to deal with it. I wonder if it would be sufficient to
have a setting on the client (or the server if we use centralized
assignment) which dampens the rate of rebalancing. Basically the effect
would be to
Yeah, I missed the option to match serialization of offsets to data, which
solves the configuration overhead.
It still doesn't give us the ability to easily evolve the metadata messages
or to use them in monitoring tools.
And I am still not clear of the benefits of using user-defined
[
https://issues.apache.org/jira/browse/KAFKA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-2433:
Resolution: Fixed
Fix Version/s: 0.8.3
Status: Resolved (was: Patch Available)
The ignore new configs plan is good, IMO. I just don't know if its
feasible in current Copycat:
I'm not sure CC can ignore configuration on the connector behalf. Also,
this is something we will want to log very clearly and I'm not sure this is
doable outside the connector.
Regarding complex
Ewen,
I agree that if there is a churn in metadata, the consumers need several
rounds of rebalances to succeed. The difference I am thinking is that with
coordinator as single source of truth, we can let the consumer finish one
round of rebalance, work for a while and start the next round of
[
https://issues.apache.org/jira/browse/KAFKA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-1782:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Issue resolved by pull request
[
https://issues.apache.org/jira/browse/KAFKA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698109#comment-14698109
]
ASF GitHub Bot commented on KAFKA-1782:
---
Github user asfgit closed the pull request
Github user asfgit closed the pull request at:
https://github.com/apache/kafka/pull/140
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
[
https://issues.apache.org/jira/browse/KAFKA-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aditya A Auradkar updated KAFKA-2084:
-
Attachment: KAFKA-2084_2015-08-14_17:43:00.patch
byte rate metrics per client ID
On Aug. 14, 2015, 1:57 a.m., Jun Rao wrote:
clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java,
lines 69-77
https://reviews.apache.org/r/33049/diff/25/?file=1039169#file1039169line69
This is probably not the right place to throw QuotaViolationException.
[
https://issues.apache.org/jira/browse/KAFKA-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joel Koshy resolved KAFKA-2084.
---
Resolution: Fixed
Thanks for the patches - committed to trunk.
byte rate metrics per client ID
For sink side:
I'm a bit more comfortable with batch mode than with run once and it
will do something every hour because the former puts scheduling firmly in
the user hands (and their cron), the latter means that connector developers
need to figure out schedules.
For source side:
I'm not
On Fri, Aug 14, 2015 at 6:35 PM, Gwen Shapira g...@confluent.io wrote:
Yeah, I missed the option to match serialization of offsets to data, which
solves the configuration overhead.
It still doesn't give us the ability to easily evolve the metadata messages
or to use them in monitoring tools.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37480/
---
Review request for kafka.
Bugs: KAFKA-2434
[
https://issues.apache.org/jira/browse/KAFKA-2434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697264#comment-14697264
]
Andrew Olson commented on KAFKA-2434:
-
Created reviewboard
[
https://issues.apache.org/jira/browse/KAFKA-2434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Olson updated KAFKA-2434:
Attachment: KAFKA-2434.patch
remove roundrobin identical topic constraint in consumer coordinator
[
https://issues.apache.org/jira/browse/KAFKA-2434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Olson updated KAFKA-2434:
Status: Patch Available (was: Open)
remove roundrobin identical topic constraint in consumer
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37481/
---
Review request for kafka.
Bugs: KAFKA-2435
[
https://issues.apache.org/jira/browse/KAFKA-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697266#comment-14697266
]
Andrew Olson commented on KAFKA-2435:
-
Created reviewboard
[
https://issues.apache.org/jira/browse/KAFKA-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Olson updated KAFKA-2435:
Status: Patch Available (was: Open)
More optimally balanced partition assignment strategy
[
https://issues.apache.org/jira/browse/KAFKA-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Olson updated KAFKA-2435:
Attachment: KAFKA-2435.patch
More optimally balanced partition assignment strategy
[
https://issues.apache.org/jira/browse/KAFKA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Powis updated KAFKA-2433:
-
Attachment: documentation-update.diff
updated fixed patch.
Remove documentation on dead
[
https://issues.apache.org/jira/browse/KAFKA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Powis updated KAFKA-2433:
-
Attachment: (was: documentation-update.diff)
Remove documentation on dead configuration
[
https://issues.apache.org/jira/browse/KAFKA-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flavio Junqueira updated KAFKA-1387:
Reviewer: Flavio Junqueira (was: Jun Rao)
Kafka getting stuck creating ephemeral node it
[
https://issues.apache.org/jira/browse/KAFKA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14696934#comment-14696934
]
Stephen Powis commented on KAFKA-2433:
--
You're totally right. Uploading a patch over
Hi, Everyone,
Since the release of Kafka 0.8.2.1, a number of people have reported an
issue with snappy compression (
https://issues.apache.org/jira/browse/KAFKA-2189). Basically, if they use
snappy in 0.8.2.1, they will experience a 2-3X space increase. The issue
has since been fixed in trunk
I think this is a good idea as the change is minimal on our side and it has
been tested in production for some time by the reporter.
Best,
Ismael
On Fri, Aug 14, 2015 at 1:15 PM, Jun Rao j...@confluent.io wrote:
Hi, Everyone,
Since the release of Kafka 0.8.2.1, a number of people have
+1 for 0.8.2.2 release
On Fri, Aug 14, 2015 at 5:49 PM, Ismael Juma ism...@juma.me.uk wrote:
I think this is a good idea as the change is minimal on our side and it has
been tested in production for some time by the reporter.
Best,
Ismael
On Fri, Aug 14, 2015 at 1:15 PM, Jun Rao
Just to clarify. Will KAFKA-2189 be the only patch in the release?
On Fri, Aug 14, 2015 at 7:35 AM, Manikumar Reddy ku...@nmsworks.co.in
wrote:
+1 for 0.8.2.2 release
On Fri, Aug 14, 2015 at 5:49 PM, Ismael Juma ism...@juma.me.uk wrote:
I think this is a good idea as the change is minimal
[
https://issues.apache.org/jira/browse/KAFKA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Powis updated KAFKA-2433:
-
Affects Version/s: (was: 0.8.2.1)
(was: 0.8.1.2)
85 matches
Mail list logo
