[jira] [Created] (KAFKA-2483) Add support for batch/scheduled Copycat connectors

2015-08-27 Thread Ewen Cheslack-Postava (JIRA)
Ewen Cheslack-Postava created KAFKA-2483: Summary: Add support for batch/scheduled Copycat connectors Key: KAFKA-2483 URL: https://issues.apache.org/jira/browse/KAFKA-2483 Project: Kafka

[jira] [Created] (KAFKA-2484) Add schema projection utilities

2015-08-27 Thread Ewen Cheslack-Postava (JIRA)
Ewen Cheslack-Postava created KAFKA-2484: Summary: Add schema projection utilities Key: KAFKA-2484 URL: https://issues.apache.org/jira/browse/KAFKA-2484 Project: Kafka Issue Type:

[jira] [Updated] (KAFKA-2480) Handle non-CopycatExceptions from SinkTasks

2015-08-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2480: - Summary: Handle non-CopycatExceptions from SinkTasks (was: Handle

[jira] [Created] (KAFKA-2482) Allow copycat sink tasks to pause/resume consumption of specific topic partitions

2015-08-27 Thread Ewen Cheslack-Postava (JIRA)
Ewen Cheslack-Postava created KAFKA-2482: Summary: Allow copycat sink tasks to pause/resume consumption of specific topic partitions Key: KAFKA-2482 URL: https://issues.apache.org/jira/browse/KAFKA-2482

[jira] [Created] (KAFKA-2480) Handle non-CopycatExceptions from

2015-08-27 Thread Ewen Cheslack-Postava (JIRA)
Ewen Cheslack-Postava created KAFKA-2480: Summary: Handle non-CopycatExceptions from Key: KAFKA-2480 URL: https://issues.apache.org/jira/browse/KAFKA-2480 Project: Kafka Issue Type:

[jira] [Created] (KAFKA-2479) Add CopycatExceptions to indicate transient and permanent errors in a connector/task

2015-08-27 Thread Ewen Cheslack-Postava (JIRA)
Ewen Cheslack-Postava created KAFKA-2479: Summary: Add CopycatExceptions to indicate transient and permanent errors in a connector/task Key: KAFKA-2479 URL:

[jira] [Created] (KAFKA-2481) Allow copycat sinks to request periodic invocation of put even if no new data is available

2015-08-27 Thread Ewen Cheslack-Postava (JIRA)
Ewen Cheslack-Postava created KAFKA-2481: Summary: Allow copycat sinks to request periodic invocation of put even if no new data is available Key: KAFKA-2481 URL:

Kafka New Consumer Performance Test ThroughPut Degradation

2015-08-27 Thread Poorna Chandra Tejashvi Reddy
Hi, We have built the latest kafka from https://github.com/apache/kafka based on this commit id 436b7ddc386eb688ba0f12836710f5e4bcaa06c8 . We ran the performance test on a 3 node kafka cluster. There is a huge throughput degradation using the new-consumer compared to the regular consumer. Below

[GitHub] kafka pull request: KAFKA-2367: Add Copycat runtime data API.

2015-08-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/163 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Jenkins build is back to normal : Kafka-trunk #604

2015-08-27 Thread Apache Jenkins Server
See https://builds.apache.org/job/Kafka-trunk/604/changes

[jira] [Updated] (KAFKA-2367) Add Copycat runtime data API

2015-08-27 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2367: Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request

[jira] [Commented] (KAFKA-2367) Add Copycat runtime data API

2015-08-27 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717315#comment-14717315 ] ASF GitHub Bot commented on KAFKA-2367: --- Github user asfgit closed the pull request

Re: [DISCUSS] Client-side Assignment for New Consumer

2015-08-27 Thread Jay Kreps
Hey Joel, I really don't think we should do both. There are pros and cons but we should make a decision and work on operationalizing one approach. Much of really making something like this work is getting all the bugs out, getting monitoring in place, getting rigorous system tests in place.

[jira] [Updated] (KAFKA-2377) Add copycat system tests

2015-08-27 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2377: Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request

[jira] [Commented] (KAFKA-2377) Add copycat system tests

2015-08-27 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717506#comment-14717506 ] ASF GitHub Bot commented on KAFKA-2377: --- Github user asfgit closed the pull request

Re: [DISCUSS] Client-side Assignment for New Consumer

2015-08-27 Thread Joel Koshy
I actually feel these set of tests (whatever they may be) are somewhat irrelevant here. My main concern with the current client-side proposal (i.e., without Becket's follow-up suggestions) is that it makes a significant compromise to the original charter of the new consumer - i.e.,

[jira] [Commented] (KAFKA-2475) Reduce copycat configs to only specify a converter or serializer, not both

2015-08-27 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717551#comment-14717551 ] ASF GitHub Bot commented on KAFKA-2475: --- GitHub user ewencp opened a pull request:

[jira] [Updated] (KAFKA-2475) Reduce copycat configs to only specify a converter or serializer, not both

2015-08-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2475: - Reviewer: Gwen Shapira Status: Patch Available (was: Open) Reduce copycat

[GitHub] kafka pull request: KAFKA-2475: Make Copycat only have a Converter...

2015-08-27 Thread ewencp
GitHub user ewencp opened a pull request: https://github.com/apache/kafka/pull/172 KAFKA-2475: Make Copycat only have a Converter class instead of Serializer, Deserializer, and Converter. The Converter class now translates directly between byte[] and Copycat's data API instead

Re: Kafka New Consumer Performance Test ThroughPut Degradation

2015-08-27 Thread Ewen Cheslack-Postava
I don't think the commands are really equivalent despite just adding the --new-consumer flag. ConsumerPerformance uses a single thread when using the new consumer (it literally just allocates the consumer, loops until it's consumed enough, then exits), whereas the old consumer uses a bunch of

Re: Kafka New Consumer Performance Test ThroughPut Degradation

2015-08-27 Thread Jay Kreps
I think this is likely a regression. The two clients had more or less equivalent performance when we checked in the code (see my post on this earlier in the year). Looks like maybe we broke something up in the interim? On my laptop the new consumer perf seems to have dropped from about ~200MB/sec

[jira] [Commented] (KAFKA-1912) Create a simple request re-routing facility

2015-08-27 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14718092#comment-14718092 ] Ashish K Singh commented on KAFKA-1912: --- [~jkreps], [~guozhang] are we still

[jira] [Commented] (KAFKA-2485) Allow producer performance to take properties from a file via --producer.config command line parameter

2015-08-27 Thread Dong Lin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717963#comment-14717963 ] Dong Lin commented on KAFKA-2485: - [~ewencp] I didn't add this to the Java producer

Re: Kafka New Consumer Performance Test ThroughPut Degradation

2015-08-27 Thread Guozhang Wang
436b7ddc386eb688ba0f12836710f5e4bcaa06c8 is pretty recent and there could be some current consumer improvement patches that introduces some regression. I would suggest doing a binary search in the log from 3f8480ccfb011eb43da774737597c597f703e11b (maybe even earlier?) to do a quick check.

[jira] [Commented] (KAFKA-2471) Replicas Order and Leader out of sync

2015-08-27 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14718087#comment-14718087 ] Ashish K Singh commented on KAFKA-2471: --- Could you attach server.log Replicas

[jira] [Commented] (KAFKA-2485) Allow producer performance to take properties from a file via --consumer.config command line parameter

2015-08-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717952#comment-14717952 ] Ewen Cheslack-Postava commented on KAFKA-2485: -- [~lindong] This looks good

[jira] [Updated] (KAFKA-2485) Allow producer performance to take properties from a file via --producer.config command line parameter

2015-08-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2485: - Summary: Allow producer performance to take properties from a file via

[GitHub] kafka pull request: KAFKA-2468: SIGINT during Kafka server startup...

2015-08-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/167 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Updated] (KAFKA-2461) request logger no longer logs extra information in debug mode

2015-08-27 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-2461: -- Status: Patch Available (was: Open) request logger no longer logs extra information in debug

[jira] [Resolved] (KAFKA-2468) SIGINT during Kafka server startup can leave server deadlocked

2015-08-27 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-2468. -- Resolution: Fixed Fix Version/s: 0.8.3 Issue resolved by pull request 167

[jira] [Commented] (KAFKA-2468) SIGINT during Kafka server startup can leave server deadlocked

2015-08-27 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14718017#comment-14718017 ] ASF GitHub Bot commented on KAFKA-2468: --- Github user asfgit closed the pull request

[jira] [Commented] (KAFKA-2478) KafkaConsumer javadoc example seems wrong

2015-08-27 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716428#comment-14716428 ] Ismael Juma commented on KAFKA-2478: [~devstr], yes, you can submit a PR. Instructions

[GitHub] kafka pull request: KAFKA-2425; Copy latest docs to kafka repo doc...

2015-08-27 Thread omkreddy
GitHub user omkreddy opened a pull request: https://github.com/apache/kafka/pull/171 KAFKA-2425; Copy latest docs to kafka repo docs/ directory This PR copies the latest kafka docs to kafka repo docs directory. Here I have copied 0.8.3/ directory contents from svn website repo to

[jira] [Commented] (KAFKA-2425) Migrate website from SVN to Git

2015-08-27 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716436#comment-14716436 ] ASF GitHub Bot commented on KAFKA-2425: --- GitHub user omkreddy opened a pull request:

[jira] [Commented] (KAFKA-2425) Migrate website from SVN to Git

2015-08-27 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716460#comment-14716460 ] ASF GitHub Bot commented on KAFKA-2425: --- GitHub user omkreddy reopened a pull

[GitHub] kafka pull request: KAFKA-2425; Copy latest docs to kafka repo doc...

2015-08-27 Thread omkreddy
GitHub user omkreddy reopened a pull request: https://github.com/apache/kafka/pull/171 KAFKA-2425; Copy latest docs to kafka repo docs/ directory This PR copies the latest kafka docs to kafka repo docs directory. Here I have copied 0.8.3/ directory contents from svn website repo to

[GitHub] kafka pull request: KAFKA-2425; Copy latest docs to kafka repo doc...

2015-08-27 Thread omkreddy
Github user omkreddy closed the pull request at: https://github.com/apache/kafka/pull/171 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (KAFKA-2425) Migrate website from SVN to Git

2015-08-27 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716459#comment-14716459 ] ASF GitHub Bot commented on KAFKA-2425: --- Github user omkreddy closed the pull

[jira] [Commented] (KAFKA-2072) Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module

2015-08-27 Thread David Jacot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716229#comment-14716229 ] David Jacot commented on KAFKA-2072: [~ijuma] Do you also need rest of this one for

[jira] [Work started] (KAFKA-2074) Add UpdateMetadata request/response to o.a.k.common.requests and replace its use in core module

2015-08-27 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-2074 started by Ismael Juma. -- Add UpdateMetadata request/response to o.a.k.common.requests and replace its use

[jira] [Assigned] (KAFKA-2074) Add UpdateMetadata request/response to o.a.k.common.requests and replace its use in core module

2015-08-27 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reassigned KAFKA-2074: -- Assignee: Ismael Juma (was: Sriharsha Chintalapani) Add UpdateMetadata request/response to

[jira] [Commented] (KAFKA-2074) Add UpdateMetadata request/response to o.a.k.common.requests and replace its use in core module

2015-08-27 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716259#comment-14716259 ] Ismael Juma commented on KAFKA-2074: Doing this as part of KAFKA-2411, so assigning it

[jira] [Commented] (KAFKA-2072) Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module

2015-08-27 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716261#comment-14716261 ] Ismael Juma commented on KAFKA-2072: Migration of `KafkaAPIs` is not needed for

[jira] [Commented] (KAFKA-2478) KafkaConsumer javadoc example seems wrong

2015-08-27 Thread Dmitry Stratiychuk (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716378#comment-14716378 ] Dmitry Stratiychuk commented on KAFKA-2478: --- Can certainly do for the javadoc

[jira] [Work started] (KAFKA-2065) Add ControlledShutdown to org.apache.kafka.common.requests and replace current use in core module

2015-08-27 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-2065 started by Ismael Juma. -- Add ControlledShutdown to org.apache.kafka.common.requests and replace current

[jira] [Work started] (KAFKA-2067) Add LeaderAndISR request/response to org.apache.kafka.common.requests and replace usage in core module

2015-08-27 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-2067 started by Ismael Juma. -- Add LeaderAndISR request/response to org.apache.kafka.common.requests and replace

[jira] [Work started] (KAFKA-2485) Allow producer performance to take properties from a file via --consumer.config command line parameter

2015-08-27 Thread Dong Lin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-2485 started by Dong Lin. --- Allow producer performance to take properties from a file via --consumer.config command

[jira] [Updated] (KAFKA-2485) Allow producer performance to take properties from a file via --consumer.config command line parameter

2015-08-27 Thread Dong Lin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-2485: Status: Patch Available (was: In Progress) Allow producer performance to take properties from a file via

[jira] [Commented] (KAFKA-2485) Allow producer performance to take properties from a file via --consumer.config command line parameter

2015-08-27 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717815#comment-14717815 ] ASF GitHub Bot commented on KAFKA-2485: --- GitHub user lindong28 opened a pull

[GitHub] kafka pull request: MINOR: kafkatest add manifest

2015-08-27 Thread granders
GitHub user granders opened a pull request: https://github.com/apache/kafka/pull/173 MINOR: kafkatest add manifest This patch makes it possible to publish kafkatest (system test package) to pypi and use it as a library in other projects by: - including necessary static

[GitHub] kafka pull request: KAFKA-2485; Allow producer performance to take...

2015-08-27 Thread lindong28
GitHub user lindong28 opened a pull request: https://github.com/apache/kafka/pull/174 KAFKA-2485; Allow producer performance to take properties from a file… … via --consumer.config command line parameter You can merge this pull request into a Git repository by running: $