[GitHub] kafka-site pull request #34: Fix typo on introduction page

2016-12-01 Thread ashishg-qburst
GitHub user ashishg-qburst opened a pull request: https://github.com/apache/kafka-site/pull/34 Fix typo on introduction page You can merge this pull request into a Git repository by running: $ git pull https://github.com/ashishg-qburst/kafka-site intro Alternatively you can

[jira] [Commented] (KAFKA-4391) On Windows, Kafka server stops with uncaught exception after coming back from sleep

2016-12-01 Thread Yiquan Zhou (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712034#comment-15712034 ] Yiquan Zhou commented on KAFKA-4391: [~huxi_2b] Can you think of any possible workaround to this

[GitHub] kafka pull request #2194: KAFKA-4443: Minor comment clean-up

2016-12-01 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2194 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (KAFKA-4443) Controller should send UpdateMetadataRequest prior to LeaderAndIsrRequest during failover

2016-12-01 Thread Alexandre Vermeerbergen (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712243#comment-15712243 ] Alexandre Vermeerbergen commented on KAFKA-4443: Hello, Could this fix be back-ported to

[jira] [Commented] (KAFKA-4443) Controller should send UpdateMetadataRequest prior to LeaderAndIsrRequest during failover

2016-12-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712288#comment-15712288 ] ASF GitHub Bot commented on KAFKA-4443: --- Github user asfgit closed the pull request at:

[jira] [Created] (KAFKA-4474) Poor kafka-streams throughput

2016-12-01 Thread Juan Chorro (JIRA)
Juan Chorro created KAFKA-4474: -- Summary: Poor kafka-streams throughput Key: KAFKA-4474 URL: https://issues.apache.org/jira/browse/KAFKA-4474 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

2016-12-01 Thread Apurva Mehta
Hi Daniel, That is a very good point. You are correct in saying that one does not need a transaction coordinator to get idempotent semantics. There are, however, three reasons why we chose this route: 1. The request to find a transaction coordinator is exactly the same as the request

Re: [VOTE] KIP-84: Support SASL SCRAM mechanisms

2016-12-01 Thread Ismael Juma
Hi Rajini, Sorry for the delay. For some reason, both of your replies (for this and KIP-85) were marked as spam by Gmail. Comments inline. On Mon, Nov 28, 2016 at 3:47 PM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote: > > 1. I think you had asked earlier for SCRAM-SHA-1 to be removed

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-12-01 Thread radai
"For use cases within an organization, one could always use other approaches such as company-wise containers" this is what linkedin has traditionally done but there are now cases (read - topics) where this is not acceptable. this makes headers useful even within single orgs for cases where

Build failed in Jenkins: kafka-trunk-jdk8 #1071

2016-12-01 Thread Apache Jenkins Server
See Changes: [ismael] KAFKA-4443; Minor comment clean-up -- [...truncated 7978 lines...] kafka.integration.SaslPlaintextTopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack STARTED

[jira] [Created] (KAFKA-4475) Poor kafka-streams throughput

2016-12-01 Thread Juan Chorro (JIRA)
Juan Chorro created KAFKA-4475: -- Summary: Poor kafka-streams throughput Key: KAFKA-4475 URL: https://issues.apache.org/jira/browse/KAFKA-4475 Project: Kafka Issue Type: Bug

Kafka JDBC client Phoenix support

2016-12-01 Thread Songo Songo
Hello community, Do you know if there is some activity to support Apache Phoenix in Kafka Connect JDBC ? Thanks Alex

Re: [VOTE] KIP-85: Dynamic JAAS configuration for Kafka clients

2016-12-01 Thread Ismael Juma
Thanks for explaining your reasoning, Rajini. I do agree with all of it and that's why I voted +1. :) The reason for my comment was to highlight some of the areas that can be improved in case someone has the time and interest. The Kerberos situation is the obvious one, but one could also imagine

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

2016-12-01 Thread Apurva Mehta
Hi Ismael, That is a good suggestion. We did not plan to move the design to a wiki, but I think it is valuable to move at least the message format and RPC changes to the wiki. We shall do so once the design is close to final so that we do not have to edit multiple places as we iterate. Thanks,

[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time

2016-12-01 Thread Harald Kirsch (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712362#comment-15712362 ] Harald Kirsch commented on KAFKA-1194: -- [~soumyajitsahu] Here are the server configs as logged:

[GitHub] kafka pull request #2199: HOTFIX: Temporary suspension of 2 tests

2016-12-01 Thread enothereska
GitHub user enothereska opened a pull request: https://github.com/apache/kafka/pull/2199 HOTFIX: Temporary suspension of 2 tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/enothereska/kafka hotfix-streams-test-reset-ignore

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-12-01 Thread Gwen Shapira
On Thu, Dec 1, 2016 at 10:24 AM, radai wrote: > "For use cases within an organization, one could always use other > approaches such as company-wise containers" > this is what linkedin has traditionally done but there are now cases (read > - topics) where this is not

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-12-01 Thread Ignacio Solis
This is a very good question. Do we try to enable third party "vendors" by building a platform? I'm assuming the other (main) alternative is to let everybody implement their own. Disadvantages of a platform: - can potentially decrease performance if done incorrectly - requires the "platform" to

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

2016-12-01 Thread Guozhang Wang
@Henry Cai, I am working on a separate KIP on Streams to leverage this KIP to have exactly-once processing semantics (note the exactly-once processing is a bit different from exactly-once delivery semantics), which should cover your question. The short answer is that writing the changelog

[jira] [Commented] (KAFKA-4474) Poor kafka-streams throughput

2016-12-01 Thread Eno Thereska (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712966#comment-15712966 ] Eno Thereska commented on KAFKA-4474: - [~jjchorrobe] Thanks for reporting. A couple of questions: -

Re: Kafka JDBC client Phoenix support

2016-12-01 Thread Gwen Shapira
I don't think we tried it yet. If Pheonix supports JDBC, feel free to test and report what you find. Maybe it "just works"? On Thu, Dec 1, 2016 at 8:19 AM, Songo Songo wrote: > Hello community, > > Do you know if there is some activity to support Apache Phoenix in Kafka >

[jira] [Resolved] (KAFKA-4475) Poor kafka-streams throughput

2016-12-01 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-4475. -- Resolution: Duplicate Duplicate of KAFKA-4474. > Poor kafka-streams throughput >

[jira] [Updated] (KAFKA-4474) Poor kafka-streams throughput

2016-12-01 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-4474: - Assignee: Eno Thereska > Poor kafka-streams throughput > - > >

[jira] [Comment Edited] (KAFKA-4474) Poor kafka-streams throughput

2016-12-01 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712821#comment-15712821 ] Guozhang Wang edited comment on KAFKA-4474 at 12/1/16 7:19 PM: ---

[jira] [Commented] (KAFKA-4474) Poor kafka-streams throughput

2016-12-01 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712821#comment-15712821 ] Guozhang Wang commented on KAFKA-4474: -- [~enothereska] Could you take a look and see if there is any

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

2016-12-01 Thread Henry Cai
I see, having both topic data and changelog in kafka can make the kafka transaction atomic. But in the case of streaming, the RocksDB is an external data source during applying changelog to RocksDB. In terms of rollback and resume/re-apply kafka transaction, how do we make sure the RocksDB

[jira] [Updated] (KAFKA-4461) When using ProcessorTopologyTestDriver, the combination of map and .groupByKey does not produce any result

2016-12-01 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-4461: - Labels: newbie (was: ) > When using ProcessorTopologyTestDriver, the combination of map and >

[jira] [Commented] (KAFKA-4454) Authorizer should also include the Principal generated by the PrincipalBuilder.

2016-12-01 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712970#comment-15712970 ] Ismael Juma commented on KAFKA-4454: [~mgharat], thanks. That could work. Do you have some examples of

Build failed in Jenkins: kafka-trunk-jdk7 #1722

2016-12-01 Thread Apache Jenkins Server
See Changes: [ismael] KAFKA-4443; Minor comment clean-up -- [...truncated 14222 lines...] org.apache.kafka.streams.processor.TopologyBuilderTest > shouldNotAddNullStateStoreSupplier PASSED

[jira] [Commented] (KAFKA-4473) KafkaStreams does *not* guarantee at-least-once delivery

2016-12-01 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713038#comment-15713038 ] Guozhang Wang commented on KAFKA-4473: -- [~Thomas Schulz] Thanks for reporting this issue. I have a

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-12-01 Thread Todd Palino
So, Gwen, to your question (even though I’m not a committer)... I have always been a strong supporter of introducing the concept of an envelope to messages, which headers accomplishes. The message key is already an example of a piece of envelope information. By providing a means to do this within

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-12-01 Thread Todd Palino
> > C. per message encryption > One drawback of this approach is that this significantly reduce the > effectiveness of compression, which happens on a set of serialized > messages. An alternative is to enable SSL for wire encryption and rely on > the storage system (e.g. LUKS) for at rest

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-12-01 Thread Gwen Shapira
I know why people who support headers support them, and I've seen what the discussion is like. This is why I'm asking people who are against headers (especially committers) what will make them change their mind - so we can get this part over one way or another. If I sound frustrated it is not at

[jira] [Commented] (KAFKA-4161) Decouple flush and offset commits

2016-12-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713356#comment-15713356 ] ASF GitHub Bot commented on KAFKA-4161: --- Github user asfgit closed the pull request at:

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-12-01 Thread Gwen Shapira
I'm pretty satisfied with the current workarounds (Avro container format), so I'm not too excited about the extra work required to do headers in Kafka. I absolutely don't mind it if you do it... I think the Apache convention for "good idea, but not willing to put any work toward it" is +0.5?

[jira] [Resolved] (KAFKA-3008) Connect should parallelize task start/stop

2016-12-01 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-3008. -- Resolution: Fixed Fix Version/s: 0.10.2.0 Issue resolved by pull request

[GitHub] kafka pull request #1788: KAFKA-3008: Parallel start and stop of connectors ...

2016-12-01 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1788 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Created] (KAFKA-4476) Kafka Streams gets stuck if metadata is missing

2016-12-01 Thread Matthias J. Sax (JIRA)
Matthias J. Sax created KAFKA-4476: -- Summary: Kafka Streams gets stuck if metadata is missing Key: KAFKA-4476 URL: https://issues.apache.org/jira/browse/KAFKA-4476 Project: Kafka Issue

[jira] [Commented] (KAFKA-3008) Connect should parallelize task start/stop

2016-12-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713303#comment-15713303 ] ASF GitHub Bot commented on KAFKA-3008: --- Github user asfgit closed the pull request at:

[GitHub] kafka pull request #2139: KAFKA-4161: KIP-89: Allow sink connectors to decou...

2016-12-01 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2139 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-4161) Decouple flush and offset commits

2016-12-01 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-4161. -- Resolution: Fixed Fix Version/s: 0.10.2.0 Issue resolved by pull request

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-12-01 Thread Todd Palino
Well I guess my question for you, then, is what is holding you back from full support for headers? What’s the bit that you’re missing that has you under a full +1? -Todd On Thu, Dec 1, 2016 at 1:59 PM, Gwen Shapira wrote: > I know why people who support headers support

[jira] [Commented] (KAFKA-4472) offsetRetentionMs miscalculated in GroupCoordinator

2016-12-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713380#comment-15713380 ] ASF GitHub Bot commented on KAFKA-4472: --- GitHub user kichristensen opened a pull request:

[jira] [Commented] (KAFKA-4306) Connect workers won't shut down if brokers are not available

2016-12-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713416#comment-15713416 ] ASF GitHub Bot commented on KAFKA-4306: --- GitHub user kkonstantine opened a pull request:

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-12-01 Thread Sean McCauliff
Hi Jun, It seems like the issue you are bringing up is "are headers really needed" rather than "are headers useful"? Internally LinkedIn has a large, diverse user base for Kafka. We now have enough diversity where the engineering costs of putting the metadata for messages into the message value

[jira] [Work started] (KAFKA-4476) Kafka Streams gets stuck if metadata is missing

2016-12-01 Thread Matthias J. Sax (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-4476 started by Matthias J. Sax. -- > Kafka Streams gets stuck if metadata is missing >

[jira] [Assigned] (KAFKA-4472) offsetRetentionMs miscalculated in GroupCoordinator

2016-12-01 Thread Kim Christensen (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kim Christensen reassigned KAFKA-4472: -- Assignee: Kim Christensen > offsetRetentionMs miscalculated in GroupCoordinator >

[jira] [Commented] (KAFKA-4454) Authorizer should also include the Principal generated by the PrincipalBuilder.

2016-12-01 Thread Mayuresh Gharat (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713402#comment-15713402 ] Mayuresh Gharat commented on KAFKA-4454: [~ijuma] Thanks for pointing me to the patch. The idea of

[GitHub] kafka pull request #2201: KAFKA-4306: Shutdown distributed herder with a tim...

2016-12-01 Thread kkonstantine
GitHub user kkonstantine opened a pull request: https://github.com/apache/kafka/pull/2201 KAFKA-4306: Shutdown distributed herder with a timeout. Resolves KAFKA-4306: Connect workers won't shut down if brokers are not available KAFKA-4154: Kafka Connect fails to

[GitHub] kafka pull request #2200: KAFKA-4472: offsetRetentionMs miscalculated in Gro...

2016-12-01 Thread kichristensen
GitHub user kichristensen opened a pull request: https://github.com/apache/kafka/pull/2200 KAFKA-4472: offsetRetentionMs miscalculated in GroupCoordinator Fix possible integer overflow You can merge this pull request into a Git repository by running: $ git pull

[jira] [Commented] (KAFKA-4399) Deadlock between cleanupGroupMetadata and offset commit

2016-12-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713941#comment-15713941 ] ASF GitHub Bot commented on KAFKA-4399: --- Github user asfgit closed the pull request at:

[GitHub] kafka pull request #2125: KAFKA-4399; deadlock cleanupGroupMetadata and offs...

2016-12-01 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2125 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

2016-12-01 Thread Jay Kreps
Looks great! A few questions: 1. What is the relationship between transaction.app.id and the existing config application.id in streams? 2. The initTransactions() call is a little annoying. Can we get rid of that and call it automatically if you set a transaction.app.id when we do

[jira] [Updated] (KAFKA-4399) Deadlock between cleanupGroupMetadata and offset commit

2016-12-01 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-4399: --- Resolution: Fixed Fix Version/s: 0.10.2.0 Status: Resolved (was: Patch

Re: [DISCUSS] KIP-93: Improve invalid timestamp handling in Kafka Streams

2016-12-01 Thread Guozhang Wang
You mean "it is a backward incompatible change" right? On Wed, Nov 30, 2016 at 4:28 PM, Matthias J. Sax wrote: > Thanks for this clarification (and your +1) > > I completely agree and just want to add my thoughts: > > 1. Yes, it is a backward compatible change but as I

[GitHub] kafka pull request #2179: MINOR: Fix typos in KafkaConsumer docs

2016-12-01 Thread jeffwidman
Github user jeffwidman closed the pull request at: https://github.com/apache/kafka/pull/2179 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: A strange controller log in Kafka 0.9.0.1

2016-12-01 Thread Json Tu
Hi, Can someone else help to review the pr in jira: https://issues.apache.org/jira/browse/KAFKA-4447 . > 在 2016年11月23日,下午11:28,Json Tu 写道: > > Hi, > We have a cluster of kafka 0.9.0.1 with 3 nodes, and we found

Re: [DISCUSS] 0.10.1.1 Plan

2016-12-01 Thread Sean McCauliff
Well I would like KAFKA-4250 (make ProducerRecord and ConsumerRecord extensible) in the 0.10.1 branch if is not a big deal. They are just dumb structs. But they are final so no extensibility is possible. Sean On Tue, Nov 29, 2016 at 5:32 PM, Ignacio Solis wrote: > I don't

Build failed in Jenkins: kafka-trunk-jdk8 #1072

2016-12-01 Thread Apache Jenkins Server
See Changes: [me] KAFKA-3008: Parallel start and stop of connectors and tasks in Connect [me] KAFKA-4161: KIP-89: Allow sink connectors to decouple flush and offset -- [...truncated 12182

Build failed in Jenkins: kafka-trunk-jdk7 #1723

2016-12-01 Thread Apache Jenkins Server
See Changes: [me] KAFKA-3008: Parallel start and stop of connectors and tasks in Connect [me] KAFKA-4161: KIP-89: Allow sink connectors to decouple flush and offset -- [...truncated 14426

[GitHub] kafka pull request #2185: MINOR: added logging to debug test

2016-12-01 Thread mjsax
Github user mjsax closed the pull request at: https://github.com/apache/kafka/pull/2185 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-12-01 Thread Gwen Shapira
Based on your last sentence, consider me convinced :) I get why headers are critical for Mirroring (you need tags to prevent loops and sometimes to route messages to the correct destination). But why do you need headers to audit? We are auditing by producing counts to a side topic (and I was

Kafka SNAPSHOT artifact repositories.

2016-12-01 Thread Sean McCauliff
Is there an artifact repository where to-be-released versions of Kafka are published? There appears to be one at http://repository.apache.org/snapshots/ , but I'm not seeing anything published there after 0.8.2. Thanks! Sean

Re: Kafka SNAPSHOT artifact repositories.

2016-12-01 Thread Gwen Shapira
We are not. Do you mean nightly snapshots? We'll need to set this up in Jenkins so it will run as part of the nightly build (with the hope that build will pass most of the time). If you can figure out what's the magic we need to publish snapshots into that repo, I'll be happy to do whatever

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-12-01 Thread Todd Palino
Got it. As an ops guy, I'm not very happy with the workaround. Avro means that I have to be concerned with the format of the messages in order to run the infrastructure (audit, mirroring, etc.). That means that I have to handle the schemas, and I have to enforce rules about good formats. This is

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

2016-12-01 Thread Jason Gustafson
Hey Jay, Thanks for the questions! Let me take a couple of them. 2. The initTransactions() call is a little annoying. Can we get rid of >that and call it automatically if you set a transaction.app.id when we >do the first message send as we do with metadata? Arguably we should > have >

[GitHub] kafka pull request #2198: HOTFIX: Forgot one transition

2016-12-01 Thread enothereska
GitHub user enothereska opened a pull request: https://github.com/apache/kafka/pull/2198 HOTFIX: Forgot one transition You can merge this pull request into a Git repository by running: $ git pull https://github.com/enothereska/kafka hotfix-stream-states Alternatively you can

[jira] [Created] (KAFKA-4473) KafkaStreams does *not* guarantee at-least-once delivery

2016-12-01 Thread Thomas Schulz (JIRA)
Thomas Schulz created KAFKA-4473: Summary: KafkaStreams does *not* guarantee at-least-once delivery Key: KAFKA-4473 URL: https://issues.apache.org/jira/browse/KAFKA-4473 Project: Kafka Issue

Build failed in Jenkins: kafka-trunk-jdk8 #1070

2016-12-01 Thread Apache Jenkins Server
See Changes: [wangguoz] KAFKA-3637: Added initial states -- [...truncated 14449 lines...] org.apache.kafka.streams.KafkaStreamsTest > shouldReturnFalseOnCloseWhenThreadsHaventTerminated

[jira] [Commented] (KAFKA-4405) Kafka consumer improperly send prefetch request

2016-12-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711450#comment-15711450 ] ASF GitHub Bot commented on KAFKA-4405: --- Github user enothereska closed the pull request at:

[GitHub] kafka pull request #2193: KAFKA-4405: Check max.poll.records before prefetch...

2016-12-01 Thread enothereska
Github user enothereska closed the pull request at: https://github.com/apache/kafka/pull/2193 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] kafka pull request #2193: KAFKA-4405: Check max.poll.records before prefetch...

2016-12-01 Thread enothereska
GitHub user enothereska reopened a pull request: https://github.com/apache/kafka/pull/2193 KAFKA-4405: Check max.poll.records before prefetching You can merge this pull request into a Git repository by running: $ git pull https://github.com/enothereska/kafka

[jira] [Commented] (KAFKA-4405) Kafka consumer improperly send prefetch request

2016-12-01 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711475#comment-15711475 ] ASF GitHub Bot commented on KAFKA-4405: --- GitHub user enothereska reopened a pull request:

[jira] [Assigned] (KAFKA-4405) Kafka consumer improperly send prefetch request

2016-12-01 Thread Eno Thereska (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eno Thereska reassigned KAFKA-4405: --- Assignee: Eno Thereska > Kafka consumer improperly send prefetch request >

[jira] [Commented] (KAFKA-4405) Kafka consumer improperly send prefetch request

2016-12-01 Thread Eno Thereska (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711481#comment-15711481 ] Eno Thereska commented on KAFKA-4405: - [~guozhang] Ignore previous comment, I was testing by mistake

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

2016-12-01 Thread Jason Gustafson
Hey Neha, Thanks for the thoughtful questions. I'll try to address the first question since Apurva addressed the second. Since most readers are probably getting up to speed with this large proposal, let me first take a step back and explain why we need the AppID at all. As Confluent tradition

[jira] [Commented] (KAFKA-4405) Kafka consumer improperly send prefetch request

2016-12-01 Thread Eno Thereska (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711461#comment-15711461 ] Eno Thereska commented on KAFKA-4405: - [~guozhang]I tested with the new trunk containing KAFKA-4469

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

2016-12-01 Thread Daniel Schierbeck
Hi there, I'm the author of ruby-kafka, and as such am slightly biased towards simplicity of implementation :-) I like the proposal, and would love to use idempotent producer semantics in our projects at Zendesk, but I'm a bit worried about the complexity that would go into the clients;

Re: [DISCUSS] KIP-95: Incremental Batch Processing for Kafka Streams

2016-12-01 Thread Michael Noll
> Instead, why would we not have batch.mode=true/false? Matthias already replied in detail, but let me also throw in my thoughts: I'd argue that "there is no batch [mode]" (just like "there is no spoon" for the Matrix fans out there :-P). What would a batch mode look like? It is primarily

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

2016-12-01 Thread Ismael Juma
Thanks for submitting this KIP as it includes important improvements to Kafka's semantics. I will send a follow-up with more detailed feedback, but I have a process question in the meantime: is there a plan to move part or all of the Google Doc content to the wiki? At least protocol and message

Build failed in Jenkins: kafka-trunk-jdk7 #1721

2016-12-01 Thread Apache Jenkins Server
See Changes: [wangguoz] KAFKA-3637: Added initial states -- [...truncated 14223 lines...] org.apache.kafka.streams.processor.TopologyBuilderTest > shouldNotAllowNullNameWhenAddingSource

Re: [VOTE] KIP-96 - Add per partition metrics for in-sync and replica count

2016-12-01 Thread Tom Crayford
+1 (non-binding) On Thu, Dec 1, 2016 at 12:11 AM, Apurva Mehta wrote: > +1 (non-binding) > > On Wed, Nov 30, 2016 at 2:00 PM, Jason Gustafson > wrote: > > > +1. Thanks for the KIP! > > > > On Wed, Nov 30, 2016 at 1:47 PM, Gwen Shapira

[jira] [Updated] (KAFKA-4461) When using ProcessorTopologyTestDriver, the combination of map and .groupByKey does not produce any result

2016-12-01 Thread Hamidreza Afzali (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hamidreza Afzali updated KAFKA-4461: Description: *Problem* When using ProcessorTopologyTestDriver in the latest Kafka 0.10.1,

Build failed in Jenkins: kafka-trunk-jdk8 #1073

2016-12-01 Thread Apache Jenkins Server
See Changes: [jason] KAFKA-4399; Fix deadlock between cleanupGroupMetadata and offset commit -- [...truncated 14660 lines...] org.apache.kafka.streams.processor.TopologyBuilderTest >

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

2016-12-01 Thread Apurva Mehta
Hi Jay, Thanks for your comments. Answers to some of your points are below: 2. There have been long debates about the necessity of the initTransactions method. Let's consider the options for doing without the initTransactions method: - If we do it on demand, we have to consider that the

Jenkins build is back to normal : kafka-trunk-jdk7 #1724

2016-12-01 Thread Apache Jenkins Server
See

Re: [DISCUSS] KIP-93: Improve invalid timestamp handling in Kafka Streams

2016-12-01 Thread Matthias J. Sax
Yes! Sorry for that typo. On 12/1/16 5:07 PM, Guozhang Wang wrote: > You mean "it is a backward incompatible change" right? > > On Wed, Nov 30, 2016 at 4:28 PM, Matthias J. Sax > wrote: > >> Thanks for this clarification (and your +1) >> >> I completely agree and just

[GitHub] kafka pull request #2202: MINOR: Improvements in group metadata cleanup and ...

2016-12-01 Thread hachikuji
GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/2202 MINOR: Improvements in group metadata cleanup and test coverage You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-12-01 Thread Todd Palino
Come on, I’ve done at least 2 talks on this one :) Producing counts to a topic is part of it, but that’s only part. So you count you have 100 messages in topic A. When you mirror topic A to another cluster, you have 99 messages. Where was your problem? Or worse, you have 100 messages, but one