date:20151009


[ 
https://issues.apache.org/jira/browse/KAFKA-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951391#comment-14951391
 ] 

ASF GitHub Bot commented on KAFKA-2613:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/293

KAFKA-2613: Make maxParallelForks configurable via Gradle config so it can 
be turned down on shared build infrastructure.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-2613-user-configurable-max-forks

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/293.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #293


commit 946a3ad65d848f2864f4ec81c12a1a55a96d4b50
Author: Ewen Cheslack-Postava 
Date:   2015-10-09T23:31:02Z

KAFKA-2613: Make maxParallelForks configurable via Gradle config so it can 
be turned down on shared build infrastructure.




> Consider capping `maxParallelForks` for Jenkins builds
> --
>
> Key: KAFKA-2613
> URL: https://issues.apache.org/jira/browse/KAFKA-2613
> Project: Kafka
>  Issue Type: Sub-task
>  Components: build
>Reporter: Ismael Juma
>
> We currently set `maxParallelForks` to the number returned by 
> `Runtime.availableProcessors`.
> {code}
>   tasks.withType(Test) {
> maxParallelForks = Runtime.runtime.availableProcessors()
>   }
> {code}
> This returns the number of logical cores (including hyperthreaded cores) in 
> the machine.
> This is usually OK when running the tests locally, but the Apache Jenkins 
> slaves run 2 to 3 jobs simultaneously causing a higher number of timing 
> related failures.
> A potential solution is to allow `maxParallelForks` to be set via a Gradle 
> property and use that property to set it to an appropriate value when the 
> build is run from Jenkins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #13

See 

Changes:

[wangguoz] MINOR: putting back kstream stateful transform methods

--
[...truncated 3238 lines...]

kafka.api.SSLConsumerTest > testSimpleConsumption PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.SslTopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack 
PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.api.ConsumerTest > testAutoCommitOnClose PASSED

kafka.integration.SslTopicMetadataTest > testAutoCreateTopicWithCollision PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.SslTopicMetadataTest > testAliveBrokerListWithNoTopics PASSED

kafka.api.SSLConsumerTest > testAutoOffsetReset PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.SslTopicMetadataTest > testAutoCreateTopic PASSED

kafka.api.ConsumerTest > testListTopics PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.SslTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.utils.ByteBoundedBlockingQueueTest > testByteBoundedBlockingQueue PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorOneConsumerNonexistentTopic PASSED

kafka.coordinator.PartitionAssignorTest > 
testRangeAssignorOnlyAssignsPartitionsFromSubscribedTopics PASSED

kafka.coordinator.PartitionAssignorTest > testRangeAssignorOneConsumerNoTopic 
PASSED

kafka.coordinator.PartitionAssignorTest > 
testRangeAssignorOneConsumerMultipleTopics PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorOnlyAssignsPartitionsFromSubscribedTopics PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorTwoConsumersOneTopicOnePartition PASSED

kafka.coordinator.PartitionAssignorTest > 
testRangeAssignorTwoConsumersOneTopicTwoPartitions PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorTwoConsumersOneTopicTwoPartitions PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorMultipleConsumersMixedTopics PASSED

kafka.coordinator.PartitionAssignorTest > 
testRangeAssignorTwoConsumersOneTopicOnePartition PASSED

kafka.coordinator.PartitionAssignorTest > 
testRangeAssignorMultipleConsumersMixedTopics PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorOneConsumerNoTopic PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorOneConsumerOneTopic PASSED

kafka.coordinator.PartitionAssignorTest > testRangeAssignorOneConsumerOneTopic 
PASSED

kafka.coordinator.PartitionAssignorTest > 
testRangeAssignorOneConsumerNonexistentTopic PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorOneConsumerMultipleTopics PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorTwoConsumersTwoTopicsSixPartitions PASSED

kafka.coordinator.PartitionAssignorTest > 
testRangeAssignorTwoConsumersTwoTopicsSixPartitions PASSED

kafka.api.ConsumerTest > testExpandingTopicSubscriptions PASSED

kafka.integration.SslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SslTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.SslTopicMetadataTest > testTopicMetadataRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.api.ConsumerTest > testPatternUnsubscription PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.utils.timer.TimerTest > testTaskExpiration PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.api.ConsumerTest > testGroupConsumption PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.api.ConsumerTest > testPartitionsFor PASSED

[DISCUSS] KIP-37 - Add namespaces in Kafka

2015-10-09 Thread Ashish Singh

Hey Guys,

I just created KIP-37 for adding namespaces to Kafka.

KIP-37

tracks the proposal.

The idea is to make Kafka support multi-tenancy via namespaces.

Feedback and comments are welcome.

-- 

Regards,
Ashish

Jenkins build is back to normal : kafka-trunk-jdk7 #670

See

[GitHub] kafka pull request: KAFKA-2613: Make maxParallelForks configurable...

2015-10-09 Thread ewencp

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/293

KAFKA-2613: Make maxParallelForks configurable via Gradle config so it can 
be turned down on shared build infrastructure.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-2613-user-configurable-max-forks

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/293.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #293


commit 946a3ad65d848f2864f4ec81c12a1a55a96d4b50
Author: Ewen Cheslack-Postava 
Date:   2015-10-09T23:31:02Z

KAFKA-2613: Make maxParallelForks configurable via Gradle config so it can 
be turned down on shared build infrastructure.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-658) Implement "Exact Mirroring" functionality in mirror maker

2015-10-09 Thread Justen Walker (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951388#comment-14951388
 ] 

Justen Walker commented on KAFKA-658:
-

+1 for Exact Mirroring.  I have a use-case where I'd like to replicate a 
cluster for DR purposes, and the current system does not maintain ordering, so 
MirrorMaker cannot be used as-is.

> Implement "Exact Mirroring" functionality in mirror maker
> -
>
> Key: KAFKA-658
> URL: https://issues.apache.org/jira/browse/KAFKA-658
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Jay Kreps
>  Labels: project
>
> There are two ways to implement "mirroring" (i.e. replicating a topic from 
> one cluster to another):
> 1. Do a simple read from the source and write to the destination with no 
> attempt to maintain the same partitioning or offsets in the destination 
> cluster. In this case the destination cluster may have a different number of 
> partitions, and you can even read from many clusters to create a merged 
> cluster. This flexibility is nice. The downside is that since the 
> partitioning and offsets are not the same a consumer of the source cluster 
> has no equivalent position in the destination cluster. This is the style of 
> mirroring we have implemented in the mirror-maker tool and use for datacenter 
> replication today.
> 2. The second style of replication only would allow creating an exact replica 
> of a source cluster (i.e. all partitions and offsets exactly the same). The 
> nice thing about this is that the offsets and partitions would match exactly. 
> The downside is that it is not possible to merge multiple source clusters 
> this way or have different partitioning. We do not currently support this in 
> mirror maker.
> It would be nice to implement the second style as an option in mirror maker 
> as having an exact replica would be a nice option to have in the case where 
> you are replicating a single cluster only.
> There are some nuances: In order to maintain the exact offsets it is 
> important to guarantee that the producer never resends a message or loses a 
> message. As a result it would be important to have only a single producer for 
> each destination partition, and check the last produced message on startup 
> (using the getOffsets api) so that in the case of a hard crash messages that 
> are re-consumed are not re-emitted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: putting back kstream stateful transform...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/292


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2613) Consider capping `maxParallelForks` for Jenkins builds

2015-10-09 Thread Ewen Cheslack-Postava (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-2613:
-
Assignee: Ewen Cheslack-Postava
Reviewer: Gwen Shapira
  Status: Patch Available  (was: Open)

Not sure whether this patch should actually close this issue, but it at least 
makes the maxParallelForks configurable so we could set that flag on the 
Jenkins builds.

> Consider capping `maxParallelForks` for Jenkins builds
> --
>
> Key: KAFKA-2613
> URL: https://issues.apache.org/jira/browse/KAFKA-2613
> Project: Kafka
>  Issue Type: Sub-task
>  Components: build
>Reporter: Ismael Juma
>Assignee: Ewen Cheslack-Postava
>
> We currently set `maxParallelForks` to the number returned by 
> `Runtime.availableProcessors`.
> {code}
>   tasks.withType(Test) {
> maxParallelForks = Runtime.runtime.availableProcessors()
>   }
> {code}
> This returns the number of logical cores (including hyperthreaded cores) in 
> the machine.
> This is usually OK when running the tests locally, but the Apache Jenkins 
> slaves run 2 to 3 jobs simultaneously causing a higher number of timing 
> related failures.
> A potential solution is to allow `maxParallelForks` to be set via a Gradle 
> property and use that property to set it to an appropriate value when the 
> build is run from Jenkins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2630) Add Namespaces to Kafka

Ashish K Singh created KAFKA-2630:
-

 Summary: Add Namespaces to Kafka
 Key: KAFKA-2630
 URL: https://issues.apache.org/jira/browse/KAFKA-2630
 Project: Kafka
  Issue Type: New Feature
Reporter: Ashish K Singh
Assignee: Ashish K Singh


Apache Kafka is rapidly finding its place in data heavy organizations as a 
fault-tolerant message bus. One of the goals of Kafka is data integration, 
which makes it important to support many users in one Kafka system. With 
increasing adoption and user community, support for multi-tenancy is becoming a 
popular demand. There have been a few discussions on Apache Kafka’s mailing 
lists regarding the same, indicating importance of the feature. Namespaces will 
allow/ enable many functionalities that require logical grouping of topics. If 
you think topic as a SQL table, then namespace is a SQL database that lets you 
group tables together.

[KIP-37|https://cwiki.apache.org/confluence/display/KAFKA/KIP-37+-+Add+Namespaces+to+Kafka]
 covers the details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2629) Enable getting SSL password from an executable rather than passing plaintext password


[ 
https://issues.apache.org/jira/browse/KAFKA-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951200#comment-14951200
 ] 

Ashish K Singh commented on KAFKA-2629:
---

Thanks for your thoughts [~sriharsha]. I think you raised following concerns, 
trying to address them below.

bq. The distribution of ssl.properties along with a plaintext password is been 
a common way of doing things. In Hadoop they do this as well.

Hadoop implemented something called the CredentialProvider specifically for the 
purpose of encrypting passwords. See 
https://issues.apache.org/jira/browse/HADOOP-10607.  This functionality is now 
supported by other projects, including Hive, HBase, etc. 

If you google, you will find many well known products use similar approaches. 
Even if someone is not, they should as per the PCI DSS v3 standard. This comes 
as an ask from our most security-conscious customers. I am sure this will be 
soon asked by other Kafka users as well. We have a choice of creating a wrapper 
that does this and calls Kafka, however I am sure this will be required in 
Apache Kafka some day, if not today.

bq. I never seen any system doing this so far for SSL. Why do you think 
filesystem permission not suffice?

As per PCI DSS v3:

https://www.pcisecuritystandards.org/documents/PCI_DSS_v3.pdf

8.2.1 Using strong cryptography, render all authentication credentials (such as 
passwords/phrases) unreadable during transmission and storage on all system 
components.

bq. In your proposal you are saying an executable is also protected by same 
file system permissions than how it is providing any additional security?

The idea is that executable can get the password in secure way. The degree of 
security is customizable - it can get the password from an external secret 
manager with centralized control and audit functionality or can decrypt a 
locally stored password using a secret passed through the environment. Sure, 
one can access the executable with proper permissions, but will not be able to 
access the env variables that the process, starting Kafka server, used to get 
the password.

> Enable getting SSL password from an executable rather than passing plaintext 
> password
> -
>
> Key: KAFKA-2629
> URL: https://issues.apache.org/jira/browse/KAFKA-2629
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> Currently there are a couple of options to pass SSL passwords to Kafka, i.e., 
> via properties file or via command line argument. Both of these are not 
> recommended security practices.
> * A password on a command line is a no-no: it's trivial to see that password 
> just by using the 'ps' utility.
> * Putting a password into a file, and then passing the location to that file, 
> is the next best option. The access to the file will be governed by unix 
> access permissions which we all know and love. The downside is that the 
> password is still just sitting there in a file, and those who have access can 
> still see it trivially.
> * The most general, secure solution is to provide a layer of abstraction: 
> provide functionality to get the password from "somewhere else".  The most 
> flexible and generic way to do this is to simply call an executable which 
> returns the desired password. 
> ** The executable is again protected with normal file system privileges
> ** The simplest form, a script that looks like "echo 'my-password'", devolves 
> back to putting the password in a file
> ** A more interesting implementation could open up a local encrypted password 
> store and extract the password from it
> ** A maximally secure implementation could contact an external secret manager 
> with centralized control and audit functionality.
> ** In short: getting the password as the output of a script/executable is 
> maximally generic and enables both simple and complex use cases.
> This JIRA intend to add a config param to enable passing an executable to 
> Kafka for SSL passwords.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2629) Enable getting SSL password from an executable rather than passing plaintext password

2015-10-09 Thread Haohui Mai (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951306#comment-14951306
 ] 

Haohui Mai commented on KAFKA-2629:
---

While flexibility is generally a good thing, in my opinion the security 
consequences in real-world of this approach needs to be carefully evaluated.

My question is that why this approach provides more security? Why trusting the 
operating system kernel is insufficient in this case? Why bringing in yet 
another another application to the trusted computing base? How does it widen 
the attack surfaces? What are the kinds of attacks that this approach prevents, 
but putting permissions on top of the file fails to do so?

bq. Hadoop implemented something called the CredentialProvider specifically for 
the purpose of encrypting passwords. See 
https://issues.apache.org/jira/browse/HADOOP-10607. This functionality is now 
supported by other projects, including Hive, HBase, etc.

I believe the Hadoop jira is irrelevant. This is specifically tailored to 
support the transparent encryption at rest project. It provides a pluggable way 
to hook with services like active directory, Apache Ranger and Apache Sentry.

bq. If you google, you will find many well known products use similar 
approaches. Even if someone is not, they should as per the PCI DSS v3 standard. 

Can you list what you specifically referring to?

bq. 8.2.1 Using strong cryptography, render all authentication credentials 
(such as passwords/phrases) unreadable during transmission and storage on all 
system components.

Please correct me if I'm wrong, it looks to me that you misread the standard? 
The statement you specifically refers to user authentication services. The 
password in SSL keystore serves as a master keyphrase to protect the private 
key stored in the keystore.

bq. Sure, one can access the executable with proper permissions, but will not 
be able to access the env variables that the process, starting Kafka server, 
used to get the password.

I'd like to point out that the other side of the argument is that the system 
needs to trust the your customized application to be secure. Unfortunately 
building a secure application is often tricky[1,2,3,4]. Programs like gpg have 
mechanisms to defend against these attacks.

If you really want a workflow like that it is possible to use gpg to extract 
the passphase from a centralized keystore, putting it into a temporary file 
with proper permissions and start the Kafka server. Indeed there are some 
tricky issues to handle in order to secure the temporary file, but in practice 
it is much easier for security reviews and give much better security 
guarantees. Does it solve your use cases?

In short, encryption is a way to achieve security but by no means that it is 
equivalent to security. The trusts and security consequences have to be 
carefully evaluated.

References:

1. Moxie Marlinspike, 
https://www.blackhat.com/presentations/bh-usa-09/MARLINSPIKE/BHUSA09-Marlinspike-DefeatSSL-SLIDES.pdf
2. Xiang Cai et al. Exploiting Unix File-System Races via Algorithmic 
Complexity Attacks, In IEEE S, 2009.
3. J. Alex Halderman et al. Lest We Remember: Cold Boot Attacks on Encryption 
Keys. In USENIX Security, 2008.
4. Ian Goldberg and David Wagner, Randomness and the Netscape Browser. January 
1996 Dr. Dobb's Journal


> Enable getting SSL password from an executable rather than passing plaintext 
> password
> -
>
> Key: KAFKA-2629
> URL: https://issues.apache.org/jira/browse/KAFKA-2629
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> Currently there are a couple of options to pass SSL passwords to Kafka, i.e., 
> via properties file or via command line argument. Both of these are not 
> recommended security practices.
> * A password on a command line is a no-no: it's trivial to see that password 
> just by using the 'ps' utility.
> * Putting a password into a file, and then passing the location to that file, 
> is the next best option. The access to the file will be governed by unix 
> access permissions which we all know and love. The downside is that the 
> password is still just sitting there in a file, and those who have access can 
> still see it trivially.
> * The most general, secure solution is to provide a layer of abstraction: 
> provide functionality to get the password from "somewhere else".  The most 
> flexible and generic way to do this is to simply call an executable which 
> returns the desired password. 
> ** The executable is again protected with normal file system privileges
> ** The simplest form, a script that looks like "echo 'my-password'", devolves 
> back to putting the password

Re: [VOTE] KIP-31 - Move to relative offsets in compressed message sets.

2015-10-09 Thread Jiangjie Qin

Thanks a lot for the reply, Jun. Yes, during implementation we can validate
the version configurations to make sure they make sense.

Since there haven't been objections in a couple of days. I am just closing
this KIP with a pass.

KIP-31 is passed with four +1(binding), four +1(non binding) and no -1.

Thanks a lot for all the reviews and votes.

Jiangjie (Becket) Qin

On Thu, Oct 8, 2015 at 6:08 PM, Jun Rao  wrote:

> The updated upgrade path looks reasonable to me. Not all combinations of
> the configs are valid though. For example, we probably should disallow
> message.format.version=1
> and intra.cluster.protocol = 0.9.0.
>
> Thanks,
>
> Jun
>
> On Tue, Oct 6, 2015 at 2:58 PM, Jiangjie Qin 
> wrote:
>
> > Hi folks,
> >
> > Sorry for this prolonged voting session and thanks for the votes.
> >
> > There is an additional broker configuration change added to the KIP after
> > the vote. We propose to add a message.format.version configuration to the
> > broker to indicate which version it should use to store the message on
> > disk.
> >
> > It is mainly trying to minimize the format conversion for consumption
> > during rolling out. Because the client upgrade could take some time and
> it
> > can be expensive to give up zero-copy for the majority of the consumers,
> we
> > want to avoid doing that.
> >
> > I would like to see if people have concerns over this change or not. If
> > there is no concerns, I will close the vote as passed. Otherwise I will
> > initiate another vote.
> >
> > Thanks,
> >
> > Jiangjie (Becket) QIn
> >
> >
> > On Fri, Sep 25, 2015 at 4:41 PM, Ewen Cheslack-Postava <
> e...@confluent.io>
> > wrote:
> >
> > > +1
> > >
> > > -Ewen
> > >
> > > On Fri, Sep 25, 2015 at 11:15 AM, Jun Rao  wrote:
> > >
> > > > +1. I agree that it's worth thinking through the migration plan a bit
> > > more.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, Sep 24, 2015 at 6:14 PM, Joel Koshy 
> > wrote:
> > > >
> > > > > +1 on everything but the upgrade plan, which is a bit scary - will
> > > > > comment on the discuss thread.
> > > > >
> > > > > On Thu, Sep 24, 2015 at 9:51 AM, Mayuresh Gharat
> > > > >  wrote:
> > > > > > +1
> > > > > >
> > > > > > On Wed, Sep 23, 2015 at 10:16 PM, Guozhang Wang <
> > wangg...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> ＋1
> > > > > >>
> > > > > >> On Wed, Sep 23, 2015 at 9:32 PM, Aditya Auradkar <
> > > > > >> aaurad...@linkedin.com.invalid> wrote:
> > > > > >>
> > > > > >> > +1
> > > > > >> >
> > > > > >> > On Wed, Sep 23, 2015 at 8:03 PM, Neha Narkhede <
> > n...@confluent.io
> > > >
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > +1
> > > > > >> > >
> > > > > >> > > On Wed, Sep 23, 2015 at 6:21 PM, Todd Palino <
> > tpal...@gmail.com
> > > >
> > > > > >> wrote:
> > > > > >> > >
> > > > > >> > > > +1000
> > > > > >> > > >
> > > > > >> > > > !
> > > > > >> > > >
> > > > > >> > > > -Todd
> > > > > >> > > >
> > > > > >> > > > On Wednesday, September 23, 2015, Jiangjie Qin
> > > > > >> >  > > > > >> > > >
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi,
> > > > > >> > > > >
> > > > > >> > > > > Thanks a lot for the reviews and feedback on KIP-31. It
> > > looks
> > > > > all
> > > > > >> the
> > > > > >> > > > > concerns of the KIP has been addressed. I would like to
> > > start
> > > > > the
> > > > > >> > > voting
> > > > > >> > > > > process.
> > > > > >> > > > >
> > > > > >> > > > > The short summary for the KIP:
> > > > > >> > > > > We are going to use the relative offset in the message
> > > format
> > > > to
> > > > > >> > avoid
> > > > > >> > > > > server side recompression.
> > > > > >> > > > >
> > > > > >> > > > > In case you haven't got a chance to check, here is the
> KIP
> > > > link.
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-31+-+Move+to+relative+offsets+in+compressed+message+sets
> > > > > >> > > > >
> > > > > >> > > > > Thanks,
> > > > > >> > > > >
> > > > > >> > > > > Jiangjie (Becket) Qin
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > --
> > > > > >> > > Thanks,
> > > > > >> > > Neha
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> -- Guozhang
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -Regards,
> > > > > > Mayuresh R. Gharat
> > > > > > (862) 250-7125
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Ewen
> > >
> >
>

Build failed in Jenkins: kafka-trunk-jdk7 #668

See 

Changes:

[wangguoz] MINOR: typing ProcessorDef

[wangguoz] KAFKA-2596: reject commits from unknown groups with positive 
generations

--
[...truncated 164 lines...]
:380:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
if (value.expireTimestamp == 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP)

  ^
:115:
 value METADATA_FETCH_TIMEOUT_CONFIG in object ProducerConfig is deprecated: 
see corresponding Javadoc for more information.
props.put(ProducerConfig.METADATA_FETCH_TIMEOUT_CONFIG, 
config.metadataFetchTimeoutMs.toString)
 ^
:117:
 value TIMEOUT_CONFIG in object ProducerConfig is deprecated: see corresponding 
Javadoc for more information.
props.put(ProducerConfig.TIMEOUT_CONFIG, config.requestTimeoutMs.toString)
 ^
:121:
 value BLOCK_ON_BUFFER_FULL_CONFIG in object ProducerConfig is deprecated: see 
corresponding Javadoc for more information.
  props.put(ProducerConfig.BLOCK_ON_BUFFER_FULL_CONFIG, "false")
   ^
:75:
 value BLOCK_ON_BUFFER_FULL_CONFIG in object ProducerConfig is deprecated: see 
corresponding Javadoc for more information.
producerProps.put(ProducerConfig.BLOCK_ON_BUFFER_FULL_CONFIG, "true")
 ^
:185:
 value BLOCK_ON_BUFFER_FULL_CONFIG in object ProducerConfig is deprecated: see 
corresponding Javadoc for more information.
maybeSetDefaultProperty(producerProps, 
ProducerConfig.BLOCK_ON_BUFFER_FULL_CONFIG, "true")
  ^
:234:
 method readLine in class DeprecatedConsole is deprecated: Use the method in 
scala.io.StdIn
Console.readLine().equalsIgnoreCase("y")
^
:389:
 class BrokerEndPoint in object UpdateMetadataRequest is deprecated: see 
corresponding Javadoc for more information.
  new UpdateMetadataRequest.BrokerEndPoint(brokerEndPoint.id, 
brokerEndPoint.host, brokerEndPoint.port)
^
:391:
 constructor UpdateMetadataRequest in class UpdateMetadataRequest is 
deprecated: see corresponding Javadoc for more information.
new UpdateMetadataRequest(controllerId, controllerEpoch, 
liveBrokers.asJava, partitionStates.asJava)
^
:129:
 method readFromReadableChannel in class NetworkReceive is deprecated: see 
corresponding Javadoc for more information.
  response.readFromReadableChannel(channel)
   ^
there were 14 feature warnings; re-run with -feature for details
17 warnings found
:kafka-trunk-jdk7:core:processResources UP-TO-DATE
:kafka-trunk-jdk7:core:classes
:kafka-trunk-jdk7:clients:compileTestJava UP-TO-DATE
:kafka-trunk-jdk7:clients:processTestResources UP-TO-DATE
:kafka-trunk-jdk7:clients:testClasses UP-TO-DATE
:kafka-trunk-jdk7:core:copyDependantLibs
:kafka-trunk-jdk7:core:jar
:clients:compileJava UP-TO-DATE
:clients:processResources UP-TO-DATE
:clients:classes UP-TO-DATE
:clients:determineCommitId UP-TO-DATE
:clients:createVersionFile
:clients:jar UP-TO-DATE
:log4j-appender:compileJava UP-TO-DATE
:log4j-appender:processResources UP-TO-DATE
:log4j-appender:classes UP-TO-DATE
:log4j-appender:jar UP-TO-DATE
:core:compileJava UP-TO-DATE
:core:compileScala
:78:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.

org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP

Re: Kafka dotnet SDK suggestion

2015-10-09 Thread sugumar analysis

Hi Ewen,

Thanks for your quickest and valuable response. Because i struggled lot to
get response for my queries.

Some more clarification needed

As per my understanding Confluent is Open source and we can use it in our
Production environment.

Assume we are using confluent.
In our project there are 300 system each have 5 product so totally 1500
request may reach simultaneously to confluent Rest API server.

In this case how the load will be balanced?

we don't find any documentation for  Confluent Rest Api call. Is there any
documentation available for this?

Please help me regarding these clarification.. waiting for your valuable
response

Thanks,
Sugumar J






On Thu, Oct 8, 2015 at 7:23 AM, Ewen Cheslack-Postava 
wrote:

> The project doesn't make official recommendations about third party
> clients. The only client directly supported is the Java/Scala library
> included in the project's source repository.
>
> However, the community does maintain a list of third-party clients here:
> https://cwiki.apache.org/confluence/display/KAFKA/Clients Since it is
> community maintained, we rely on people telling us about new libraries to
> get them on there. I've just added the second library you mentioned to the
> .net section.
>
> As for which you should use, you'll have to evaluate the functionality of
> both and which fits your needs best. The first one looks much more active
> (the second hasn't had any commits since 2014), but that doesn't
> necessarily say anything about their relative maturity.
>
> If you're not sure about the quality of either of them, another option is
> to use a REST proxy. Shameless plug since I work on it -- Confluent has one
> as part of its platform (and it's open source). Since it uses the Java
> clients, it covers all the functionality provided by them.
>
> -Ewen
>
> On Wed, Oct 7, 2015 at 8:51 AM, sugumar analysis <
> sugumar.analy...@gmail.com
> > wrote:
>
> > Hi All,
> >
> > We are developing Messaging system with Kafka using DotNet (C#).
> >
> > We found there are 2 different SDK available for Dotnet
> >
> > 1. https://github.com/Jroland/kafka-net/
> > 2. https://github.com/ExactTargetDev/kafka-net
> >
> >
> > First one is officially referred by Apache Kafka. But It is an initial
> > stage, there are some functionalities not available ex. ConsumerGroup,
> > AutoCommit features.
> >
> > In Second one it has all the features but its not referred by Apache
> Kafka
> >
> > *Can we choose second one for our development? *
> > *  or *
> > *should we use official SDK which is mentioned in Kafka site.?*
> >
> > Please give us your suggestions.
> >
> >
> > Thanks,
> > Sugumar J
> >
>
>
>
> --
> Thanks,
> Ewen
>

[GitHub] kafka pull request: MINOR: typing ProcessorDef

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/289


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2596) Coordinator should return illegal generation for commits from unknown groups with non-negative generation


[ 
https://issues.apache.org/jira/browse/KAFKA-2596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949969#comment-14949969
 ] 

ASF GitHub Bot commented on KAFKA-2596:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/267


> Coordinator should return illegal generation for commits from unknown groups 
> with non-negative generation
> -
>
> Key: KAFKA-2596
> URL: https://issues.apache.org/jira/browse/KAFKA-2596
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>
> Currently the consumer coordinator accepts offset commits blindly if it has 
> no state stored for the associated groupId. This means that upon coordinator 
> failover, offset commits from any member of a group will be accepted, even if 
> that member is from an older generation. A better way of handling this case 
> would be to return an ILLEGAL_GENERATION error when the generation in the 
> commit request is greater than or equal to 0. Consumers that are not using 
> group management will always send a generation of -1, so their commits will 
> still be accepted as valid. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2596: reject commits from unknown groups...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/267


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-2596) Coordinator should return illegal generation for commits from unknown groups with non-negative generation

2015-10-09 Thread Guozhang Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-2596.
--
   Resolution: Fixed
Fix Version/s: 0.9.0.0

Issue resolved by pull request 267
[https://github.com/apache/kafka/pull/267]

> Coordinator should return illegal generation for commits from unknown groups 
> with non-negative generation
> -
>
> Key: KAFKA-2596
> URL: https://issues.apache.org/jira/browse/KAFKA-2596
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.9.0.0
>
>
> Currently the consumer coordinator accepts offset commits blindly if it has 
> no state stored for the associated groupId. This means that upon coordinator 
> failover, offset commits from any member of a group will be accepted, even if 
> that member is from an older generation. A better way of handling this case 
> would be to return an ILLEGAL_GENERATION error when the generation in the 
> commit request is greater than or equal to 0. Consumers that are not using 
> group management will always send a generation of -1, so their commits will 
> still be accepted as valid. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Weird error on machine restart when creating ConsumerConnector reference

2015-10-09 Thread Sivananda Reddys Thummala Abbigari

Hi,

*Following is the consumer related code*:

Line#1 > ConsumerConnector consumer =
kafka.consumer.Consumer.createJavaConsumerConnector(createConsumerConfig(zkConnect,
consumerGroupId, offSetStorage));

private ConsumerConfig createConsumerConfig(String zookeeperConnectString,
String consumerGroupId, String offSetStorage) {
Properties props = new Properties();
props.put("zookeeper.connect", zookeeperConnectString);
props.put("group.id", consumerGroupId);
props.put("offsets.storage", offSetStorage);
return new ConsumerConfig(props);
}

Kafka/ZK/Client processes are run on a single machine(dev environment).
Client process created bunch of topics(*with replication factor of 1*),
performed R/W operations on them. Then I re-started the machine and then
started all the three processes(Kafka/ZK/Client), I see the following error
in the kafka server logs:

*kafka.admin.AdminOperationException: replication factor: 3 larger than
available brokers: 1*
at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:70)
at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:171)
at kafka.server.KafkaApis$$anonfun$19.apply(KafkaApis.scala:513)
at kafka.server.KafkaApis$$anonfun$19.apply(KafkaApis.scala:503)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at
scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
at scala.collection.SetLike$class.map(SetLike.scala:93)
at scala.collection.AbstractSet.map(Set.scala:47)
at kafka.server.KafkaApis.getTopicMetadata(KafkaApis.scala:503)
at kafka.server.KafkaApis.handleConsumerMetadataRequest(KafkaApis.scala:607)
at kafka.server.KafkaApis.handle(KafkaApis.scala:69)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
at java.lang.Thread.run(Thread.java:745)

This exception occurs when Line#1 is executed, there is no topic with
replication factor of 3.

*Kafka Version*: kafka_2.10-0.8.2.1

So I re-started Kafka + ZK + client processes again and the error is gone.
Was wondering what could the reason for this behavior?. Please advice.

Thank you,
Siva.

Build failed in Jenkins: kafka-trunk-jdk8 #11

See 

Changes:

[wangguoz] MINOR: typing ProcessorDef

[wangguoz] KAFKA-2596: reject commits from unknown groups with positive 
generations

--
[...truncated 2044 lines...]

kafka.api.ConsumerBounceTest > testSeekAndCommitWithBrokerFailures PASSED

kafka.admin.DeleteTopicTest > testAddPartitionDuringDeleteTopic PASSED

kafka.api.ProducerFailureHandlingTest > testInvalidPartition PASSED

kafka.integration.SslTopicMetadataTest > testAliveBrokerListWithNoTopics PASSED

kafka.api.QuotasTest > testProducerConsumerOverrideUnthrottled PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SslTopicMetadataTest > testAutoCreateTopic PASSED

kafka.api.ProducerFailureHandlingTest > testSendAfterClosed PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicWithAllAliveReplicas PASSED

kafka.producer.ProducerTest > testSendWithDeadBroker PASSED

kafka.api.ApiUtilsTest > testShortStringNonASCII PASSED

kafka.integration.PlaintextTopicMetadataTest > testTopicMetadataRequest PASSED

kafka.api.ApiUtilsTest > testShortStringASCII PASSED

kafka.integration.SslTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckZero PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionEnabledByTopicOverride PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicDuringAddPartition PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.server.OffsetCommitTest > testUpdateOffsets PASSED

kafka.api.QuotasTest > testThrottledProducerConsumer PASSED

unit.kafka.security.auth.PermissionTypeTest > testFromString PASSED

kafka.integration.SslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.server.ReplicaManagerTest > testHighWaterMarkDirectoryMapping PASSED

kafka.integration.MinIsrConfigTest > testDefaultKafkaConfig PASSED

kafka.server.ReplicaManagerTest > testIllegalRequiredAcks PASSED

kafka.server.HighwatermarkPersistenceTest > 
testHighWatermarkPersistenceMultiplePartitions PASSED

kafka.server.ReplicaManagerTest > testHighwaterMarkRelativeDirectoryMapping 
PASSED

kafka.server.HighwatermarkPersistenceTest > 
testHighWatermarkPersistenceSinglePartition PASSED

kafka.producer.AsyncProducerTest > testFailedSendRetryLogic PASSED

kafka.integration.SslTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.producer.AsyncProducerTest > testQueueTimeExpired PASSED

kafka.producer.AsyncProducerTest > testPartitionAndCollateEvents PASSED

kafka.producer.AsyncProducerTest > testBatchSize PASSED

kafka.producer.AsyncProducerTest > testSerializeEvents PASSED

kafka.producer.SyncProducerTest > testReachableServer PASSED

kafka.server.OffsetCommitTest > testLargeMetadataPayload PASSED

kafka.producer.SyncProducerTest > testMessageSizeTooLarge PASSED

kafka.integration.SslTopicMetadataTest > testTopicMetadataRequest PASSED

kafka.server.OffsetCommitTest > testCommitAndFetchOffsets PASSED

kafka.producer.SyncProducerTest > testNotEnoughReplicas PASSED

kafka.producer.SyncProducerTest > testMessageSizeTooLargeWithAckZero PASSED

kafka.producer.AsyncProducerTest > testProducerQueueSize PASSED

kafka.producer.AsyncProducerTest > testRandomPartitioner PASSED

kafka.producer.AsyncProducerTest > testInvalidConfiguration PASSED

kafka.producer.AsyncProducerTest > testInvalidPartition PASSED

kafka.producer.AsyncProducerTest > testNoBroker PASSED

kafka.producer.AsyncProducerTest > testProduceAfterClosed PASSED

kafka.producer.AsyncProducerTest > testJavaProducer PASSED

kafka.producer.AsyncProducerTest > testIncompatibleEncoder PASSED

kafka.producer.SyncProducerTest > testProducerCanTimeout PASSED

kafka.integration.SslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.server.OffsetCommitTest > testOffsetExpiration PASSED

kafka.producer.SyncProducerTest > testProduceRequestWithNoResponse PASSED

kafka.producer.SyncProducerTest > testEmptyProduceRequest PASSED

kafka.server.SslReplicaFetchTest > testReplicaFetcherThread PASSED

kafka.log.CleanerTest > testBuildOffsetMap PASSED

kafka.log.CleanerTest > testSegmentGrouping PASSED

kafka.log.CleanerTest > testCleanSegmentsWithAbort PASSED

kafka.log.CleanerTest > testSegmentGroupingWithSparseOffsets PASSED

kafka.log.CleanerTest > testRecoveryAfterCrash PASSED

kafka.log.CleanerTest > testCleaningWithDeletes PASSED

kafka.log.CleanerTest > testCleanSegments PASSED

kafka.log.CleanerTest > testCleaningWithUnkeyedMessages PASSED

kafka.server.OffsetCommitTest > testNonExistingTopicOffsetCommit PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest >

[GitHub] kafka pull request: MINOR: putting back kstream stateful transform...

2015-10-09 Thread ymatsuda

GitHub user ymatsuda opened a pull request:

https://github.com/apache/kafka/pull/292

MINOR: putting back kstream stateful transform methods

@guozhangwang 

* added back type safe stateful transform methods (kstream.transform() and 
kstream.transformValues())
* changed kstream.process() to void

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ymatsuda/kafka transform_method

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/292.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #292


commit 9d8654b9536115c25af7f58658248cde041a4646
Author: Yasuhiro Matsuda 
Date:   2015-10-09T21:36:34Z

MINOR: putting back kstream stateful transform methods

commit 7ccbf05a5087ad77d312add0e46b4369284dc131
Author: Yasuhiro Matsuda 
Date:   2015-10-09T22:04:39Z

MINOR: putting back kstream stateful transform methods




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk7 #669

See 

Changes:

[wangguoz] KAFKA-2622: Add Time logical type for Copycat.

[wangguoz] KAFKA-2600: Align Kafka Streams' interfaces with Java 8 functional

[wangguoz] MINOR: Fix exception message in Copycat's Time logical type.

--
[...truncated 1520 lines...]
kafka.zk.ZKPathTest > testCreatePersistentSequentialThrowsException PASSED

kafka.zk.ZKPathTest > testCreatePersistentSequentialExists PASSED

kafka.zk.ZKPathTest > testCreateEphemeralPathExists PASSED

kafka.zk.ZKPathTest > testCreatePersistentPath PASSED

kafka.zk.ZKPathTest > testMakeSurePersistsPathExistsThrowsException PASSED

kafka.api.ConsumerTest > testAutoCommitOnClose PASSED

kafka.zk.ZKPathTest > testCreateEphemeralPathThrowsException PASSED

kafka.zk.ZKPathTest > testCreatePersistentPathThrowsException PASSED

kafka.zk.ZKPathTest > testMakeSurePersistsPathExists PASSED

kafka.server.ServerShutdownTest > testCleanShutdownAfterFailedStartup PASSED

kafka.server.ServerShutdownTest > testConsecutiveShutdown PASSED

kafka.server.ServerShutdownTest > testCleanShutdown PASSED

kafka.api.ConsumerTest > testListTopics PASSED

kafka.api.ProducerSendTest > testCloseWithZeroTimeoutFromCallerThread PASSED

kafka.server.ServerShutdownTest > testCleanShutdownWithDeleteTopicEnabled PASSED

kafka.api.SSLConsumerTest > testPositionAndCommit PASSED

kafka.api.ConsumerTest > testExpandingTopicSubscriptions PASSED

kafka.api.SSLConsumerTest > testGroupConsumption PASSED

kafka.api.ProducerSendTest > testCloseWithZeroTimeoutFromSenderThread PASSED

kafka.api.ConsumerTest > testPatternUnsubscription PASSED

kafka.api.SSLConsumerTest > testPartitionsFor PASSED

kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures PASSED

kafka.log.LogManagerTest > testCleanupSegmentsToMaintainSize PASSED

kafka.log.LogManagerTest > testRecoveryDirectoryMappingWithRelativeDirectory 
PASSED

kafka.log.LogManagerTest > testGetNonExistentLog PASSED

kafka.log.LogManagerTest > testTwoLogManagersUsingSameDirFails PASSED

kafka.log.LogManagerTest > testLeastLoadedAssignment PASSED

kafka.api.ConsumerTest > testGroupConsumption PASSED

kafka.api.ProducerBounceTest > testBrokerFailure PASSED

kafka.log.LogSegmentTest > testRecoveryWithCorruptMessage PASSED

kafka.log.LogSegmentTest > testRecoveryFixesCorruptIndex PASSED

kafka.log.LogSegmentTest > testReadFromGap PASSED

kafka.log.LogSegmentTest > testTruncate PASSED

kafka.log.LogSegmentTest > testReadBeforeFirstOffset PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeAppendMessage PASSED

kafka.log.LogSegmentTest > testChangeFileSuffixes PASSED

kafka.log.LogSegmentTest > testMaxOffset PASSED

kafka.log.LogSegmentTest > testNextOffsetCalculation PASSED

kafka.log.LogSegmentTest > testReadOnEmptySegment PASSED

kafka.log.LogSegmentTest > testReadAfterLast PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeClearShutdown PASSED

kafka.log.LogSegmentTest > testTruncateFull PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

kafka.api.SSLConsumerTest > testSimpleConsumption PASSED

kafka.api.ConsumerTest > testPartitionsFor PASSED

kafka.api.ConsumerTest > testAutoCommitOnRebalance PASSED

kafka.api.SSLConsumerTest > testAutoOffsetReset PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingTopic PASSED

kafka.log.LogTest > testIndexRebuild PASSED

kafka.log.LogTest > testLogRolls PASSED

kafka.log.LogTest > testMessageSizeCheck PASSED

kafka.api.ConsumerTest > testSimpleConsumption PASSED

kafka.log.LogTest > testAsyncDelete PASSED

kafka.log.LogTest > testReadOutOfRange PASSED

kafka.log.LogTest > testReadAtLogGap PASSED

kafka.log.LogTest > testTimeBasedLogRoll PASSED

kafka.log.LogTest > testLoadEmptyLog PASSED

kafka.log.LogTest > testMessageSetSizeCheck PASSED

kafka.log.LogTest > testIndexResizingAtTruncation PASSED

kafka.log.LogTest > testCompactedTopicConstraints PASSED

kafka.log.LogTest > testThatGarbageCollectingSegmentsDoesntChangeOffset PASSED

kafka.log.LogTest > testAppendAndReadWithSequentialOffsets PASSED

kafka.log.LogTest > testParseTopicPartitionNameForNull PASSED

kafka.log.LogTest > testAppendAndReadWithNonSequentialOffsets PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingSeparator PASSED

kafka.api.ConsumerTest > testPartitionPauseAndResume PASSED

kafka.log.LogTest > testCorruptIndexRebuild PASSED

kafka.log.LogTest > testBogusIndexSegmentsAreRemoved PASSED

kafka.log.LogTest > testCompressedMessages PASSED

kafka.log.LogTest > testAppendMessageWithNullPayload PASSED

kafka.api.ConsumerTest > testPartitionReassignmentCallback PASSED

kafka.api.ConsumerTest > testAutoOffsetReset PASSED

kafka.log.LogTest > testCorruptLog PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] PASSED

kafka.log.LogTest > testLogRecoversToCorrectOffset PASSED

kafka.log.LogTest > testReopenThenTruncate PASSED

kafka.log.LogTest >

[jira] [Comment Edited] (KAFKA-2629) Enable getting SSL password from an executable rather than passing plaintext password

2015-10-09 Thread Haohui Mai (JIRA)

[
https://issues.apache.org/jira/browse/KAFKA-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951306#comment-14951306
]

Haohui Mai edited comment on KAFKA-2629 at 10/9/15 10:40 PM:
-

While flexibility is generally a good thing, in my opinion the security
consequences in real-world of this approach needs to be carefully evaluated.

My question is that why this approach provides more security? Why trusting the
operating system kernel is insufficient in this case? Why bringing in yet
another another application to the trusted computing base? How does it widen
the attack surfaces? What are the kinds of attacks that this approach prevents,
but putting permissions on top of the file fails to do so?

bq. Hadoop implemented something called the CredentialProvider specifically for
the purpose of encrypting passwords. See
https://issues.apache.org/jira/browse/HADOOP-10607. This functionality is now
supported by other projects, including Hive, HBase, etc.

I believe the Hadoop jira is irrelevant. This is specifically tailored to hook
with services like Apache Ranger and Apache Sentry -- all of which for user
authentication / authorization.

bq. If you google, you will find many well known products use similar
approaches. Even if someone is not, they should as per the PCI DSS v3 standard.

Can you list what you specifically referring to?

bq. 8.2.1 Using strong cryptography, render all authentication credentials
(such as passwords/phrases) unreadable during transmission and storage on all
system components.

Please correct me if I'm wrong, it looks to me that you misread the standard?
The statement you specifically refers to user authentication services. The
password in SSL keystore serves as a master keyphrase to protect the private
key stored in the keystore.

bq. Sure, one can access the executable with proper permissions, but will not
be able to access the env variables that the process, starting Kafka server,
used to get the password.

I'd like to point out that the other side of the argument is that the system
needs to trust the your customized application to be secure. Unfortunately
building a secure application is often tricky[1,2,3,4]. Programs like gpg have
mechanisms to defend against these attacks.

If you really want a workflow like that it is possible to use gpg to extract
the passphase from a centralized keystore, putting it into a temporary file
with proper permissions and start the Kafka server. Indeed there are some
tricky issues to handle in order to secure the temporary file, but in practice
it is much easier for security reviews and give much better security
guarantees. Does it solve your use cases?

In short, encryption is a way to achieve security but by no means that it is
equivalent to security. The trusts and security consequences have to be
carefully evaluated.

References:

1. Moxie Marlinspike,
https://www.blackhat.com/presentations/bh-usa-09/MARLINSPIKE/BHUSA09-Marlinspike-DefeatSSL-SLIDES.pdf
2. Xiang Cai et al. Exploiting Unix File-System Races via Algorithmic
Complexity Attacks, In IEEE S, 2009.
3. J. Alex Halderman et al. Lest We Remember: Cold Boot Attacks on Encryption
Keys. In USENIX Security, 2008.
4. Ian Goldberg and David Wagner, Randomness and the Netscape Browser. January
1996 Dr. Dobb's Journal

was (Author: wheat9):
While flexibility is generally a good thing, in my opinion the security
consequences in real-world of this approach needs to be carefully evaluated.

I believe the Hadoop jira is irrelevant. This is specifically tailored to
support the transparent encryption at rest project. It provides a pluggable way
to hook with services like active directory, Apache Ranger and Apache Sentry.

bq. If you google, you will find many well known products use similar
approaches. Even if someone is not, they should as per the PCI DSS v3 standard.

Can you list what you specifically referring to?

bq. 8.2.1 Using strong cryptography, render all authentication credentials
(such as passwords/phrases) unreadable during transmission and storage on all
system components.

Please correct me if I'm wrong, it looks to me that you misread the standard?
The statement you specifically refers to user authentication services. The

Build failed in Jenkins: kafka-trunk-jdk8 #12

See 

Changes:

[wangguoz] KAFKA-2622: Add Time logical type for Copycat.

[wangguoz] KAFKA-2600: Align Kafka Streams' interfaces with Java 8 functional

[wangguoz] MINOR: Fix exception message in Copycat's Time logical type.

--
[...truncated 3256 lines...]

kafka.api.SSLConsumerTest > testSeek PASSED

kafka.server.LogRecoveryTest > testHWCheckpointWithFailuresMultipleLogSegments 
PASSED

kafka.admin.AddPartitionsTest > testManualAssignmentOfReplicas PASSED

kafka.server.ServerGenerateBrokerIdTest > testUserConfigAndGeneratedBrokerId 
PASSED

kafka.api.ConsumerTest > testSimpleConsumption PASSED

kafka.server.LogRecoveryTest > testHWCheckpointNoFailuresSingleLogSegment PASSED

kafka.server.ServerGenerateBrokerIdTest > 
testConsistentBrokerIdFromUserConfigAndMetaProps PASSED

kafka.admin.AddPartitionsTest > testReplicaPlacement PASSED

kafka.api.SSLConsumerTest > testPositionAndCommit PASSED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr PASSED

kafka.utils.ReplicationUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.api.ConsumerTest > testPartitionPauseAndResume PASSED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot PASSED

kafka.server.LogRecoveryTest > testHWCheckpointWithFailuresSingleLogSegment 
PASSED

kafka.server.DelayedOperationTest > testRequestPurge PASSED

kafka.server.DelayedOperationTest > testRequestExpiry PASSED

kafka.server.DelayedOperationTest > testRequestSatisfaction PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.server.ServerStartupTest > testConflictBrokerRegistration PASSED

kafka.producer.SyncProducerTest > testReachableServer PASSED

kafka.producer.SyncProducerTest > testMessageSizeTooLarge PASSED

kafka.api.ConsumerTest > testPartitionReassignmentCallback PASSED

kafka.api.SSLConsumerTest > testGroupConsumption PASSED

kafka.producer.SyncProducerTest > testNotEnoughReplicas PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testCompressionSetConsumption 
PASSED

kafka.producer.SyncProducerTest > testMessageSizeTooLargeWithAckZero PASSED

kafka.producer.SyncProducerTest > testProducerCanTimeout PASSED

kafka.api.ConsumerTest > testAutoOffsetReset PASSED

kafka.api.SSLConsumerTest > testPartitionsFor PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testLeaderSelectionForPartition 
PASSED

kafka.producer.SyncProducerTest > testProduceRequestWithNoResponse PASSED

kafka.producer.SyncProducerTest > testEmptyProduceRequest PASSED

kafka.producer.SyncProducerTest > testProduceCorrectlyReceivesResponse PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testConsumerDecoder PASSED

kafka.api.ConsumerTest > testCommitSpecifiedOffsets PASSED

kafka.api.SSLConsumerTest > testSimpleConsumption PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testConsumerRebalanceListener 
PASSED

kafka.api.ConsumerTest > testCommitMetadata PASSED

kafka.consumer.PartitionAssignorTest > testRoundRobinPartitionAssignor PASSED

kafka.consumer.PartitionAssignorTest > testRangePartitionAssignor PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testCompression PASSED

kafka.tools.ConsoleProducerTest > testParseKeyProp PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsOldProducer PASSED

kafka.tools.ConsoleProducerTest > testInvalidConfigs PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsNewProducer PASSED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.api.SSLConsumerTest > testAutoOffsetReset PASSED

kafka.server.DynamicConfigChangeTest > testProcessNotification PASSED

kafka.api.ConsumerTest > testPatternSubscription PASSED

kafka.server.DynamicConfigChangeTest > testClientConfigChange PASSED

kafka.server.OffsetCommitTest > testUpdateOffsets PASSED

kafka.server.DynamicConfigChangeTest > testConfigChangeOnNonExistingTopic PASSED

kafka.server.DynamicConfigChangeTest > testConfigChange PASSED

kafka.server.OffsetCommitTest > testLargeMetadataPayload PASSED

kafka.server.ClientQuotaManagerTest > testQuotaViolation PASSED

kafka.server.ClientQuotaManagerTest > testOverrideParse PASSED

kafka.server.ClientQuotaManagerTest > testQuotaParsing PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[0] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[1] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[2] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[3] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[4] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[5] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[6] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[7] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[8] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[9] PASSED

[jira] [Updated] (KAFKA-2459) Connection backoff/blackout period should start when a connection is disconnected, not when the connection attempt was initiated

2015-10-09 Thread Ewen Cheslack-Postava (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-2459:
-
Reviewer: Guozhang Wang
  Status: Patch Available  (was: Open)

> Connection backoff/blackout period should start when a connection is 
> disconnected, not when the connection attempt was initiated
> 
>
> Key: KAFKA-2459
> URL: https://issues.apache.org/jira/browse/KAFKA-2459
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer, producer 
>Affects Versions: 0.8.2.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Eno Thereska
>
> Currently the connection code for new clients marks the time when a 
> connection was initiated (NodeConnectionState.lastConnectMs) and then uses 
> this to compute blackout periods for nodes, during which connections will not 
> be attempted and the node is not considered a candidate for leastLoadedNode.
> However, in cases where the connection attempt takes longer than the 
> blackout/backoff period (default 10ms), this results in incorrect behavior. 
> If a broker is not available and, for example, the broker does not explicitly 
> reject the connection, instead waiting for a connection timeout (e.g. due to 
> firewall settings), then the backoff period will have already elapsed and the 
> node will immediately be considered ready for a new connection attempt and a 
> node to be selected by leastLoadedNode for metadata updates. I think it 
> should be easy to reproduce and verify this problem manually by using tc to 
> introduce enough latency to make connection failures take > 10ms.
> The correct behavior would use the disconnection event to mark the end of the 
> last connection attempt and then wait for the backoff period to elapse after 
> that.
> See 
> http://mail-archives.apache.org/mod_mbox/kafka-users/201508.mbox/%3CCAJY8EofpeU4%2BAJ%3Dw91HDUx2RabjkWoU00Z%3DcQ2wHcQSrbPT4HA%40mail.gmail.com%3E
>  for the original description of the problem.
> This is related to KAFKA-1843 because leastLoadedNode currently will 
> consistently choose the same node if this blackout period is not handled 
> correctly, but is a much smaller issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2431) Test SSL/TLS impact on performance

2015-10-09 Thread Ben Stopford (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950641#comment-14950641
 ] 

Ben Stopford commented on KAFKA-2431:
-

Summary report can be found here: 
https://docs.google.com/document/d/1HlHkjAPK9kR_lk0DF34c3NDblCYTnN6D8GENV6YI2f8

> Test SSL/TLS impact on performance
> --
>
> Key: KAFKA-2431
> URL: https://issues.apache.org/jira/browse/KAFKA-2431
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Ben Stopford
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Test new Producer and new Consumer performance with and without SSL/TLS once 
> the SSL/TLS branch is integrated.
> The ideal scenario is that SSL/TLS would not have an impact if disabled. When 
> enabled, there will be some overhead (encryption and the inability to use 
> `SendFile`) and it will be good to quantify it. The encryption overhead is 
> reduced if recent JDKs are used with CPUs that support AES-specific 
> instructions (https://en.wikipedia.org/wiki/AES_instruction_set).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2431) Test SSL/TLS impact on performance


[ 
https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950644#comment-14950644
 ] 

ASF GitHub Bot commented on KAFKA-2431:
---

Github user benstopford closed the pull request at:

https://github.com/apache/kafka/pull/217


> Test SSL/TLS impact on performance
> --
>
> Key: KAFKA-2431
> URL: https://issues.apache.org/jira/browse/KAFKA-2431
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Ben Stopford
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Test new Producer and new Consumer performance with and without SSL/TLS once 
> the SSL/TLS branch is integrated.
> The ideal scenario is that SSL/TLS would not have an impact if disabled. When 
> enabled, there will be some overhead (encryption and the inability to use 
> `SendFile`) and it will be good to quantify it. The encryption overhead is 
> reduced if recent JDKs are used with CPUs that support AES-specific 
> instructions (https://en.wikipedia.org/wiki/AES_instruction_set).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2431) Test SSL/TLS impact on performance

2015-10-09 Thread Ben Stopford (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Stopford resolved KAFKA-2431.
-
Resolution: Done

> Test SSL/TLS impact on performance
> --
>
> Key: KAFKA-2431
> URL: https://issues.apache.org/jira/browse/KAFKA-2431
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Ben Stopford
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Test new Producer and new Consumer performance with and without SSL/TLS once 
> the SSL/TLS branch is integrated.
> The ideal scenario is that SSL/TLS would not have an impact if disabled. When 
> enabled, there will be some overhead (encryption and the inability to use 
> `SendFile`) and it will be good to quantify it. The encryption overhead is 
> reduced if recent JDKs are used with CPUs that support AES-specific 
> instructions (https://en.wikipedia.org/wiki/AES_instruction_set).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2431: Easier Testing of SSL

2015-10-09 Thread benstopford

Github user benstopford closed the pull request at:

https://github.com/apache/kafka/pull/217


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-2627) Kafka Heap Size increase impact performance badly

2015-10-09 Thread Mihir Pandya (JIRA)

Mihir Pandya created KAFKA-2627:
---

 Summary: Kafka Heap Size increase impact performance badly
 Key: KAFKA-2627
 URL: https://issues.apache.org/jira/browse/KAFKA-2627
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.2.1
 Environment: CentOS Linux release 7.0.1406 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/;
BUG_REPORT_URL="https://bugs.centos.org/;

CentOS Linux release 7.0.1406 (Core)
CentOS Linux release 7.0.1406 (Core)

Reporter: Mihir Pandya


Initial Kafka server was configured with 

KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"

As we have high resource to utilize, we changed it to below value 

KAFKA_HEAP_OPTS="-Xmx16G -Xms8G"

Change highly impacted Kafka & Zookeeper, we started getting various issue at 
both end.

We were not getting all replica in ISR. And it was an issue with Leader 
Selection which in-turn throwing Socket Connection Error.

To debug, we checked kafaServer-gc.log, we were getting GC(Allocation Failure) 
though we have lot more Memory is avalable.

== GC Error ===
2015-10-08T09:43:08.796+: 4.651: [GC (Allocation Failure) 4.651: [ParNew: 
272640K->7265K(306688K), 0.0277514 secs] 272640K->7265K(1014528K), 0.0281243 
secs] [Times: user=0.03 sys=0.05, real=0.03 secs]
2015-10-08T09:43:11.317+: 7.172: [GC (Allocation Failure) 7.172: [ParNew: 
279905K->3793K(306688K), 0.0157898 secs] 279905K->3793K(1014528K), 0.0159913 
secs] [Times: user=0.03 sys=0.01, real=0.02 secs]
2015-10-08T09:43:13.522+: 9.377: [GC (Allocation Failure) 9.377: [ParNew: 
276433K->2827K(306688K), 0.0064236 secs] 276433K->2827K(1014528K), 0.0066834 
secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2015-10-08T09:43:15.518+: 11.372: [GC (Allocation Failure) 11.373: [ParNew: 
275467K->3090K(306688K), 0.0055454 secs] 275467K->3090K(1014528K), 0.0057979 
secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2015-10-08T09:43:17.558+: 13.412: [GC (Allocation Failure) 13.412: [ParNew: 
275730K->3346K(306688K), 0.0053757 secs] 275730K->3346K(1014528K), 0.0055039 
secs] [Times: user=0.02 sys=0.00, real=0.01 secs]



= Other Kafka Errors =
[2015-10-01 15:35:19,039] INFO conflict in /brokers/ids/3 data: 
{"jmx_port":-1,"timestamp":"1443709506024","host":"","version":1,"port":9092}
 stored data: 
{"jmx_port":-1,"timestamp":"1443702430352","host":"","version":1,"port":9092}
 (kafka.utils.ZkUtils$)
[2015-10-01 15:35:19,042] INFO I wrote this conflicted ephemeral node 
[{"jmx_port":-1,"timestamp":"1443709506024","host":"","version":1,"port":9092}]
 at /brokers/ids/3 a while back in a different session, hence I will backoff 
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)


[2015-10-01 15:23:12,378] INFO Closing socket connection to /172.28.72.162. 
(kafka.network.Processor)
[2015-10-01 15:23:12,378] INFO Closing socket connection to /172.28.72.162. 
(kafka.network.Processor)

[2015-10-01 15:21:53,831] ERROR [ReplicaFetcherThread-4-1], Error for partition 
[workorder-topic,1] to broker 1:class 
kafka.common.NotLeaderForPartitionException (kafka.server.ReplicaFetcherThread)
[2015-10-01 15:21:53,834] ERROR [ReplicaFetcherThread-4-1], Error for partition 
[workorder-topic,1] to broker 1:class 
kafka.common.NotLeaderForPartitionException (kafka.server.ReplicaFetcherThread)
[2015-10-01 15:21:53,835] ERROR [ReplicaFetcherThread-4-1], Error for partition 
[workorder-topic,1] to broker 1:class 
kafka.common.NotLeaderForPartitionException (kafka.server.ReplicaFetcherThread)
[2015-10-01 15:21:53,837] ERROR [ReplicaFetcherThread-4-1], Error for partition 
[workorder-topic,1] to broker 1:class 
kafka.common.NotLeaderForPartitionException (kafka.server.ReplicaFetcherThread)

[2015-10-01 15:20:36,210] WARN [ReplicaFetcherThread-0-2], Error in fetch Name: 
FetchRequest; Version: 0; CorrelationId: 9; ClientId: ReplicaFetcherThread-0-2; 
ReplicaId: 3; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo: 
[__consumer_offsets,17] -> 
PartitionFetchInfo(0,1048576),[__consumer_offsets,23] -> 
PartitionFetchInfo(0,1048576),[__consumer_offsets,29] -> 
PartitionFetchInfo(0,1048576),[__consumer_offsets,35] -> 
PartitionFetchInfo(0,1048576),[__consumer_offsets,41] -> 
PartitionFetchInfo(0,1048576),[__consumer_offsets,5] -> 
PartitionFetchInfo(0,1048576),[__consumer_offsets,11] -> 
PartitionFetchInfo(0,1048576),[__consumer_offsets,47] -> 
PartitionFetchInfo(0,1048576). Possible cause: java.net.SocketTimeoutException 
(kafka.server.ReplicaFetcherThread)
[2015-10-01 15:20:36,210] INFO Reconnect due to socket error: 
java.nio.channels.ClosedChannelException (kafka.consumer.SimpleConsumer)
[2015-10-01

[jira] [Resolved] (KAFKA-2625) Unable to compile scala 2.10 with java 1.6

2015-10-09 Thread Ismael Juma (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-2625.

Resolution: Not A Problem

As I said, this is expected behaviour, so resolving this as "Not a problem".

> Unable to compile scala 2.10 with java 1.6
> --
>
> Key: KAFKA-2625
> URL: https://issues.apache.org/jira/browse/KAFKA-2625
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.9.0.0
> Environment: java version 1.6 javac version 1.6 gradle 2.7 scala 
> 2.10.5 kafka 2.10-0.9.0.0
>Reporter: Arvind Viswanathan
>
> when I issue ./gradlew releaseTarGz -x signArchives, I get the following,
> Building project 'core' with Scala version 2.10.5
> :clients:compileJava FAILED
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task ':clients:compileJava'.
> > invalid source release: 1.7
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or 
> --debug option to get more log output.
> BUILD FAILED
> I can provide more information if needed. Dint change any property file



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2628) KafkaOffsetBackingStoreTest.testGetSet transient test failure

2015-10-09 Thread Ewen Cheslack-Postava (JIRA)

Ewen Cheslack-Postava created KAFKA-2628:


 Summary: KafkaOffsetBackingStoreTest.testGetSet transient test 
failure
 Key: KAFKA-2628
 URL: https://issues.apache.org/jira/browse/KAFKA-2628
 Project: Kafka
  Issue Type: Bug
  Components: copycat
Reporter: Ewen Cheslack-Postava
Assignee: Ewen Cheslack-Postava


{quote}
org.apache.kafka.copycat.storage.KafkaOffsetBackingStoreTest > testGetSet FAILED
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.kafka.copycat.storage.KafkaOffsetBackingStoreTest.testGetSet(KafkaOffsetBackingStoreTest.java:308)
{quote}

Haven't noticed this on Apache's Jenkins yet, but have seen it on Confluent's. 
May be due to limited resources under some conditions, although the timeout is 
already quite generous at 10s.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Confluent Clarification

2015-10-09 Thread Grant Henke

Hi Dinesh,

You may get a response here, as many members of the confluent team read
this list regularly. However, you may want to check the Confluent platform
mailing list to see if your question has been answered before.

https://groups.google.com/forum/#!forum/confluent-platform

Thanks,
Grant

On Fri, Oct 9, 2015 at 8:11 AM, Dinesh J  wrote:

> Hi All,
>
> We are Planning to use Confluent Rest API in our messaging system since we
> don't find any good Dot Net Client.
>
> Is the Kafka and ZooKeeper  delivered by Confluent Zip is same as Apache
> Kafka? Or Changes made in Apache Kafka?
>
> Any one please Confirm this?
>
>
> Thanks,
> Dinesh
>

-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Confluent Clarification

2015-10-09 Thread Dinesh J

Hi All,

We are Planning to use Confluent Rest API in our messaging system since we
don't find any good Dot Net Client.

Is the Kafka and ZooKeeper  delivered by Confluent Zip is same as Apache
Kafka? Or Changes made in Apache Kafka?

Any one please Confirm this?


Thanks,
Dinesh

[jira] [Commented] (KAFKA-2629) Enable getting SSL password from an executable rather than passing plaintext password


[ 
https://issues.apache.org/jira/browse/KAFKA-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950873#comment-14950873
 ] 

Ashish K Singh commented on KAFKA-2629:
---

[~junrao], [~gwenshap], [~sriharsha] thoughts?

> Enable getting SSL password from an executable rather than passing plaintext 
> password
> -
>
> Key: KAFKA-2629
> URL: https://issues.apache.org/jira/browse/KAFKA-2629
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> Currently there are a couple of options to pass SSL passwords to Kafka, i.e., 
> via properties file or via command line argument. Both of these are not 
> recommended security practices.
> * A password on a command line is a no-no: it's trivial to see that password 
> just by using the 'ps' utility.
> * Putting a password into a file, and then passing the location to that file, 
> is the next best option. The access to the file will be governed by unix 
> access permissions which we all know and love. The downside is that the 
> password is still just sitting there in a file, and those who have access can 
> still see it trivially.
> * The most general, secure solution is to provide a layer of abstraction: 
> provide functionality to get the password from "somewhere else".  The most 
> flexible and generic way to do this is to simply call an executable which 
> returns the desired password. 
> ** The executable is again protected with normal file system privileges
> ** The simplest form, a script that looks like "echo 'my-password'", devolves 
> back to putting the password in a file
> ** A more interesting implementation could open up a local encrypted password 
> store and extract the password from it
> ** A maximally secure implementation could contact an external secret manager 
> with centralized control and audit functionality.
> ** In short: getting the password as the output of a script/executable is 
> maximally generic and enables both simple and complex use cases.
> This JIRA intend to add a config param to enable passing an executable to 
> Kafka for SSL passwords.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (KAFKA-2479) Add CopycatExceptions to indicate transient and permanent errors in a connector/task


 [ 
https://issues.apache.org/jira/browse/KAFKA-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-2479 started by Liquan Pei.
-
> Add CopycatExceptions to indicate transient and permanent errors in a 
> connector/task
> 
>
> Key: KAFKA-2479
> URL: https://issues.apache.org/jira/browse/KAFKA-2479
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Liquan Pei
> Fix For: 0.9.0.0
>
>
> Sometimes the connector will need to indicate to the framework that an error 
> occurred, but the error could have multiple responses by the framework.
> For source connectors, there's not much they need to indicate since they can 
> block indefinitely. They probably only need to indicate permanent errors for 
> correctness, though we may want them to indicate transient errors so we can 
> report health of the task in a metric.
> For sink connectors, there are at least a couple of scenarios:
> 1. A task encounters some error while processing a {{put(records)}} call and 
> was unable to fully process it, but thinks it could be resolved in the 
> future. The task doesn't want to see any new records until the issue is 
> resolved, but will need to see the same set of records again. (It would be 
> nice if the task doesn't have to deal with saving these to a buffer itself.)
> 2. A task encounters some error while processing data, but it has 
> enqueued/handled the data passed into the {{put(records)}} call. For example, 
> it may have passed it to some library which buffers it, but then the library 
> indicated that it is having some connection issues. The connector might be 
> able accept more data, but the task is not in a healthy state.
> 3. The task encounters some error that it decides is unrecoverable. This 
> might just be transient errors that repeat for long enough that the task 
> thinks its time to give up. Unclear what to do here, but one option is 
> relocating the task to another worker, hoping that the issue is specific to 
> the worker.
> Note that it is not, generally, safe for sink tasks to do their own backoff 
> or we'd potentially starve the consumer, which needs to poll() in order to 
> heartbeat. So we need to make sure whatever mechanism we implement encourages 
> the user to throw an exception and pass control back to us instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (KAFKA-2484) Add schema projection utilities


 [ 
https://issues.apache.org/jira/browse/KAFKA-2484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-2484 started by Liquan Pei.
-
> Add schema projection utilities
> ---
>
> Key: KAFKA-2484
> URL: https://issues.apache.org/jira/browse/KAFKA-2484
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Liquan Pei
>Priority: Minor
> Fix For: 0.9.0.0
>
>
> Since Copycat has support for versioned schemas and connectors may encounter 
> different versions of the same schema, it will be useful for some connectors 
> to be able to project between different versions of a schema, or have an 
> automatic way to try to project to a target schema (e.g. an existing database 
> table the connector is trying to write data to).
> These utilities should be pretty small because the complex types we support 
> are fairly limited. The primary code required will be for Structs. However, 
> we should take care in designing these utilities since there may be 
> performance implications. For example, when projecting between two schemas, 
> it would be better to come up with a plan object that can efficiently perform 
> the project and be able to reuse that plan many times.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (KAFKA-2480) Handle non-CopycatExceptions from SinkTasks


 [ 
https://issues.apache.org/jira/browse/KAFKA-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-2480 started by Liquan Pei.
-
> Handle non-CopycatExceptions from SinkTasks
> ---
>
> Key: KAFKA-2480
> URL: https://issues.apache.org/jira/browse/KAFKA-2480
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Liquan Pei
> Fix For: 0.9.0.0
>
>
> Currently we catch Throwable in WorkerSinkTask, but we just log the 
> exception. This can lead to data loss because it indicates the messages in 
> the {{put(records)}} call probably were not handled properly. We need to 
> decide what the policy for handling these types of exceptions should be -- 
> try repeating the same records again, risking duplication? or skip them, 
> risking loss? or kill the task immediately and require intervention since 
> it's unclear what happened?
> SourceTasks don't have the same concern -- they can throw other exceptions 
> and as long as we catch them, it is up to the connector to ensure that it 
> does not lose data as a result.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2629) Enable getting SSL password from an executable rather than passing plaintext password

2015-10-09 Thread Sriharsha Chintalapani (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950932#comment-14950932
 ] 

Sriharsha Chintalapani commented on KAFKA-2629:
---

[~singhashish] The distribution of ssl.properties along with a plaintext 
password is been a common way of doing things. In Hadoop they do this as well.  
Not just for ssl in case of kerberos you depend on file system permissions for 
keytabs to keep it secure.  I don't see ssl properties file any different than 
keystore file permissions.
Honestly, I never seen any system doing this so far for SSL. Why do you think 
filesystem permission not suffice and do you have any example anyone else doing 
this.

> Enable getting SSL password from an executable rather than passing plaintext 
> password
> -
>
> Key: KAFKA-2629
> URL: https://issues.apache.org/jira/browse/KAFKA-2629
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> Currently there are a couple of options to pass SSL passwords to Kafka, i.e., 
> via properties file or via command line argument. Both of these are not 
> recommended security practices.
> * A password on a command line is a no-no: it's trivial to see that password 
> just by using the 'ps' utility.
> * Putting a password into a file, and then passing the location to that file, 
> is the next best option. The access to the file will be governed by unix 
> access permissions which we all know and love. The downside is that the 
> password is still just sitting there in a file, and those who have access can 
> still see it trivially.
> * The most general, secure solution is to provide a layer of abstraction: 
> provide functionality to get the password from "somewhere else".  The most 
> flexible and generic way to do this is to simply call an executable which 
> returns the desired password. 
> ** The executable is again protected with normal file system privileges
> ** The simplest form, a script that looks like "echo 'my-password'", devolves 
> back to putting the password in a file
> ** A more interesting implementation could open up a local encrypted password 
> store and extract the password from it
> ** A maximally secure implementation could contact an external secret manager 
> with centralized control and audit functionality.
> ** In short: getting the password as the output of a script/executable is 
> maximally generic and enables both simple and complex use cases.
> This JIRA intend to add a config param to enable passing an executable to 
> Kafka for SSL passwords.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2629) Enable getting SSL password from an executable rather than passing plaintext password

Ashish K Singh created KAFKA-2629:
-

 Summary: Enable getting SSL password from an executable rather 
than passing plaintext password
 Key: KAFKA-2629
 URL: https://issues.apache.org/jira/browse/KAFKA-2629
 Project: Kafka
  Issue Type: Improvement
  Components: security
Affects Versions: 0.9.0.0
Reporter: Ashish K Singh
Assignee: Ashish K Singh


Currently there are a couple of options to pass SSL passwords to Kafka, i.e., 
via properties file or via command line argument. Both of these are not 
recommended security practices.

* A password on a command line is a no-no: it's trivial to see that password 
just by using the 'ps' utility.
* Putting a password into a file, and then passing the location to that file, 
is the next best option. The access to the file will be governed by unix access 
permissions which we all know and love. The downside is that the password is 
still just sitting there in a file, and those who have access can still see it 
trivially.
* The most general, secure solution is to provide a layer of abstraction: 
provide functionality to get the password from "somewhere else".  The most 
flexible and generic way to do this is to simply call an executable which 
returns the desired password. 
** The executable is again protected with normal file system privileges
** The simplest form, a script that looks like "echo 'my-password'", devolves 
back to putting the password in a file
** A more interesting implementation could open up a local encrypted password 
store and extract the password from it
** A maximally secure implementation could contact an external secret manager 
with centralized control and audit functionality.
** In short: getting the password as the output of a script/executable is 
maximally generic and enables both simple and complex use cases.

This JIRA intend to add a config param to enable passing an executable to Kafka 
for SSL passwords.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-2369) Add Copycat REST API


 [ 
https://issues.apache.org/jira/browse/KAFKA-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liquan Pei reassigned KAFKA-2369:
-

Assignee: Liquan Pei  (was: Ewen Cheslack-Postava)

> Add Copycat REST API
> 
>
> Key: KAFKA-2369
> URL: https://issues.apache.org/jira/browse/KAFKA-2369
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Liquan Pei
> Fix For: 0.9.0.0
>
>
> Add a REST API for Copycat. At a minimum, for a single worker this should 
> support:
> * add/remove connector
> * connector status
> * task status
> * worker status
> In distributed mode this should handle forwarding if necessary, but it may 
> make sense to defer the distributed support for a later JIRA.
> This will require the addition of new dependencies to support implementing 
> the REST API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-2629) Enable getting SSL password from an executable rather than passing plaintext password

2015-10-09 Thread Sriharsha Chintalapani (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950932#comment-14950932
 ] 

Sriharsha Chintalapani edited comment on KAFKA-2629 at 10/9/15 6:30 PM:


[~singhashish] The distribution of ssl.properties along with a plaintext 
password is been a common way of doing things. In Hadoop they do this as well.  
Not just for ssl in case of kerberos you depend on file system permissions for 
keytabs to keep it secure.  I don't see ssl properties file any different than 
keystore file permissions.
Honestly, I never seen any system doing this so far for SSL. Why do you think 
filesystem permission not suffice and do you have any example anyone else doing 
this.

In your proposal you are saying an executable is also protected by same file 
system permissions than how it is providing any additional security ?. 


was (Author: sriharsha):
[~singhashish] The distribution of ssl.properties along with a plaintext 
password is been a common way of doing things. In Hadoop they do this as well.  
Not just for ssl in case of kerberos you depend on file system permissions for 
keytabs to keep it secure.  I don't see ssl properties file any different than 
keystore file permissions.
Honestly, I never seen any system doing this so far for SSL. Why do you think 
filesystem permission not suffice and do you have any example anyone else doing 
this.

> Enable getting SSL password from an executable rather than passing plaintext 
> password
> -
>
> Key: KAFKA-2629
> URL: https://issues.apache.org/jira/browse/KAFKA-2629
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> Currently there are a couple of options to pass SSL passwords to Kafka, i.e., 
> via properties file or via command line argument. Both of these are not 
> recommended security practices.
> * A password on a command line is a no-no: it's trivial to see that password 
> just by using the 'ps' utility.
> * Putting a password into a file, and then passing the location to that file, 
> is the next best option. The access to the file will be governed by unix 
> access permissions which we all know and love. The downside is that the 
> password is still just sitting there in a file, and those who have access can 
> still see it trivially.
> * The most general, secure solution is to provide a layer of abstraction: 
> provide functionality to get the password from "somewhere else".  The most 
> flexible and generic way to do this is to simply call an executable which 
> returns the desired password. 
> ** The executable is again protected with normal file system privileges
> ** The simplest form, a script that looks like "echo 'my-password'", devolves 
> back to putting the password in a file
> ** A more interesting implementation could open up a local encrypted password 
> store and extract the password from it
> ** A maximally secure implementation could contact an external secret manager 
> with centralized control and audit functionality.
> ** In short: getting the password as the output of a script/executable is 
> maximally generic and enables both simple and complex use cases.
> This JIRA intend to add a config param to enable passing an executable to 
> Kafka for SSL passwords.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2622: Add Time logical type for Copycat.

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/285


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2622) Add Time logical type for Copycat


[ 
https://issues.apache.org/jira/browse/KAFKA-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950995#comment-14950995
 ] 

ASF GitHub Bot commented on KAFKA-2622:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/285


> Add Time logical type for Copycat
> -
>
> Key: KAFKA-2622
> URL: https://issues.apache.org/jira/browse/KAFKA-2622
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> KAFKA-2476 defined Decimal, Date, and Timestamp types. Initially I didn't 
> include a separate Time (time of day) type because I was trying to keep the 
> number of types small. However, I realized that the JDBC connector needs this 
> to round out its support for SQL types.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2622) Add Time logical type for Copycat

2015-10-09 Thread Guozhang Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2622:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 285
[https://github.com/apache/kafka/pull/285]

> Add Time logical type for Copycat
> -
>
> Key: KAFKA-2622
> URL: https://issues.apache.org/jira/browse/KAFKA-2622
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> KAFKA-2476 defined Decimal, Date, and Timestamp types. Initially I didn't 
> include a separate Time (time of day) type because I was trying to keep the 
> number of types small. However, I realized that the JDBC connector needs this 
> to round out its support for SQL types.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2615) Poll() method is broken wrt time

2015-10-09 Thread Eno Thereska (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eno Thereska updated KAFKA-2615:

   Labels: patch  (was: )
 Reviewer: Guozhang Wang
Fix Version/s: 0.8.2.1
   Status: Patch Available  (was: Open)

GitHub user enothereska opened a pull request:
https://github.com/apache/kafka/pull/290
KAFKA-2459: connection backoff, timeouts and retries
This fix applies to three JIRAs, since they are all connected.
KAFKA-2459Connection backoff/blackout period should start when a connection is 
disconnected, not when the connection attempt was initiated
Backoff when connection is disconnected
KAFKA-2615Poll() method is broken wrt time
Added Time through the NetworkClient API. Minimal change.
KAFKA-1843Metadata fetch/refresh in new producer should handle all node 
connection states gracefully
I’ve partially addressed this for a specific failure case in the JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/enothereska/kafka trunk
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/290.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #290

> Poll() method is broken wrt time
> 
>
> Key: KAFKA-2615
> URL: https://issues.apache.org/jira/browse/KAFKA-2615
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer, producer 
>Affects Versions: 0.8.2.1
>Reporter: Eno Thereska
>Assignee: Eno Thereska
>  Labels: patch
> Fix For: 0.8.2.1
>
>
> Initially reported by [~ewencp] and discussed with [~hachikuji]. In 
> NetworkClient.java, the poll() method receives as input a "now" parameter, 
> does a whole bunch of work (e.g., selector.poll()) and then keeps using "now" 
> in all the subsequent method calls. 
> Passing Time everywhere instead of "now" is a potential fix, but might be 
> expensive since it's a new system call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1843) Metadata fetch/refresh in new producer should handle all node connection states gracefully

2015-10-09 Thread Eno Thereska (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eno Thereska updated KAFKA-1843:

   Labels: patch  (was: )
 Reviewer: Guozhang Wang
Fix Version/s: 0.8.2.1
   Status: Patch Available  (was: Open)

GitHub user enothereska opened a pull request:
https://github.com/apache/kafka/pull/290
KAFKA-2459: connection backoff, timeouts and retries
This fix applies to three JIRAs, since they are all connected.
KAFKA-2459Connection backoff/blackout period should start when a connection is 
disconnected, not when the connection attempt was initiated
Backoff when connection is disconnected
KAFKA-2615Poll() method is broken wrt time
Added Time through the NetworkClient API. Minimal change.
KAFKA-1843Metadata fetch/refresh in new producer should handle all node 
connection states gracefully
I’ve partially addressed this for a specific failure case in the JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/enothereska/kafka trunk
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/290.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #290

> Metadata fetch/refresh in new producer should handle all node connection 
> states gracefully
> --
>
> Key: KAFKA-1843
> URL: https://issues.apache.org/jira/browse/KAFKA-1843
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, producer 
>Affects Versions: 0.8.2.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Eno Thereska
>Priority: Blocker
>  Labels: patch
> Fix For: 0.8.2.1
>
>
> KAFKA-1642 resolved some issues with the handling of broker connection states 
> to avoid high CPU usage, but made the minimal fix rather than the ideal one. 
> The code for handling the metadata fetch is difficult to get right because it 
> has to handle a lot of possible connectivity states and failure modes across 
> all the known nodes. It also needs to correctly integrate with the 
> surrounding event loop, providing correct poll() timeouts to both avoid busy 
> looping and make sure it wakes up and tries new nodes in the face of both 
> connection and request failures.
> A patch here should address a few issues:
> 1. Make sure connection timeouts, as implemented in KAFKA-1842, are cleanly 
> integrated. This mostly means that when a connecting node is selected to 
> fetch metadata from, that the code notices that and sets the next timeout 
> based on the connection timeout rather than some other backoff.
> 2. Rethink the logic and naming of NetworkClient.leastLoadedNode. That method 
> actually takes into account a) the current connectivity of each node, b) 
> whether the node had a recent connection failure, c) the "load" in terms of 
> in flight requests. It also needs to ensure that different clients don't use 
> the same ordering across multiple calls (which is already addressed in the 
> current code by nodeIndexOffset) and that we always eventually try all nodes 
> in the face of connection failures (which isn't currently handled by 
> leastLoadedNode and probably cannot be without tracking additional state). 
> This method also has to work for new consumer use cases even though it is 
> currently only used by the new producer's metadata fetch. Finally it has to 
> properly handle when other code calls initiateConnect() since the normal path 
> for sending messages also initiates connections.
> We can already say that there is an order of preference given a single call 
> (as follows), but making this work across multiple calls when some initial 
> choices fail to connect or return metadata *and* connection states may be 
> changing is much more difficult.
>  * Connected, zero in flight requests - the request can be sent immediately
>  * Connecting node - it will hopefully be connected very soon and by 
> definition has no in flight requests
>  * Disconnected - same reasoning as for a connecting node
>  * Connected, > 0 in flight requests - we consider any # of in flight 
> requests as a big enough backlog to delay the request a lot.
> We could use an approach that better accounts for # of in flight requests 
> rather than just turning it into a boolean variable, but that probably 
> introduces much more complexity than it is worth.
> 3. The most difficult case to handle so far has been when leastLoadedNode 
> returns a disconnected node to maybeUpdateMetadata as its best option. 
> Properly handling the two resulting cases (initiateConnect fails immediately 
> vs. taking some time to possibly establish the connection) is tricky.
> 4. Consider optimizing

[GitHub] kafka pull request: KAFKA-2600 Align Kafka Streams' interfaces wit...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/270


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2600) Make KStream interfaces compatible with Java 8 java.util.function


[ 
https://issues.apache.org/jira/browse/KAFKA-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951036#comment-14951036
 ] 

ASF GitHub Bot commented on KAFKA-2600:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/270


> Make KStream interfaces compatible with Java 8 java.util.function
> -
>
> Key: KAFKA-2600
> URL: https://issues.apache.org/jira/browse/KAFKA-2600
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Randall Hauch
> Fix For: 0.9.0.0
>
>
> As suggested by [~rhauch], if we make the interface method names as the same 
> to java.util.function.[Functions]:
> https://docs.oracle.com/javase/8/docs/api/java/util/function/package-summary.html
> Our goal is to simply align names and concepts with those in the Java 8 API 
> for ease of learning and use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2600) Make KStream interfaces compatible with Java 8 java.util.function

2015-10-09 Thread Guozhang Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2600:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 270
[https://github.com/apache/kafka/pull/270]

> Make KStream interfaces compatible with Java 8 java.util.function
> -
>
> Key: KAFKA-2600
> URL: https://issues.apache.org/jira/browse/KAFKA-2600
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Randall Hauch
> Fix For: 0.9.0.0
>
>
> As suggested by [~rhauch], if we make the interface method names as the same 
> to java.util.function.[Functions]:
> https://docs.oracle.com/javase/8/docs/api/java/util/function/package-summary.html
> Our goal is to simply align names and concepts with those in the Java 8 API 
> for ease of learning and use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: Fix exception message in Copycat's Time...

2015-10-09 Thread ewencp

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/291

MINOR: Fix exception message in Copycat's Time logical type.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka fixup-time-logical-type

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/291.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #291


commit 47a4c180d4d8724d0217d2d529864c50006f3fa4
Author: Ewen Cheslack-Postava 
Date:   2015-10-09T19:41:21Z

MINOR: Fix exception message in Copycat's Time logical type.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: MINOR: Fix exception message in Copycat's Time...