[jira] [Commented] (KAFKA-5915) Support unmapping of mapped/direct buffers in Java 9

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168769#comment-16168769
 ] 

ASF GitHub Bot commented on KAFKA-5915:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/3879

KAFKA-5915: Support unmapping of mapped/direct buffers in Java 9

As mentioned in MappedByteBuffers' class documentation, its
implementation was inspired by Lucene's MMapDirectory:


https://github.com/apache/lucene-solr/blob/releases/lucene-solr/6.6.1/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L315

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-5915-unmap-mapped-buffers-java-9

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3879.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3879


commit e168f970ed726524b1d7f6ec70dfb4bfff8da754
Author: Ismael Juma 
Date:   2017-09-16T02:05:50Z

KAFKA-5915: Support unmapping of mapped/direct buffers in Java 9




> Support unmapping of mapped/direct buffers in Java 9
> 
>
> Key: KAFKA-5915
> URL: https://issues.apache.org/jira/browse/KAFKA-5915
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 1.0.0
>
>
> This currently fails with:
> {code}
> java.lang.IllegalAccessError: class kafka.log.AbstractIndex (in unnamed 
> module @0x45103d6b) cannot access class jdk.internal.ref.Cleaner (in module 
> java.base) because module java.base does not export jdk.internal.ref to 
> unnamed module @0x45103d6b
> {code}
> A commit that shows how Lucene changed their code to run without warnings: 
> https://github.com/apache/lucene-solr/commit/7e03427fa14a024ce257babcb8362d2451941e21



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (KAFKA-5883) Run tests on Java 9 with –illegal-access=deny

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164635#comment-16164635
 ] 

Ismael Juma edited comment on KAFKA-5883 at 9/16/17 1:55 AM:
-

A few changes that are likely to be required:

1. Accessing `sun.security.krb5.Config` for the `getDefaultRealm` and `refresh` 
methods.
2. Upgrade to newer versions of EasyMock and PowerMock.


was (Author: ijuma):
A few changes that are likely to be required:

1. Accessing `sun.nio.ch.DirectBuffer` in `AbstractIndex`. A commit that shows 
how Lucene changed their code to run without warnings: 
https://github.com/apache/lucene-solr/commit/7e03427fa14a024ce257babcb8362d2451941e21
2. Accessing `sun.security.krb5.Config` for the `getDefaultRealm` method.
3. Upgrade to newer versions of EasyMock and PowerMock.

> Run tests on Java 9 with –illegal-access=deny
> -
>
> Key: KAFKA-5883
> URL: https://issues.apache.org/jira/browse/KAFKA-5883
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
> Fix For: 1.1.0
>
>
> The default was changed from –illegal-access=deny to –illegal-access=warn 
> late in the Java 9 cycle. By using the former, we will ensure that our code 
> is not relying on functionality that will be removed in a future Java version.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5890) records.lag should use tags for topic and partition rather than using metric name.

[James Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168759#comment-16168759
 ] 

James Cheng commented on KAFKA-5890:


I noticed the same thing in https://github.com/apache/kafka/pull/2993

Metric names are a public API, and so would require a KIP before changes could 
be made. Is that something you want to take on?

> records.lag should use tags for topic and partition rather than using metric 
> name.
> --
>
> Key: KAFKA-5890
> URL: https://issues.apache.org/jira/browse/KAFKA-5890
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.2.0
>Reporter: Charly Molter
>
> As part of KIP-92[1] a per partition lag metric was added.
> These metrics are really useful, however in the implementation  it was 
> implemented as a prefix to the metric name: 
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L1321-L1344
> Usually these kind of metrics use tags and the name is constant for all 
> topics, partitions.
> We have a custom reporter which aggregates topics/partitions together to 
> avoid explosion of the number of KPIs and this KPI doesn't support this as it 
> doesn't have tags but a complex name.
> [1] 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5906) Change metric.reporters configuration key to metrics.reporters to be consistent

[Kevin Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lu updated KAFKA-5906:

Description: 
The "metric.reporters" configuration key should be consistent with the actual 
classes. Clients have a MetricsReporter.class while the broker has a 
KafkaMetricsReporter.class. 

We have seen quite a few people configure this field incorrectly by setting it 
as "metrics.reporters".

The configuration key could be renamed to "metrics.reporters" to match the 
classes, or the classes can be renamed to MetricReporter.class and 
KafkaMetricReporter.class.

The broker configuration description for "metric.reporters" also mentions 
MetricReporter, but KafkaMetricsReporter is the actual interface to implement. 

There also seems to be a discrepancy with "MetricReporter" in the description 
as the class name is actually "MetricsReporter". 
https://github.com/apache/kafka/pull/3875

  was:
The "metric.reporters" configuration key should be consistent with the actual 
classes. Clients have a MetricsReporter.class while the broker has a 
KafkaMetricsReporter.class. 

We have seen quite a few people configure this field incorrectly by setting it 
as "metrics.reporters".

The configuration key could be renamed to "metrics.reporters" to match the 
classes, or the classes can be renamed to MetricReporter.class and 
KafkaMetricReporter.class.

The broker configuration description for "metric.reporters" also mentions 
MetricReporter, but KafkaMetricsReporter is the actual interface to implement. 

There also seems to be a discrepancy with "MetricReporter" in the description 
as the class name is actually "MetricsReporter". 
https://github.com/apache/kafka/pull/3867


> Change metric.reporters configuration key to metrics.reporters to be 
> consistent
> ---
>
> Key: KAFKA-5906
> URL: https://issues.apache.org/jira/browse/KAFKA-5906
> Project: Kafka
>  Issue Type: Improvement
>  Components: config, metrics
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: usability
>
> The "metric.reporters" configuration key should be consistent with the actual 
> classes. Clients have a MetricsReporter.class while the broker has a 
> KafkaMetricsReporter.class. 
> We have seen quite a few people configure this field incorrectly by setting 
> it as "metrics.reporters".
> The configuration key could be renamed to "metrics.reporters" to match the 
> classes, or the classes can be renamed to MetricReporter.class and 
> KafkaMetricReporter.class.
> The broker configuration description for "metric.reporters" also mentions 
> MetricReporter, but KafkaMetricsReporter is the actual interface to 
> implement. 
> There also seems to be a discrepancy with "MetricReporter" in the description 
> as the class name is actually "MetricsReporter". 
> https://github.com/apache/kafka/pull/3875



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (KAFKA-2376) Add Kafka Connect metrics

[James Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168752#comment-16168752
 ] 

James Cheng edited comment on KAFKA-2376 at 9/16/17 1:25 AM:
-

[~rhauch]: I'm not a Committer, obviously, but feel free to add me to any PRs 
if you want an extra pair of eyes.


was (Author: wushujames):
[~rhauch]: I'm not a committer, obviously, but feel free to add me to any PRs 
if you want an extra pair of eyes.

> Add Kafka Connect metrics
> -
>
> Key: KAFKA-2376
> URL: https://issues.apache.org/jira/browse/KAFKA-2376
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Affects Versions: 0.9.0.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Randall Hauch
>  Labels: needs-kip
>
> Kafka Connect needs good metrics for monitoring since that will be the 
> primary insight into the health of connectors as they copy data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-2376) Add Kafka Connect metrics

[James Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168752#comment-16168752
 ] 

James Cheng commented on KAFKA-2376:


[~rhauch]: I'm not a committer, obviously, but feel free to add me to any PRs 
if you want an extra pair of eyes.

> Add Kafka Connect metrics
> -
>
> Key: KAFKA-2376
> URL: https://issues.apache.org/jira/browse/KAFKA-2376
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Affects Versions: 0.9.0.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Randall Hauch
>  Labels: needs-kip
>
> Kafka Connect needs good metrics for monitoring since that will be the 
> primary insight into the health of connectors as they copy data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5906) Change metric.reporters configuration key to metrics.reporters to be consistent

[James Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168750#comment-16168750
 ] 

James Cheng commented on KAFKA-5906:


I believe these are public APIs (the config keys, as well as the 
interfaces/classes) and therefore would require a KIP before being able to 
change them.

Fixing the docs, though, is doable without a KIP. Good catch!

> Change metric.reporters configuration key to metrics.reporters to be 
> consistent
> ---
>
> Key: KAFKA-5906
> URL: https://issues.apache.org/jira/browse/KAFKA-5906
> Project: Kafka
>  Issue Type: Improvement
>  Components: config, metrics
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: usability
>
> The "metric.reporters" configuration key should be consistent with the actual 
> classes. Clients have a MetricsReporter.class while the broker has a 
> KafkaMetricsReporter.class. 
> We have seen quite a few people configure this field incorrectly by setting 
> it as "metrics.reporters".
> The configuration key could be renamed to "metrics.reporters" to match the 
> classes, or the classes can be renamed to MetricReporter.class and 
> KafkaMetricReporter.class.
> The broker configuration description for "metric.reporters" also mentions 
> MetricReporter, but KafkaMetricsReporter is the actual interface to 
> implement. 
> There also seems to be a discrepancy with "MetricReporter" in the description 
> as the class name is actually "MetricsReporter". 
> https://github.com/apache/kafka/pull/3867



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5913) Add RecordMetadataNotAvailableException to indicate that ProduceResponse did not contain offset and timestamp information

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168748#comment-16168748
 ] 

ASF GitHub Bot commented on KAFKA-5913:
---

GitHub user apurvam opened a pull request:

https://github.com/apache/kafka/pull/3878

KAFKA-5913: Add the RecordMetadataNotAvailableException

We return this exception from `RecordMetadata.offset()` or 
`RecordMetadata.timestamp()` if these pieces of metadata were not returned by 
the broker. 

This will happen, for instance, when the broker returns a 
`DuplicateSequenceException`.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apurvam/kafka 
KAFKA-5913-add-record-metadata-not-available-exception

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3878.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3878


commit e9770a65203f7bdcea8c25a5fbaaa9366d12851c
Author: Apurva Mehta 
Date:   2017-09-16T01:15:15Z

Initial commit




> Add RecordMetadataNotAvailableException to indicate that ProduceResponse did 
> not contain offset and timestamp information
> -
>
> Key: KAFKA-5913
> URL: https://issues.apache.org/jira/browse/KAFKA-5913
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
> Fix For: 1.0.0
>
>
> One of the changes in KIP-192 is to send a {{DUPLICATE_SEQUENCE}} error code 
> with a {{ProduceResponse}} when we detect a duplicate on the broker but don't 
> have the batch metadata for the sequence in question in memory.
> To handle this on the client, we mark the batch as successful, but cannot 
> return the offset and timestamp information in the {{RecordMetadata}} 
> returned in the produce future. Instead of returning implicit invalid values 
> (like -1), we should throw a {{RecordMetadataNotAvailableException}} to 
> ensure that applications don't suffer from faulty processing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5914) Return MessageFormatVersion and MessageMaxBytes in MetadataResponse

[Apurva Mehta (JIRA)]

Apurva Mehta created KAFKA-5914:
---

 Summary: Return MessageFormatVersion and MessageMaxBytes in 
MetadataResponse
 Key: KAFKA-5914
 URL: https://issues.apache.org/jira/browse/KAFKA-5914
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apurva Mehta
Assignee: Apurva Mehta
 Fix For: 1.0.0


As part of KIP-192, we want to send two additional fields in the 
{{TopicMetadata}} which is part of the {{MetadataResponse}}. These fields are 
the {{MessageFormatVersion}} and the {{MessageMaxBytes}}.

The {{MessageFormatVersion}} is required to implement 
https://issues.apache.org/jira/browse/KAFKA-5794 . The latter will be 
implemented in a future release, but with the changes proposed here, the said 
future release will be backward compatible with 1.0.0



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5794) Introduce new idempotence mode to gracefully deal with topics on the older message format

[Apurva Mehta (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168737#comment-16168737
 ] 

Apurva Mehta commented on KAFKA-5794:
-

Bumping this to a future release, we will just add the message format version 
to the {{TopicMetadata}} returned in {{MetadatResponse}} for this release. This 
will be tracked separately.

> Introduce new idempotence mode to gracefully deal with topics on the older 
> message format
> -
>
> Key: KAFKA-5794
> URL: https://issues.apache.org/jira/browse/KAFKA-5794
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.11.0.0
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>
> In the discussion of KIP-185: Make exactly once in order delivery per 
> partition the default producer setting, it was realized that we don't have 
> graceful handling when an idempotence-enabled producer is writing to a broker 
> with a message format older than v2 (ie. the 0.11.0 message format). 
> In particular, if we enable idempotence, any produce requests to topics with 
> an older message format will fail with an UnsupportedVersionException. Thus 
> if the idempotent producer was to be made the default, the out of the box 
> producer would fail to produce when used with clusters which haven't upgraded 
> the message format yet.
> This is particularly problematic since the recommended upgrade path is to 
> upgrade broker code while keeping the message format at the older version, 
> then upgrade all clients, and only finally upgrade the message format on the 
> server. With the current behavior, the middle step is actually untenable if 
> we enable idempotence as the default.
> More details available at: 
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Exactly+Once+-+Dealing+with+older+message+formats+when+idempotence+is+enabled



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5794) Introduce new idempotence mode to gracefully deal with topics on the older message format

[Apurva Mehta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apurva Mehta updated KAFKA-5794:

Fix Version/s: (was: 1.0.0)

> Introduce new idempotence mode to gracefully deal with topics on the older 
> message format
> -
>
> Key: KAFKA-5794
> URL: https://issues.apache.org/jira/browse/KAFKA-5794
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.11.0.0
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>
> In the discussion of KIP-185: Make exactly once in order delivery per 
> partition the default producer setting, it was realized that we don't have 
> graceful handling when an idempotence-enabled producer is writing to a broker 
> with a message format older than v2 (ie. the 0.11.0 message format). 
> In particular, if we enable idempotence, any produce requests to topics with 
> an older message format will fail with an UnsupportedVersionException. Thus 
> if the idempotent producer was to be made the default, the out of the box 
> producer would fail to produce when used with clusters which haven't upgraded 
> the message format yet.
> This is particularly problematic since the recommended upgrade path is to 
> upgrade broker code while keeping the message format at the older version, 
> then upgrade all clients, and only finally upgrade the message format on the 
> server. With the current behavior, the middle step is actually untenable if 
> we enable idempotence as the default.
> More details available at: 
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Exactly+Once+-+Dealing+with+older+message+formats+when+idempotence+is+enabled



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5913) Add RecordMetadataNotAvailableException to indicate that ProduceResponse did not contain offset and timestamp information

[Apurva Mehta (JIRA)]

Apurva Mehta created KAFKA-5913:
---

 Summary: Add RecordMetadataNotAvailableException to indicate that 
ProduceResponse did not contain offset and timestamp information
 Key: KAFKA-5913
 URL: https://issues.apache.org/jira/browse/KAFKA-5913
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apurva Mehta
Assignee: Apurva Mehta
 Fix For: 1.0.0


One of the changes in KIP-192 is to send a {{DUPLICATE_SEQUENCE}} error code 
with a {{ProduceResponse}} when we detect a duplicate on the broker but don't 
have the batch metadata for the sequence in question in memory.

To handle this on the client, we mark the batch as successful, but cannot 
return the offset and timestamp information in the {{RecordMetadata}} returned 
in the produce future. Instead of returning implicit invalid values (like -1), 
we should throw a {{RecordMetadataNotAvailableException}} to ensure that 
applications don't suffer from faulty processing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5765) Move merge() from StreamsBuilder to KStream

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168644#comment-16168644
 ] 

Guozhang Wang commented on KAFKA-5765:
--

Richard, I have granted you the permission, cheers.

> Move merge() from StreamsBuilder to KStream
> ---
>
> Key: KAFKA-5765
> URL: https://issues.apache.org/jira/browse/KAFKA-5765
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0, 1.0.0
>Reporter: Matthias J. Sax
>  Labels: needs-kip, newbie
> Fix For: 1.1.0
>
>
> Merging multiple {{KStream}} is done via {{StreamsBuilder#merge()}} (formally 
> {{KStreamBuilder#merge()}}). This is quite unnatural and should be done via 
> {{KStream#merge()}}.
> We need a KIP as we add a new method to a public {{KStreams}} API and 
> deprecate the old {{merge()}} method.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5765) Move merge() from StreamsBuilder to KStream

[Richard Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168623#comment-16168623
 ] 

Richard Yu commented on KAFKA-5765:
---

I have requested for wiki edit access, but that is still forthcoming. (My wiki 
handle is yohan.richard.yu)


> Move merge() from StreamsBuilder to KStream
> ---
>
> Key: KAFKA-5765
> URL: https://issues.apache.org/jira/browse/KAFKA-5765
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0, 1.0.0
>Reporter: Matthias J. Sax
>  Labels: needs-kip, newbie
> Fix For: 1.1.0
>
>
> Merging multiple {{KStream}} is done via {{StreamsBuilder#merge()}} (formally 
> {{KStreamBuilder#merge()}}). This is quite unnatural and should be done via 
> {{KStream#merge()}}.
> We need a KIP as we add a new method to a public {{KStreams}} API and 
> deprecate the old {{merge()}} method.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5792) Transient failure in KafkaAdminClientTest.testHandleTimeout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168445#comment-16168445
 ] 

ASF GitHub Bot commented on KAFKA-5792:
---

GitHub user apurvam opened a pull request:

https://github.com/apache/kafka/pull/3877

MINOR: Disable KafkaAdminClientTest.testHandleTimeout

This test is super flaky in the PR builder. 
https://issues.apache.org/jira/browse/KAFKA-5792 tracks the fix.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apurvam/kafka 
MINOR-disable-adminclient-timeout-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3877.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3877


commit d533e0c598fa5a96591ff46c20df29897596d250
Author: Apurva Mehta 
Date:   2017-09-15T20:09:51Z

Disable KafkaAdminClientTest.testHandleTimeout




> Transient failure in KafkaAdminClientTest.testHandleTimeout
> ---
>
> Key: KAFKA-5792
> URL: https://issues.apache.org/jira/browse/KAFKA-5792
> Project: Kafka
>  Issue Type: Bug
>Reporter: Apurva Mehta
>Assignee: Colin P. McCabe
>  Labels: transient-unit-test-failure
> Fix For: 1.0.0
>
>
> The {{KafkaAdminClientTest.testHandleTimeout}} test occasionally fails with 
> the following:
> {noformat}
> java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node 
> assignment.
>   at 
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
>   at 
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
>   at 
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
>   at 
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:213)
>   at 
> org.apache.kafka.clients.admin.KafkaAdminClientTest.testHandleTimeout(KafkaAdminClientTest.java:356)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting 
> for a node assignment.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5912) Trogdor AgentTest.testAgentActivatesFaults is flaky

[Apurva Mehta (JIRA)]

Apurva Mehta created KAFKA-5912:
---

 Summary: Trogdor AgentTest.testAgentActivatesFaults is flaky
 Key: KAFKA-5912
 URL: https://issues.apache.org/jira/browse/KAFKA-5912
 Project: Kafka
  Issue Type: Bug
Reporter: Apurva Mehta
Assignee: Colin P. McCabe


I have seen the the following failures occasionally in the PR builder.

{noformat}
Error Message

java.lang.AssertionError: Condition not met within timeout 15000. Timed out 
waiting for expected fault specs {bar: {state: 
{"stateName":"done","doneMs":7,"errorStr":""}}, baz: {state: 
{"stateName":"running","startedMs":7}}, foo: {state: 
{"stateName":"done","doneMs":3,"errorStr":""}}}
Stacktrace

java.lang.AssertionError: Condition not met within timeout 15000. Timed out 
waiting for expected fault specs {bar: {state: 
{"stateName":"done","doneMs":7,"errorStr":""}}, baz: {state: 
{"stateName":"running","startedMs":7}}, foo: {state: 
{"stateName":"done","doneMs":3,"errorStr":""}}}
at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:275)
at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:253)
at 
org.apache.kafka.trogdor.common.ExpectedFaults.waitFor(ExpectedFaults.java:119)
at 
org.apache.kafka.trogdor.common.ExpectedFaults.waitFor(ExpectedFaults.java:109)
at 
org.apache.kafka.trogdor.agent.AgentTest.testAgentActivatesFaults(AgentTest.java:152)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)

{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5911) Avoid creation of extra Map for futures in KafkaAdminClient

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated KAFKA-5911:
--
Attachment: 5911.v1.txt

I have run KafkaAdminClientTest thru the patch which passed.

> Avoid creation of extra Map for futures in KafkaAdminClient
> ---
>
> Key: KAFKA-5911
> URL: https://issues.apache.org/jira/browse/KAFKA-5911
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ted Yu
> Attachments: 5911.v1.txt
>
>
> In various methods from KafkaAdminClient, there is extra Map created when 
> constructing XXResult instance.
> e.g.
> {code}
> return new DescribeReplicaLogDirResult(new 
> HashMap(futures));
> {code}
> Prior to returning, futures Map is already filled.
> Calling get() and values() does not involve the internals of HashMap when we 
> consider thread-safety.
> The extra Map doesn't need to be created.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5911) Avoid creation of extra Map for futures in KafkaAdminClient

[Ted Yu (JIRA)]

Ted Yu created KAFKA-5911:
-

 Summary: Avoid creation of extra Map for futures in 
KafkaAdminClient
 Key: KAFKA-5911
 URL: https://issues.apache.org/jira/browse/KAFKA-5911
 Project: Kafka
  Issue Type: Bug
Reporter: Ted Yu


In various methods from KafkaAdminClient, there is extra Map created when 
constructing XXResult instance.
e.g.
{code}
return new DescribeReplicaLogDirResult(new 
HashMap(futures));
{code}
Prior to returning, futures Map is already filled.
Calling get() and values() does not involve the internals of HashMap when we 
consider thread-safety.

The extra Map doesn't need to be created.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5896) Kafka Connect task threads never interrupted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168318#comment-16168318
 ] 

ASF GitHub Bot commented on KAFKA-5896:
---

GitHub user 56quarters opened a pull request:

https://github.com/apache/kafka/pull/3876

Force Connect tasks to stop via thread interruption after a timeout

Interrupt the thread of Kafka Connect tasks that do not stop within
the timeout via `Worker::stopAndAwaitTasks()`. Previously tasks would
be asked to stop via setting a `stopping` flag. It was possible for
tasks to ignore this flag if they were, for example, waiting for
a lock or blocked on I/O.

This prevents issues where tasks may end up with multiple threads
all running and attempting to make progress when there should only
be a single thread running for that task at a time.

Fixes KAFKA-5896

/cc @rhauch @tedyu 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/smarter-travel-media/kafka force-task-stop

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3876.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3876


commit 31c879c1a1f0bd4f5999c021baca8e99e733ffe1
Author: Nick Pillitteri 
Date:   2017-09-13T14:54:40Z

Force Connect tasks to stop via thread interruption after a timeout

Interrupt the thread of Kafka Connect tasks that do not stop within
the timeout via Worker::stopAndAwaitTasks(). Previously tasks would
be asked to stop via setting a `stopping` flag. It was possible for
tasks to ignore this flag if they were, for example, waiting for
a lock or blocked on I/O.

This prevents issues where tasks may end up with multiple threads
all running and attempting to make progress when there should only
be a single thread running for that task at a time.

Fixes KAFKA-5896




> Kafka Connect task threads never interrupted
> 
>
> Key: KAFKA-5896
> URL: https://issues.apache.org/jira/browse/KAFKA-5896
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Nick Pillitteri
>Priority: Minor
>
> h2. Problem
> Kafka Connect tasks associated with connectors are run in their own threads. 
> When tasks are stopped or restarted, a flag is set - {{stopping}} - to 
> indicate the task should stop processing records. However, if the thread the 
> task is running in is blocked (waiting for a lock or performing I/O) it's 
> possible the task will never stop.
> I've created a connector specifically to demonstrate this issue (along with 
> some more detailed instructions for reproducing the issue): 
> https://github.com/smarter-travel-media/hang-connector
> I believe this is an issue because it means that a single badly behaved 
> connector (any connector that does I/O without timeouts) can cause the Kafka 
> Connect worker to get into a state where the only solution is to restart the 
> JVM.
> I think, but couldn't reproduce, that this is the cause of this problem on 
> Stack Overflow: 
> https://stackoverflow.com/questions/43802156/inconsistent-connector-state-connectexception-task-already-exists-in-this-work
> h2. Expected Result
> I would expect the Worker to eventually interrupt the thread that the task is 
> running in. In the past across various other libraries, this is what I've 
> seen done when a thread needs to be forcibly stopped.
> h2. Actual Result
> In actuality, the Worker sets a {{stopping}} flag and lets the thread run 
> indefinitely. It uses a timeout while waiting for the task to stop but after 
> this timeout has expired it simply sets a {{cancelled}} flag. This means that 
> every time a task is restarted, a new thread running the task will be 
> created. Thus a task may end up with multiple instances all running in their 
> own threads when there's only supposed to be a single thread.
> h2. Steps to Reproduce
> The problem can be replicated by using the connector available here: 
> https://github.com/smarter-travel-media/hang-connector
> Apologies for how involved the steps are.
> I've created a patch that forcibly interrupts threads after they fail to 
> gracefully shutdown here: 
> https://github.com/smarter-travel-media/kafka/commit/295c747a9fd82ee8b30556c89c31e0bfcce5a2c5
> I've confirmed that this fixes the issue. I can add some unit tests and 
> submit a PR if people agree that this is a bug and interrupting threads is 
> the right fix.
> Thanks!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5910) Kafka 0.11.0.1 Kafka consumer/producers retries in infinite loop when wrong SASL creds are passed

[Ramkumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168245#comment-16168245
 ] 

Ramkumar commented on KAFKA-5910:
-

Thanks Edoardo  , is Kafka 1.0  released? or any time lines?  I could nt see 
that version in  http://kafka.apache.org/downloads - the latest version is 
0.11.01 . kindly let me know. 

> Kafka 0.11.0.1 Kafka consumer/producers retries in infinite loop when wrong 
> SASL creds are passed
> -
>
> Key: KAFKA-5910
> URL: https://issues.apache.org/jira/browse/KAFKA-5910
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.11.0.0
>Reporter: Ramkumar
>
> Similar to https://issues.apache.org/jira/browse/KAFKA-4764 , but the status 
> shows patch available but the client wont disconnects after getting the 
> warning.
> Issue 1:
> Publisher flow:
> Kafka publisher goes into infinite loop if the AAF credentials are wrong when 
> authenticating in Kaka broker.
> Detail:
> If the correct user name and password are used at the kafka publisher client 
> side to connect to kafka broker, then it authenticates and authorizes fine.
> If  incorrect username or password is used at the kafka publisher client 
> side, then broker logs shows a continuous (infinite loop)  log showing client 
> is trying to reconnect the broker as it doesn’t get authentication failure 
> exception from broker. 
> JIRA defect in apache:
> https://issues.apache.org/jira/browse/KAFKA-4764
> Can you pls let me know if this issue is resolved in kafka 0.11.0.1 version 
> or still an open issue?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5407) Mirrormaker dont start after upgrade

[Fernando Vega (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fernando Vega updated KAFKA-5407:
-
Attachment: broker.hkg1.new
debug.hkg1.new

> Mirrormaker dont start after upgrade
> 
>
> Key: KAFKA-5407
> URL: https://issues.apache.org/jira/browse/KAFKA-5407
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.10.2.1
> Environment: Operating system
> CentOS 6.8
> HW
> Board Mfg : HP
>  Board Product : ProLiant DL380p Gen8
> CPU's x2
> Product Manufacturer  : Intel
>  Product Name  :  Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz
>  Memory Type   : DDR3 SDRAM
>  SDRAM Capacity: 2048 MB
>  Total Memory: : 64GB
> Hardrives size and layout:
> 9 drives using jbod
> drive size 3.6TB each
>Reporter: Fernando Vega
>Priority: Critical
> Attachments: broker.hkg1.new, debug.hkg1.new, 
> mirrormaker-repl-sjc2-to-hkg1.log.8
>
>
> Currently Im upgrading the cluster from 0.8.2-beta to 0.10.2.1
> So I followed the rolling procedure:
> Here the config files:
> Consumer
> {noformat}
> #
> # Cluster: repl
> # Topic list(goes into command line): 
> REPL-ams1-global,REPL-atl1-global,REPL-sjc2-global,REPL-ams1-global-PN_HXIDMAP_.*,REPL-atl1-global-PN_HXIDMAP_.*,REPL-sjc2-global-PN_HXIDMAP_.*,REPL-ams1-global-PN_HXCONTEXTUALV2_.*,REPL-atl1-global-PN_HXCONTEXTUALV2_.*,REPL-sjc2-global-PN_HXCONTEXTUALV2_.*
> bootstrap.servers=app001:9092,app002:9092,app003:9092,app004:9092
> group.id=hkg1_cluster
> auto.commit.interval.ms=6
> partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
> {noformat}
> Producer
> {noformat}
>  hkg1
> # # Producer
> # # hkg1
> bootstrap.servers=app001:9092,app002:9092,app003:9092,app004:9092
> compression.type=gzip
> acks=0
> {noformat}
> Broker
> {noformat}
> auto.leader.rebalance.enable=true
> delete.topic.enable=true
> socket.receive.buffer.bytes=1048576
> socket.send.buffer.bytes=1048576
> default.replication.factor=2
> auto.create.topics.enable=true
> num.partitions=1
> num.network.threads=8
> num.io.threads=40
> log.retention.hours=1
> log.roll.hours=1
> num.replica.fetchers=8
> zookeeper.connection.timeout.ms=3
> zookeeper.session.timeout.ms=3
> inter.broker.protocol.version=0.10.2
> log.message.format.version=0.8.2
> {noformat}
> I tried also using stock configuraiton with no luck.
> The error that I get is this:
> {noformat}
> 2017-06-07 12:24:45,476] INFO ConsumerConfig values:
>   auto.commit.interval.ms = 6
>   auto.offset.reset = latest
>   bootstrap.servers = [app454.sjc2.mytest.com:9092, 
> app455.sjc2.mytest.com:9092, app456.sjc2.mytest.com:9092, 
> app457.sjc2.mytest.com:9092, app458.sjc2.mytest.com:9092, 
> app459.sjc2.mytest.com:9092]
>   check.crcs = true
>   client.id = MirrorMaker_hkg1-1
>   connections.max.idle.ms = 54
>   enable.auto.commit = false
>   exclude.internal.topics = true
>   fetch.max.bytes = 52428800
>   fetch.max.wait.ms = 500
>   fetch.min.bytes = 1
>   group.id = MirrorMaker_hkg1
>   heartbeat.interval.ms = 3000
>   interceptor.classes = null
>   key.deserializer = class 
> org.apache.kafka.common.serialization.ByteArrayDeserializer
>   max.partition.fetch.bytes = 1048576
>   max.poll.interval.ms = 30
>   max.poll.records = 500
>   metadata.max.age.ms = 30
>   metric.reporters = []
>   metrics.num.samples = 2
>   metrics.recording.level = INFO
>   metrics.sample.window.ms = 3
>   partition.assignment.strategy = 
> [org.apache.kafka.clients.consumer.RoundRobinAssignor]
>   receive.buffer.bytes = 65536
>   reconnect.backoff.ms = 50
>   request.timeout.ms = 305000
>   retry.backoff.ms = 100
>   sasl.jaas.config = null
>   sasl.kerberos.kinit.cmd = /usr/bin/kinit
>   sasl.kerberos.min.time.before.relogin = 6
>   sasl.kerberos.service.name = null
>   sasl.kerberos.ticket.renew.jitter = 0.05
>   sasl.kerberos.ticket.renew.window.factor = 0.8
>   sasl.mechanism = GSSAPI
>   security.protocol = PLAINTEXT
>   send.buffer.bytes = 131072
>   session.timeout.ms = 1
>   ssl.cipher.suites = null
>   ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>   ssl.endpoint.identification.algorithm = null
>   ssl.key.password = null
>   ssl.keymanager.algorithm = SunX509
>   ssl.keystore.location = null
>   ssl.keystore.password = null
>   ssl.keystore.type = JKS
>   ssl.protocol = TLS
>   ssl.provider = null
>   ssl.secure.random.implementation = null
>   ssl.trustmanager.algorithm = PKIX
>   ssl.truststore.location = null
>   ssl.truststore.password = null
>   ssl.truststore.type 

[jira] [Commented] (KAFKA-5163) Support replicas movement between log directories (KIP-113)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168176#comment-16168176
 ] 

ASF GitHub Bot commented on KAFKA-5163:
---

GitHub user lindong28 opened a pull request:

https://github.com/apache/kafka/pull/3874

KAFKA-5163; Support replicas movement between log directories (KIP-113)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lindong28/kafka KAFKA-5163

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3874.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3874


commit d85e65be124aaf30eb39d90906197639be0da128
Author: Dong Lin 
Date:   2017-09-14T01:30:33Z

KAFKA-5163; Support replicas movement between log directories (KIP-113)




> Support replicas movement between log directories (KIP-113)
> ---
>
> Key: KAFKA-5163
> URL: https://issues.apache.org/jira/browse/KAFKA-5163
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>
> See 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-113%3A+Support+replicas+movement+between+log+directories
>  for motivation and design.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5765) Move merge() from StreamsBuilder to KStream

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168170#comment-16168170
 ] 

Matthias J. Sax commented on KAFKA-5765:


Sure. If you need any help with the KIP, let us know. Have a look here to get 
started 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals 
(If you don't have wiki access, let us know your wiki ID so we can give you 
write permission there, \cc [~guozhang] [~damianguy] -- can you also add 
[~Yohan123] to the contributor list so we can assign this JIRA to him -- this 
will also allow you to assign JIRA to yourself). Thanks a lot!

> Move merge() from StreamsBuilder to KStream
> ---
>
> Key: KAFKA-5765
> URL: https://issues.apache.org/jira/browse/KAFKA-5765
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0, 1.0.0
>Reporter: Matthias J. Sax
>  Labels: needs-kip, newbie
> Fix For: 1.1.0
>
>
> Merging multiple {{KStream}} is done via {{StreamsBuilder#merge()}} (formally 
> {{KStreamBuilder#merge()}}). This is quite unnatural and should be done via 
> {{KStream#merge()}}.
> We need a KIP as we add a new method to a public {{KStreams}} API and 
> deprecate the old {{merge()}} method.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5854) Handle SASL authentication failures as non-retriable exceptions in clients

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5854:
---
Issue Type: New Feature  (was: Sub-task)
Parent: (was: KAFKA-4764)

> Handle SASL authentication failures as non-retriable exceptions in clients
> --
>
> Key: KAFKA-5854
> URL: https://issues.apache.org/jira/browse/KAFKA-5854
> Project: Kafka
>  Issue Type: New Feature
>  Components: clients
>Reporter: Rajini Sivaram
>Assignee: Vahid Hashemian
> Fix For: 1.0.0
>
>
> Produce and consumer changes to avoid retries on authentication failures.
> Details are in 
> [KIP-152|https://cwiki.apache.org/confluence/display/KAFKA/KIP-152+-+Improve+diagnostics+for+SASL+authentication+failures]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5910) Kafka 0.11.0.1 Kafka consumer/producers retries in infinite loop when wrong SASL creds are passed

[Edoardo Comar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168126#comment-16168126
 ] 

Edoardo Comar commented on KAFKA-5910:
--

it's resolved in 1.0 not in 0.11

> Kafka 0.11.0.1 Kafka consumer/producers retries in infinite loop when wrong 
> SASL creds are passed
> -
>
> Key: KAFKA-5910
> URL: https://issues.apache.org/jira/browse/KAFKA-5910
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.11.0.0
>Reporter: Ramkumar
>
> Similar to https://issues.apache.org/jira/browse/KAFKA-4764 , but the status 
> shows patch available but the client wont disconnects after getting the 
> warning.
> Issue 1:
> Publisher flow:
> Kafka publisher goes into infinite loop if the AAF credentials are wrong when 
> authenticating in Kaka broker.
> Detail:
> If the correct user name and password are used at the kafka publisher client 
> side to connect to kafka broker, then it authenticates and authorizes fine.
> If  incorrect username or password is used at the kafka publisher client 
> side, then broker logs shows a continuous (infinite loop)  log showing client 
> is trying to reconnect the broker as it doesn’t get authentication failure 
> exception from broker. 
> JIRA defect in apache:
> https://issues.apache.org/jira/browse/KAFKA-4764
> Can you pls let me know if this issue is resolved in kafka 0.11.0.1 version 
> or still an open issue?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-4764) Improve diagnostics for SASL authentication failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168105#comment-16168105
 ] 

ASF GitHub Bot commented on KAFKA-4764:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3708


> Improve diagnostics for SASL authentication failures
> 
>
> Key: KAFKA-4764
> URL: https://issues.apache.org/jira/browse/KAFKA-4764
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.10.2.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 1.0.0
>
>
> At the moment, broker closes the client connection if SASL authentication 
> fails. Clients see this as a connection failure and do not get any feedback 
> for the reason why the connection was closed. Producers and consumers retry, 
> attempting to create successful connections, treating authentication failures 
> as transient failures. There are no log entries on the client-side which 
> indicate that any of these connection failures were due to authentication 
> failure.
> This JIRA will aim to improve diagnosis of authentication failures with the 
> changes described in 
> [KIP-152|https://cwiki.apache.org/confluence/display/KAFKA/KIP-152+-+Improve+diagnostics+for+SASL+authentication+failures].
> This JIRA also does not change handling of SSL authentication failures. 
> javax.net.debug provides sufficient diagnostics for this case. SSL changes 
> are harder to do while preserving backward compatibility.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-4190) kafka-reassign-partitions does not report syntax problem in json

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168086#comment-16168086
 ] 

ASF GitHub Bot commented on KAFKA-4190:
---

Github user chemikadze closed the pull request at:

https://github.com/apache/kafka/pull/1875


> kafka-reassign-partitions does not report syntax problem in json
> 
>
> Key: KAFKA-4190
> URL: https://issues.apache.org/jira/browse/KAFKA-4190
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Nikolay Sokolov
>Priority: Trivial
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> When specifying invalid json file, kafka-reassign-partitions fails with error 
> "file is empty" instead of reporting syntax error.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5857) Excessive heap usage on controller node during reassignment

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168071#comment-16168071
 ] 

Ismael Juma commented on KAFKA-5857:


A heap dump would be really helpful.

> Excessive heap usage on controller node during reassignment
> ---
>
> Key: KAFKA-5857
> URL: https://issues.apache.org/jira/browse/KAFKA-5857
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 0.11.0.0
> Environment: CentOs 7, Java 1.8
>Reporter: Raoufeh Hashemian
>  Labels: reliability
> Fix For: 1.1.0
>
> Attachments: CPU.png, disk_write_x.png, memory.png, 
> reassignment_plan.txt
>
>
> I was trying to expand our kafka cluster of 6 broker nodes to 12 broker 
> nodes. 
> Before expansion, we had a single topic with 960 partitions and a replication 
> factor of 3. So each node had 480 partitions. The size of data in each node 
> was 3TB . 
> To do the expansion, I submitted a partition reassignment plan (see attached 
> file for the current/new assignments). The plan was optimized to minimize 
> data movement and be rack aware. 
> When I submitted the plan, it took approximately 3 hours for moving data from 
> old to new nodes to complete. After that, it started deleting source 
> partitions (I say this based on the number of file descriptors) and 
> rebalancing leaders which has not been successful. Meanwhile, the heap usage 
> in the controller node started to go up with a large slope (along with long 
> GC times) and it took 5 hours for the controller to go out of memory and 
> another controller started to have the same behaviour for another 4 hours. At 
> this time the zookeeper ran out of disk and the service stopped.
> To recover from this condition:
> 1) Removed zk logs to free up disk and restarted all 3 zk nodes
> 2) Deleted /kafka/admin/reassign_partitions node from zk
> 3) Had to do unclean restarts of kafka service on oom controller nodes which 
> took 3 hours to complete  . After this stage there was still 676 under 
> replicated partitions.
> 4) Do a clean restart on all 12 broker nodes.
> After step 4 , number of under replicated nodes went to 0.
> So I was wondering if this memory footprint from controller is expected for 
> 1k partitions ? Did we do sth wrong or it is a bug?
> Attached are some resource usage graph during this 30 hours event and the 
> reassignment plan. I'll try to add log files as well



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5857) Excessive heap usage on controller node during reassignment

[Raoufeh Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168063#comment-16168063
 ] 

Raoufeh Hashemian commented on KAFKA-5857:
--

I was able to reproduce the problem when shrinking a topic of the same size 
from 12 to 6 brokers.

This time , before the controller runs out of memory, stopped the kafka service 
in the controller node and this caused the other nodes to get busy again and 
delete their already moved partition. The partition reassignment finished 
successfully . However, the controller node was a source node (in the list of 
nodes that were expected to have no partition replicas after reassignment) . I 
guess the result could be different if the controller node was one of the 
destination nodes.

It looks more like a bug in the controller rather than an inefficient heap 
usage.


> Excessive heap usage on controller node during reassignment
> ---
>
> Key: KAFKA-5857
> URL: https://issues.apache.org/jira/browse/KAFKA-5857
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 0.11.0.0
> Environment: CentOs 7, Java 1.8
>Reporter: Raoufeh Hashemian
>  Labels: reliability
> Fix For: 1.1.0
>
> Attachments: CPU.png, disk_write_x.png, memory.png, 
> reassignment_plan.txt
>
>
> I was trying to expand our kafka cluster of 6 broker nodes to 12 broker 
> nodes. 
> Before expansion, we had a single topic with 960 partitions and a replication 
> factor of 3. So each node had 480 partitions. The size of data in each node 
> was 3TB . 
> To do the expansion, I submitted a partition reassignment plan (see attached 
> file for the current/new assignments). The plan was optimized to minimize 
> data movement and be rack aware. 
> When I submitted the plan, it took approximately 3 hours for moving data from 
> old to new nodes to complete. After that, it started deleting source 
> partitions (I say this based on the number of file descriptors) and 
> rebalancing leaders which has not been successful. Meanwhile, the heap usage 
> in the controller node started to go up with a large slope (along with long 
> GC times) and it took 5 hours for the controller to go out of memory and 
> another controller started to have the same behaviour for another 4 hours. At 
> this time the zookeeper ran out of disk and the service stopped.
> To recover from this condition:
> 1) Removed zk logs to free up disk and restarted all 3 zk nodes
> 2) Deleted /kafka/admin/reassign_partitions node from zk
> 3) Had to do unclean restarts of kafka service on oom controller nodes which 
> took 3 hours to complete  . After this stage there was still 676 under 
> replicated partitions.
> 4) Do a clean restart on all 12 broker nodes.
> After step 4 , number of under replicated nodes went to 0.
> So I was wondering if this memory footprint from controller is expected for 
> 1k partitions ? Did we do sth wrong or it is a bug?
> Attached are some resource usage graph during this 30 hours event and the 
> reassignment plan. I'll try to add log files as well



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5910) Kafka 0.11.0.1 Kafka consumer/producers retries in infinite loop when wrong SASL creds are passed

[Ramkumar (JIRA)]

Ramkumar created KAFKA-5910:
---

 Summary: Kafka 0.11.0.1 Kafka consumer/producers retries in 
infinite loop when wrong SASL creds are passed
 Key: KAFKA-5910
 URL: https://issues.apache.org/jira/browse/KAFKA-5910
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.11.0.0
Reporter: Ramkumar


Similar to https://issues.apache.org/jira/browse/KAFKA-4764 , but the status 
shows patch available but the client wont disconnects after getting the warning.


Issue 1:
Publisher flow:
Kafka publisher goes into infinite loop if the AAF credentials are wrong when 
authenticating in Kaka broker.
Detail:
If the correct user name and password are used at the kafka publisher client 
side to connect to kafka broker, then it authenticates and authorizes fine.
If  incorrect username or password is used at the kafka publisher client side, 
then broker logs shows a continuous (infinite loop)  log showing client is 
trying to reconnect the broker as it doesn’t get authentication failure 
exception from broker. 
JIRA defect in apache:
https://issues.apache.org/jira/browse/KAFKA-4764

Can you pls let me know if this issue is resolved in kafka 0.11.0.1 version or 
still an open issue?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5716) Connect: When SourceTask.commit it is possible not everthing from SourceTask.poll has been sent

[Randall Hauch (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167956#comment-16167956
 ] 

Randall Hauch commented on KAFKA-5716:
--

[~ewencp], if you wouldn't mind, would you take a look at this? In the issue 
description above, [~steff1193] suggests two possible fixes:

1) Changing the JavaDoc to say what the code *actually* does. This _may not_ 
require a KIP if the consensus is that this *clarifies* existing behavior 
rather than changing the behavior.
2) Correcting the code to implement the specified behavior. This definitely 
requires a KIP as it would change the public API to add a method that does the 
correct thing (this could be done in a backward compatible manner).

I'm unable to think of another fix, but perhaps something is more obvious to 
you. Thoughts?

> Connect: When SourceTask.commit it is possible not everthing from 
> SourceTask.poll has been sent
> ---
>
> Key: KAFKA-5716
> URL: https://issues.apache.org/jira/browse/KAFKA-5716
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Per Steffensen
>Priority: Minor
> Attachments: KAFKA-5716.patch
>
>
> Not looking at the very latest code, so the "problem" may have been corrected 
> recently. If so, I apologize. I found the "problem" by code-inspection alone, 
> so I may be wrong. Have not had the time to write tests to confirm.
> According to java-doc on SourceTask.commit
> {quote}
> Commit the offsets, up to the offsets that have been returned by \{@link 
> #poll()}. This
> method should block until the commit is complete.
> SourceTasks are not required to implement this functionality; Kafka Connect 
> will record offsets
> automatically. This hook is provided for systems that also need to store 
> offsets internally
> in their own system.
> {quote}
> As I read this, when commit-method is called, the SourceTask-developer is 
> "told" that everything returned from poll up until "now" has been sent/stored 
> - both the outgoing messages and the associated connect-offsets. Looking at 
> the implementation it also seems that this is what it tries to 
> "guarantee/achieve".
> But as I see read the code, it is not necessarily true
> The following threads are involved
> * Task-thread: WorkerSourceTask has its own thread running 
> WorkerSourceTask.execute.
> * Committer-thread: From time to time SourceTaskOffsetCommitter is scheduled 
> to call WorkerSourceTask.commitOffsets (from a different thread)
> The two thread synchronize (on the WorkerSourceTask-object) in sendRecord and 
> commitOffsets respectively, hindering the task-thread to add to 
> outstandingMessages and offsetWriter while committer-thread is marking what 
> has to be flushed in the offsetWriter and waiting for outstandingMessages to 
> be empty. This means that the offsets committed will be consistent with what 
> has been sent out, but not necessarily what has been polled. At least I do 
> not see why the following is not possible:
> * Task-thread polls something from the task.poll
> * Before task-thread gets to add (all) the polled records to 
> outstandingMessages and offsetWriter in sendRecords, committer-thread kicks 
> in and does its commiting, while hindering the task-thread adding the polled 
> records to outstandingMessages and offsetWriter
> * Consistency will not have been compromised, but committer-thread will end 
> up calling task.commit (via WorkerSourceTask.commitSourceTask), without the 
> records just polled from task.poll has been sent or corresponding 
> connector-offsets flushed.
> If I am right, I guess there are two way to fix it
> * Either change the java-doc of SourceTask.commit, to something a-la (which I 
> do believe is true)
> {quote}
> Commit the offsets, up to the offsets that have been returned by \{@link 
> #poll()}
> *and confirmed by a call to \{@link #commitRecord(SourceRecord)}*.
> This method should block until the commit is complete.
> SourceTasks are not required to implement this functionality; Kafka Connect 
> will record offsets
> automatically. This hook is provided for systems that also need to store 
> offsets internally
> in their own system.
> {quote}
> * or, fix the "problem" so that it actually does what the java-doc says :-)
> If I am not right, of course I apologize for the inconvenience. I would 
> appreciate an explanation where my code-inspection is not correct, and why it 
> works even though I cannot see it. I will not expect such an explanation, 
> though.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5909) Remove source jars from classpath while executing CLI tools

[Kamal Chandraprakash (JIRA)]

Kamal Chandraprakash created KAFKA-5909:
---

 Summary: Remove source jars from classpath while executing CLI 
tools
 Key: KAFKA-5909
 URL: https://issues.apache.org/jira/browse/KAFKA-5909
 Project: Kafka
  Issue Type: Bug
  Components: tools
Affects Versions: 0.11.0.0
Reporter: Kamal Chandraprakash
Assignee: Kamal Chandraprakash
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-3722) PlaintextChannelBuilder should not use ChannelBuilders.createPrincipalBuilder(configs) for creating instance of PrincipalBuilder

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167717#comment-16167717
 ] 

Manikumar commented on KAFKA-3722:
--

[~mgharat] I think KIP-189/KAFKA-5783 addresses this issue. If you agree,  we 
can close this issue.

> PlaintextChannelBuilder should not use 
> ChannelBuilders.createPrincipalBuilder(configs) for creating instance of 
> PrincipalBuilder
> 
>
> Key: KAFKA-3722
> URL: https://issues.apache.org/jira/browse/KAFKA-3722
> Project: Kafka
>  Issue Type: Bug
>Reporter: Mayuresh Gharat
>Assignee: Mayuresh Gharat
>
> Consider this scenario :
> 1) We have a Kafka Broker running on  PlainText and SSL port simultaneously.
> 2)  We try to plugin a custom principal builder using the config 
> "principal.builder.class" for the request coming over the SSL port.
> 3) The ChannelBuilders.createPrincipalBuilder(configs) first checks if a 
> config "principal.builder.class" is specified in the passed in configs and 
> tries to use that even when it is building the instance of PrincipalBuilder 
> for the PlainText port, when that custom principal class is only menat for 
> SSL port.
> IMO, having a DefaultPrincipalBuilder for PalinText port should be fine.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5856) Add AdminClient.createPartitions()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167712#comment-16167712
 ] 

ASF GitHub Bot commented on KAFKA-5856:
---

GitHub user tombentley opened a pull request:

https://github.com/apache/kafka/pull/3870

KAFKA-5856: Add AdminClient.createPartitions()

See KIP-195.

The contribution is my original work and I license the work to the project 
under the project's open source license.

This patch adds AdminClient.createPartitions() and the network protocol is
uses. The broker-side algorithm is as follows:

1. KafkaApis makes some initial checks on the request, then delegates to the
   new AdminManager.createPartitions() method.
2. AdminManager.createPartitions() performs some validation then delegates 
to
   AdminUtils.addPartitions().

Aside: I felt it was safer to add the extra validation in
AdminManager.createPartitions() than in AdminUtils.addPartitions() since the
latter is used on other code paths which might fail differently with the
introduction of extra checks.

3. AdminUtils.addPartitions() does its own checks and adds the partitions.
4. AdminManager then uses the existing topic purgatory to wait for the
   PartitionInfo available from the metadata cache to become consistent with
   the new total number of partitions.

The messages of exceptions thrown in AdminUtils affecting this new API have
been made consistent with initial capital letter and terminating period.
A few have been reworded for clarity. I've also standardized on using
String.format().

cc @ijuma

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tombentley/kafka 
KAFKA-5856-AdminClient.createPartitions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3870.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3870


commit ab07f15a794c385cbfdecd33a5a44c7725e8d103
Author: Tom Bentley 
Date:   2017-09-15T09:50:59Z

KAFKA-5856: Add AdminClient.createPartitions()

See KIP-195.

This patch adds AdminClient.createPartitions() and the network protocol is
uses. The broker-side algorithm is as follows:

1. KafkaApis makes some initial checks on the request, then delegates to the
   new AdminManager.createPartitions() method.
2. AdminManager.createPartitions() performs some validation then delegates 
to
   AdminUtils.addPartitions().

Aside: I felt it was safer to add the extra validation in
AdminManager.createPartitions() than in AdminUtils.addPartitions() since the
latter is used on other code paths which might fail differently with the
introduction of extra checks.

3. AdminUtils.addPartitions() does its own checks and adds the partitions.
4. AdminManager then uses the existing topic purgatory to wait for the
   PartitionInfo available from the metadata cache to become consistent with
   the new total number of partitions.

The messages of exceptions thrown in AdminUtils affecting this new API have
been made consistent with initial capital letter and terminating period.
A few have been reworded for clarity. I've also standardized on using
String.format().




> Add AdminClient.createPartitions()
> --
>
> Key: KAFKA-5856
> URL: https://issues.apache.org/jira/browse/KAFKA-5856
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Tom Bentley
>Assignee: Tom Bentley
>  Labels: kip
>
> It should be possible to increase the partition count using the AdminClient. 
> See 
> [KIP-195|https://cwiki.apache.org/confluence/display/KAFKA/KIP-195%3A+AdminClient.increasePartitions]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-4454) Authorizer should also include the Principal generated by the PrincipalBuilder.

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4454.
--
Resolution: Fixed

This is covered in KIP-189/KAFKA-5783

> Authorizer should also include the Principal generated by the 
> PrincipalBuilder.
> ---
>
> Key: KAFKA-4454
> URL: https://issues.apache.org/jira/browse/KAFKA-4454
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Mayuresh Gharat
>Assignee: Mayuresh Gharat
>
> Currently kafka allows users to plugin a custom PrincipalBuilder and a custom 
> Authorizer.
> The Authorizer.authorize() object takes in a Session object that wraps 
> KafkaPrincipal and InetAddress.
> The KafkaPrincipal currently has a PrincipalType and Principal name, which is 
> the name of Principal generated by the PrincipalBuilder. 
> This Principal, generated by the pluggedin PrincipalBuilder might have other 
> fields that might be required by the pluggedin Authorizer but currently we 
> loose this information since we only extract the name of Principal while 
> creating KaflkaPrincipal in SocketServer.  
> It would be great if KafkaPrincipal has an additional field 
> "channelPrincipal" which is used to store the Principal generated by the 
> plugged in PrincipalBuilder.
> The pluggedin Authorizer can then use this "channelPrincipal" to do 
> authorization.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5856) Add AdminClient.createPartitions()

[Tom Bentley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Bentley updated KAFKA-5856:
---
Summary: Add AdminClient.createPartitions()  (was: AdminClient should be 
able to increase number of partitions)

> Add AdminClient.createPartitions()
> --
>
> Key: KAFKA-5856
> URL: https://issues.apache.org/jira/browse/KAFKA-5856
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Tom Bentley
>Assignee: Tom Bentley
>  Labels: kip
>
> It should be possible to increase the partition count using the AdminClient. 
> See 
> [KIP-195|https://cwiki.apache.org/confluence/display/KAFKA/KIP-195%3A+AdminClient.increasePartitions]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5908) CompositeReadOnlyWindowStore range fetch doesn't return all values when fetching with different start and end times

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167691#comment-16167691
 ] 

ASF GitHub Bot commented on KAFKA-5908:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3868


> CompositeReadOnlyWindowStore range fetch doesn't return all values when 
> fetching with different start and end times
> ---
>
> Key: KAFKA-5908
> URL: https://issues.apache.org/jira/browse/KAFKA-5908
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 1.0.0
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 1.0.0
>
>
> The {{NextIteratorFunction}} in {{CompositeReadOnlyWindowStore}} is 
> incorrectly using the {{timeFrom}} as the {{timeTo}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5908) CompositeReadOnlyWindowStore range fetch doesn't return all values when fetching with different start and end times

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy resolved KAFKA-5908.
---
Resolution: Fixed

Issue resolved by pull request 3868
[https://github.com/apache/kafka/pull/3868]

> CompositeReadOnlyWindowStore range fetch doesn't return all values when 
> fetching with different start and end times
> ---
>
> Key: KAFKA-5908
> URL: https://issues.apache.org/jira/browse/KAFKA-5908
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 1.0.0
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 1.0.0
>
>
> The {{NextIteratorFunction}} in {{CompositeReadOnlyWindowStore}} is 
> incorrectly using the {{timeFrom}} as the {{timeTo}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5908) CompositeReadOnlyWindowStore range fetch doesn't return all values when fetching with different start and end times

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167574#comment-16167574
 ] 

ASF GitHub Bot commented on KAFKA-5908:
---

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/3868

KAFKA-5908: fix range query in CompositeReadOnlyWindowStore

The `NextIteratorFunction` in `CompositeReadOnlyWindowStore` was 
incorrectly using the `timeFrom` as the `timeTo`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka window-store-range-scan

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3868.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3868


commit 925f7aa400d876187032cfda191c7f120d9a141f
Author: Damian Guy 
Date:   2017-09-15T09:02:33Z

fix range query




> CompositeReadOnlyWindowStore range fetch doesn't return all values when 
> fetching with different start and end times
> ---
>
> Key: KAFKA-5908
> URL: https://issues.apache.org/jira/browse/KAFKA-5908
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 1.0.0
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 1.0.0
>
>
> The {{NextIteratorFunction}} in {{CompositeReadOnlyWindowStore}} is 
> incorrectly using the {{timeFrom}} as the {{timeTo}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5770) AdminClient.deleteTopics future complete but topic is still here

[Vincent Maurin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167530#comment-16167530
 ] 

Vincent Maurin commented on KAFKA-5770:
---

The behavior haven't changed with version 0.11.0.1

> AdminClient.deleteTopics future complete but topic is still here
> 
>
> Key: KAFKA-5770
> URL: https://issues.apache.org/jira/browse/KAFKA-5770
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Vincent Maurin
> Attachments: Main.java
>
>
> After running some tests, it appears that a deleteTopics command futures are 
> completed even if the topic is still present on the broker.
> If it is the expected behavior, it should be documented accordingly, but it 
> is not very convenient for integration tests for example, when we create and 
> delete topics on each tests
> I am attaching a example java file that creates and deletes a bunch of topic 
> in a loop. Usually I got an error on the second loop saying that the topic 
> already exists



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5891) Cast transformation fails if record schema contains timestamp field

[Artem Plotnikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167493#comment-16167493
 ] 

Artem Plotnikov commented on KAFKA-5891:


Seems like Kafka Connect's Cast transformation loses schema information 
(basically, schema name) while doing type casting. I was able to reproduce this 
problem with the following test in org.apache.kafka.connect.transforms.CastTest 
for current trunk repository branch:
{code}
@SuppressWarnings("unchecked")
@Test
public void castWholeRecordValueWithSchemaBooleanAndTimestampField() {
final Cast xform = new Cast.Value<>();
xform.configure(Collections.singletonMap(Cast.SPEC_CONFIG, 
"int64:boolean"));

SchemaBuilder builder = SchemaBuilder.struct();
builder.field("int64", Schema.INT64_SCHEMA);
builder.field("timestamp", Timestamp.SCHEMA);
Schema supportedTypesSchema = builder.build();

Struct recordValue = new Struct(supportedTypesSchema);
recordValue.put("int64", (long) 64);
recordValue.put("timestamp", new java.sql.Timestamp(0L));

SourceRecord transformed = xform.apply(new SourceRecord(null, null, 
"topic", 0,
supportedTypesSchema, recordValue));

assertEquals(true, ((Struct) transformed.value()).get("int64"));
assertEquals(new java.sql.Timestamp(0L), ((Struct) 
transformed.value()).get("timestamp"));
}
{code}
The problem is that Timestamp.SCHEMA has schema.type = 'INT64' and schema.name 
= "org.apache.kafka.connect.data.Timestamp", but 
org.apache.kafka.connect.transforms.Cast#getOrBuildSchema method copies 
schema.type only:
{code}
SchemaBuilder fieldBuilder =
convertFieldType(casts.containsKey(field.name()) ? casts.get(field.name()) 
: field.schema().type());
{code}


> Cast transformation fails if record schema contains timestamp field
> ---
>
> Key: KAFKA-5891
> URL: https://issues.apache.org/jira/browse/KAFKA-5891
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Artem Plotnikov
>
> I have the following simple type cast transformation:
> {code}
> name=postgresql-source-simple
> connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
> tasks.max=1
> connection.url=jdbc:postgresql://localhost:5432/testdb?user=postgres=mysecretpassword
> query=SELECT 1::INT as a, '2017-09-14 10:23:54'::TIMESTAMP as b
> transforms=Cast
> transforms.Cast.type=org.apache.kafka.connect.transforms.Cast$Value
> transforms.Cast.spec=a:boolean
> mode=bulk
> topic.prefix=clients
> {code}
> Which fails with the following exception in runtime:
> {code}
> [2017-09-14 16:51:01,885] ERROR Task postgresql-source-simple-0 threw an 
> uncaught and unrecoverable exception 
> (org.apache.kafka.connect.runtime.WorkerTask:148)
> org.apache.kafka.connect.errors.DataException: Invalid Java object for schema 
> type INT64: class java.sql.Timestamp for field: "null"
>   at 
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:239)
>   at 
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:209)
>   at org.apache.kafka.connect.data.Struct.put(Struct.java:214)
>   at 
> org.apache.kafka.connect.transforms.Cast.applyWithSchema(Cast.java:152)
>   at org.apache.kafka.connect.transforms.Cast.apply(Cast.java:108)
>   at 
> org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:38)
>   at 
> org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:190)
>   at 
> org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:168)
>   at 
> org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
>   at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> If I remove the  transforms.* part of the connector it will work correctly. 
> Actually, it doesn't really matter which types I use in the transformation 
> for field 'a', just the existence of a timestamp field brings the exception.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5907) Support aggregatedJavadoc in Java 9

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167451#comment-16167451
 ] 

Ismael Juma commented on KAFKA-5907:


Hmm, not sure if the missing jar is the issue.

> Support aggregatedJavadoc in Java 9
> ---
>
> Key: KAFKA-5907
> URL: https://issues.apache.org/jira/browse/KAFKA-5907
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
> Fix For: 1.1.0
>
>
> The Java 9 Javadoc tool has some improvements including a search bar. 
> However, it currently fails with a number of errors like:
> {code}
> > Task :aggregatedJavadoc
> /Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:29:
>  error: package org.apache.kafka.streams.processor.internals does not exist
> import org.apache.kafka.streams.processor.internals.InternalTopologyBuilder;
>^
> /Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:30:
>  error: package org.apache.kafka.streams.processor.internals does not exist
> import org.apache.kafka.streams.processor.internals.ProcessorNode;
>^
> /Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:31:
>  error: package org.apache.kafka.streams.processor.internals does not exist
> import org.apache.kafka.streams.processor.internals.ProcessorTopology;
>^
> /Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:32:
>  error: package org.apache.kafka.streams.processor.internals does not exist
> import org.apache.kafka.streams.processor.internals.SinkNode;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5907) Support aggregatedJavadoc in Java 9

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167443#comment-16167443
 ] 

Ismael Juma commented on KAFKA-5907:


Looking at the javadoc command, it seems to be missing the streams jar:

{code}
-classpath 
'/Users/ijuma/.gradle/caches/modules-2/files-2.1/org.lz4/lz4-java/1.4/9bedb74f461a87ff2161bdf0778ad8ca6bad3e1c/lz4-java-1.4.jar:/Users/ijuma/.gradle/caches/modules-2/files-2.1/org.xerial.snappy/snappy-java/1.1.4/d94ae6d7d27242eaa4b6c323f881edbb98e48da6/snappy-java-1.1.4.jar:/Users/ijuma/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/Users/ijuma/src/kafka/clients/build/libs/kafka-clients-1.0.0-SNAPSHOT.jar:/Users/ijuma/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.9.1/716da1830a2043f18882fc036ec26eb32cbe5aff/jackson-databind-2.9.1.jar:/Users/ijuma/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/5.0.4/4fdac2fbe92dfad86aa6e9301736f6b4342a3f5c/jopt-simple-5.0.4.jar:/Users/ijuma/.gradle/caches/modules-2/files-2.1/com.yammer.metrics/metrics-core/2.2.0/f82c035cfa786d3cbec362c38c22a5f5b1bc8724/metrics-core-2.2.0.jar:/Users/ijuma/.gradle/caches/modules-2/files-2.1/org.scala-lang/scala-library/2.11.11/e283d2b7fde6504f6a86458b1f6af465353907cc/scala-library-2.11.11.jar:/Users/ijuma/.gradle/caches/modules-2/files-2.1/com.101tec/zkclient/0.10/c54d4b5a5e89af75a80b6d5857400165ce5188d0/zkclient-0.10.jar:/Users/ijuma/.gradle/caches/modules-2/files-2.1/org.apache.zookeeper/zookeeper/3.4.10/8eebdbb7a9df83e02eaa42d0e5da0b57bf2e4da/zookeeper-3.4.10.jar:/Users/ijuma/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-log4j12/1.7.25/110cefe2df103412849d72ef7a67e4e91e4266b4/slf4j-log4j12-1.7.25.jar:/Users/ijuma/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.9.0/7c10d545325e3a6e72e06381afe469fd40eb701/jackson-annotations-2.9.0.jar:/Users/ijuma/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.9.1/60077fe98b11e4e7cf8af9b20609326a166d6ac4/jackson-core-2.9.1.jar:/Users/ijuma/.gradle/caches/modules-2/files-2.1/log4j/log4j/1.2.17/5af35056b4d257e4b64b9e8069c0746e8b08629f/log4j-1.2.17.jar:/Users/ijuma/src/kafka/connect/json/build/libs/connect-json-1.0.0-SNAPSHOT.jar:/Users/ijuma/src/kafka/connect/api/build/libs/connect-api-1.0.0-SNAPSHOT.jar:/Users/ijuma/.gradle/caches/modules-2/files-2.1/org.rocksdb/rocksdbjni/5.3.6/7841d03e9d91148fee534e514175141e7c855a3/rocksdbjni-5.3.6.jar'
{code}

> Support aggregatedJavadoc in Java 9
> ---
>
> Key: KAFKA-5907
> URL: https://issues.apache.org/jira/browse/KAFKA-5907
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
> Fix For: 1.1.0
>
>
> The Java 9 Javadoc tool has some improvements including a search bar. 
> However, it currently fails with a number of errors like:
> {code}
> > Task :aggregatedJavadoc
> /Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:29:
>  error: package org.apache.kafka.streams.processor.internals does not exist
> import org.apache.kafka.streams.processor.internals.InternalTopologyBuilder;
>^
> /Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:30:
>  error: package org.apache.kafka.streams.processor.internals does not exist
> import org.apache.kafka.streams.processor.internals.ProcessorNode;
>^
> /Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:31:
>  error: package org.apache.kafka.streams.processor.internals does not exist
> import org.apache.kafka.streams.processor.internals.ProcessorTopology;
>^
> /Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:32:
>  error: package org.apache.kafka.streams.processor.internals does not exist
> import org.apache.kafka.streams.processor.internals.SinkNode;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5907) Support aggregatedJavadoc in Java 9

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-5907:
--

 Summary: Support aggregatedJavadoc in Java 9
 Key: KAFKA-5907
 URL: https://issues.apache.org/jira/browse/KAFKA-5907
 Project: Kafka
  Issue Type: Improvement
Reporter: Ismael Juma
 Fix For: 1.1.0


The Java 9 Javadoc tool has some improvements including a search bar. However, 
it currently fails with a number of errors like:

{code}
> Task :aggregatedJavadoc
/Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:29:
 error: package org.apache.kafka.streams.processor.internals does not exist
import org.apache.kafka.streams.processor.internals.InternalTopologyBuilder;
   ^
/Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:30:
 error: package org.apache.kafka.streams.processor.internals does not exist
import org.apache.kafka.streams.processor.internals.ProcessorNode;
   ^
/Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:31:
 error: package org.apache.kafka.streams.processor.internals does not exist
import org.apache.kafka.streams.processor.internals.ProcessorTopology;
   ^
/Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:32:
 error: package org.apache.kafka.streams.processor.internals does not exist
import org.apache.kafka.streams.processor.internals.SinkNode;
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5906) Change metric.reporters configuration key to metrics.reporters to be consistent

[Kevin Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lu updated KAFKA-5906:

Description: 
The "metric.reporters" configuration key should be consistent with the actual 
classes. Clients have a MetricsReporter.class while the broker has a 
KafkaMetricsReporter.class. 

We have seen quite a few people configure this field incorrectly by setting it 
as "metrics.reporters".

The configuration key could be renamed to "metrics.reporters" to match the 
classes, or the classes can be renamed to MetricReporter.class and 
KafkaMetricReporter.class.

The broker configuration description for "metric.reporters" also mentions 
MetricReporter, but KafkaMetricsReporter is the actual interface to implement. 

There also seems to be a discrepancy with "MetricReporter" in the description 
as the class name is actually "MetricsReporter". 
https://github.com/apache/kafka/pull/3867

  was:
The "metric.reporters" configuration key should be consistent with the actual 
classes. Clients have a MetricsReporter.class while the broker has a 
KafkaMetricsReporter.class. 

We have seen quite a few people configure this field incorrectly by setting it 
as "metrics.reporters".

The configuration key could be renamed to "metrics.reporters" to match the 
classes, or the classes can be renamed to MetricReporter.class and 
KafkaMetricReporter.class.

The broker configuration description for "metric.reporters" also mentions 
MetricReporter, but the actual interface to implement is KafkaMetricsReporter. 

There also seems to be a discrepancy with "MetricReporter" in the description 
as the class name is actually "MetricsReporter". 
https://github.com/apache/kafka/pull/3867


> Change metric.reporters configuration key to metrics.reporters to be 
> consistent
> ---
>
> Key: KAFKA-5906
> URL: https://issues.apache.org/jira/browse/KAFKA-5906
> Project: Kafka
>  Issue Type: Improvement
>  Components: config, metrics
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: usability
>
> The "metric.reporters" configuration key should be consistent with the actual 
> classes. Clients have a MetricsReporter.class while the broker has a 
> KafkaMetricsReporter.class. 
> We have seen quite a few people configure this field incorrectly by setting 
> it as "metrics.reporters".
> The configuration key could be renamed to "metrics.reporters" to match the 
> classes, or the classes can be renamed to MetricReporter.class and 
> KafkaMetricReporter.class.
> The broker configuration description for "metric.reporters" also mentions 
> MetricReporter, but KafkaMetricsReporter is the actual interface to 
> implement. 
> There also seems to be a discrepancy with "MetricReporter" in the description 
> as the class name is actually "MetricsReporter". 
> https://github.com/apache/kafka/pull/3867



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5906) Change metric.reporters configuration key to metrics.reporters to be consistent

[Kevin Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lu updated KAFKA-5906:

Description: 
The "metric.reporters" configuration key should be consistent with the actual 
classes. Clients have a MetricsReporter.class while the broker has a 
KafkaMetricsReporter.class. 

We have seen quite a few people configure this field incorrectly by setting it 
as "metrics.reporters".

The configuration key could be renamed to "metrics.reporters" to match the 
classes, or the classes can be renamed to MetricReporter.class and 
KafkaMetricReporter.class.

The broker configuration description for "metric.reporters" also mentions 
MetricReporter, but the actual interface to implement is KafkaMetricsReporter. 

There also seems to be a discrepancy with "MetricReporter" in the description 
as the class name is actually "MetricsReporter". 
https://github.com/apache/kafka/pull/3867

  was:
The "metric.reporters" configuration key should be consistent with the actual 
classes. Clients have a MetricsReporter.class while the broker has a 
KafkaMetricsReporter.class. 

We have seen quite a few people configure this field incorrectly by setting it 
as "metrics.reporters".

The configuration key could be renamed to "metrics.reporters" to match the 
classes, or the classes can be renamed to MetricReporter.class and 
KafkaMetricReporter.class.

The broker configuration description for "metric.reporters" also mentions 
MetricReporter, but the actual interface to implement is KafkaMetricsReporter. 


> Change metric.reporters configuration key to metrics.reporters to be 
> consistent
> ---
>
> Key: KAFKA-5906
> URL: https://issues.apache.org/jira/browse/KAFKA-5906
> Project: Kafka
>  Issue Type: Improvement
>  Components: config, metrics
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: usability
>
> The "metric.reporters" configuration key should be consistent with the actual 
> classes. Clients have a MetricsReporter.class while the broker has a 
> KafkaMetricsReporter.class. 
> We have seen quite a few people configure this field incorrectly by setting 
> it as "metrics.reporters".
> The configuration key could be renamed to "metrics.reporters" to match the 
> classes, or the classes can be renamed to MetricReporter.class and 
> KafkaMetricReporter.class.
> The broker configuration description for "metric.reporters" also mentions 
> MetricReporter, but the actual interface to implement is 
> KafkaMetricsReporter. 
> There also seems to be a discrepancy with "MetricReporter" in the description 
> as the class name is actually "MetricsReporter". 
> https://github.com/apache/kafka/pull/3867



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)