[jira] [Commented] (KAFKA-5670) Add Topology and deprecate TopologyBuilder

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104180#comment-16104180
 ] 

ASF GitHub Bot commented on KAFKA-5670:
---

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/3590

KAFKA-5670: (KIP-120) Add Topology and deprecate TopologyBuilder



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka 
kafka-3856-replace-topology-builder-by-topology

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3590.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3590


commit ef88eee2c97de4a7ea215e408eb6c191cbd023f9
Author: Matthias J. Sax 
Date:   2017-07-26T22:40:18Z

KAFKA-5670: (KIP-120) Add Topology and deprecate TopologyBuilder




> Add Topology and deprecate TopologyBuilder
> --
>
> Key: KAFKA-5670
> URL: https://issues.apache.org/jira/browse/KAFKA-5670
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5660) Don't throw TopologyBuilderException during runtime

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104122#comment-16104122
 ] 

Matthias J. Sax commented on KAFKA-5660:


{{TopologyBuilderException}} got deprecated via KIP-120 and new 
{{TopologyException}} was introduces. Thus, we need to double check 
{{TopologyException}}, too.

> Don't throw TopologyBuilderException during runtime
> ---
>
> Key: KAFKA-5660
> URL: https://issues.apache.org/jira/browse/KAFKA-5660
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Matthias J. Sax
>
> {{TopologyBuilderException}} is a pre-runtime exception that should only be 
> thrown {{KafkaStreams#start()}} is called.
> However, we do throw {{TopologyBuilderException}} within
> - `SourceNodeFactory#getTopics`
> - `ProcessorContextImpl#getStateStore`
> (and maybe somewhere else: we should double check if there are other places 
> in the code like those).
> We should replace those exception with either {{StreamsException}} or with a 
> new exception type.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5671) Add StreamsBuilder and deprecate KStreamBuilder

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-5671:
--

 Summary: Add StreamsBuilder and deprecate KStreamBuilder
 Key: KAFKA-5671
 URL: https://issues.apache.org/jira/browse/KAFKA-5671
 Project: Kafka
  Issue Type: Sub-task
  Components: streams
Reporter: Matthias J. Sax
Assignee: Matthias J. Sax






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-3623) Make KStreamTestDriver extending from ExternalResource

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104009#comment-16104009
 ] 

ASF GitHub Bot commented on KAFKA-3623:
---

GitHub user johnma14 opened a pull request:

https://github.com/apache/kafka/pull/3589

KAFKA-3623: KStreamTestDriver extends ExternalResource

In the streams project, there are a number of unit tests that has duplicate
code with respect to the tearDown() method, in which it tries to close the
KStreamTestDriver connection. The goal of this changeset is to eliminate
this duplication by converting the KStreamTestDriver class to an 
ExternalResource
class which is the base class of JUnit Rule.

In every unit tests that calls KStreamTestDriver, we annotate the 
KStreamTestDriver
using @Rule annotation. In the KStreamTestDriver class, we override the 
after()
method. This after() method in turn calls the close() method which was 
previously
called in the tearDown() method in the unit tests. By annotating the 
KStreamTestDriver
as a @Rule, the after() method will be called automatically after every 
testcase.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/johnma14/kafka bug/KAFKA-3623

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3589.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3589


commit 3fb1cd54f507ac35cb0fb64677c9ad386ad89b3e
Author: johnma14 
Date:   2017-07-15T08:42:23Z

KAFKA-3623: KStreamTestDriver extends ExternalResource

In the streams project, there are a number of unit tests that has duplicate
code with respect to the tearDown() method, in which it tries to close the
KStreamTestDriver connection. The goal of this changeset is to eliminate
this duplication by converting the KStreamTestDriver class to an 
ExternalResource
class which is the base class of JUnit Rule.

In every unit tests that calls KStreamTestDriver, we annotate the 
KStreamTestDriver
using @Rule annotation. In the KStreamTestDriver class, we override the 
after()
method. This after() method in turn calls the close() method which was 
previously
called in the tearDown() method in the unit tests. By annotating the 
KStreamTestDriver
as a @Rule, the after() method will be called automatically after every 
testcase.




> Make KStreamTestDriver extending from ExternalResource
> --
>
> Key: KAFKA-3623
> URL: https://issues.apache.org/jira/browse/KAFKA-3623
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Mariam John
>  Labels: newbie, test
>
> In unit test we have lots of duplicate code for closing KStreamTestDriver 
> upon completing the test:
> {code}
> @After
> public void tearDown() {
> if (driver != null) {
> driver.close();
> }
> driver = null;
> }
> {code}
> One way to remove this duplicate code is to make KStreamTestDriver extending 
> from ExternalResource. By doing this we need to move the constructor logic 
> into a setup / init function and leave the construction empty.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5576) Support Power platform by updating rocksdb

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-5576:
---
Component/s: streams

> Support Power platform by updating rocksdb
> --
>
> Key: KAFKA-5576
> URL: https://issues.apache.org/jira/browse/KAFKA-5576
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 1.0.0
> Environment: $ cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=14.04
> DISTRIB_CODENAME=trusty
> DISTRIB_DESCRIPTION="Ubuntu 14.04.2 LTS"
> $ uname -a
> Linux pts00432-vm20 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 
> 17:42:36 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Yussuf Shaikh
>Assignee: Yussuf Shaikh
>Priority: Minor
> Fix For: 1.0.0
>
> Attachments: KAFKA-5576.patch, kafka-stream.txt
>
>
> Many test cases are failing with one to the following exceptions related to 
> rocksdb.
> 1. java.lang.NoClassDefFoundError: Could not initialize class 
> org.rocksdb.Options
> at 
> org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:119)
> at 
> org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:40)
> 2. java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni4427030040392983276.so: 
> /lib64/ld64.so.2: version `GLIBC_2.22' not found (required by 
> /tmp/librocksdbjni4427030040392983276.so)
> at java.lang.ClassLoader$NativeLibrary.load(Native Method)
> at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
> at java.lang.Runtime.load0(Runtime.java:809)
> at java.lang.System.load(System.java:1086)
> at 
> org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
> 3. java.lang.AssertionError: Condition not met within timeout 3. 
> Expecting 3 records from topic output-topic-2 while only received 0: []
> at 
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:274)
> at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:160)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5621) The producer should retry expired batches when retries are enabled

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103967#comment-16103967
 ] 

Jiangjie Qin commented on KAFKA-5621:
-

[~apurva] I am trying to understand the following statement
{quote}
On the other hand, for an application, partitions are not really independent 
(and especially so if you use transactions). If one partition is down, it makes 
sense to wait for it to be ready before continuing. So we would want to handle 
as many errors internally as possible. It would mean blocking sends once the 
queue is too large and not expiring batches in the queue. This simplifies the 
application programming model.
{quote}

Is it really different from applications and MM when a partition cannot make 
progress? It seems in both cases the users would want to know that at some 
point and handle it? I think retries are also for this purpose, otherwise we 
may block forever. If I understand right, what this ticket is proposing is just 
to extend the batch expiration time from request.timeout.ms to 
request.timeout.ms * reties. And KIP-91 proposes having an additional explicit 
configuration for that batch expiration time instead of deriving it from 
request timeout. They seem not quite different except that KIP-91 decouples the 
configurations from each other.

KAFKA-5494 is a good improvement. Regarding the error/anomaly handling, If we 
are willing to make public interface changes given the next release would be 
1.0.0, I am thinking of the following configurations:
1. request.timeout.ms - needed for wire timeout
2. expiry.ms - the expiration time for a message, this is an approximate time 
to expire a message if it cannot be sent out for whatever reason after it is 
ready for sending (the batch is ready). In the worst case a message would be 
expired in (expiry.ms + request.timeout.ms) after that message is ready for 
sending (note that user defines when the message is ready for sending by 
specifying linger.ms and batch.size). expiry.ms should be longer than 
request.timeout.ms, e.g. 2x or 3x.

The following configs are optional and will be decided by the producer if not 
specified:
3. min.retries - When this config is specified, the producer will at least 
retry for min.retries times even if that will cause the message stay in the 
producer longer than expiry.ms. This is to avoid the case that the producer 
cannot even retry at least once. When retry, the producer will do exponential 
backoff internally. This could be default to 1.

Hopefully this gives us a cleaner configuration set for the producer.

> The producer should retry expired batches when retries are enabled
> --
>
> Key: KAFKA-5621
> URL: https://issues.apache.org/jira/browse/KAFKA-5621
> Project: Kafka
>  Issue Type: Bug
>Reporter: Apurva Mehta
> Fix For: 1.0.0
>
>
> Today, when a batch is expired in the accumulator, a {{TimeoutException}} is 
> raised to the user.
> It might be better the producer to retry the expired batch rather up to the 
> configured number of retries. This is more intuitive from the user's point of 
> view. 
> Further the proposed behavior makes it easier for applications like mirror 
> maker to provide ordering guarantees even when batches expire. Today, they 
> would resend the expired batch and it would get added to the back of the 
> queue, causing the output ordering to be different from the input ordering.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (KAFKA-5563) Clarify handling of connector name in config

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sönke Liebau reassigned KAFKA-5563:
---

Assignee: Sönke Liebau

> Clarify handling of connector name in config 
> -
>
> Key: KAFKA-5563
> URL: https://issues.apache.org/jira/browse/KAFKA-5563
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Sönke Liebau
>Assignee: Sönke Liebau
>Priority: Minor
>
> The connector name is currently being stored in two places, once at the root 
> level of the connector and once in the config:
> {code:java}
> {
>   "name": "test",
>   "config": {
>   "connector.class": 
> "org.apache.kafka.connect.tools.MockSinkConnector",
>   "tasks.max": "3",
>   "topics": "test-topic",
>   "name": "test"
>   },
>   "tasks": [
>   {
>   "connector": "test",
>   "task": 0
>   }
>   ]
> }
> {code}
> If no name is provided in the "config" element, then the name from the root 
> level is [copied there when the connector is being 
> created|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/resources/ConnectorsResource.java#L95].
>  If however a name is provided in the config then it is not touched, which 
> means it is possible to create a connector with a different name at the root 
> level and in the config like this:
> {code:java}
> {
>   "name": "test1",
>   "config": {
>   "connector.class": 
> "org.apache.kafka.connect.tools.MockSinkConnector",
>   "tasks.max": "3",
>   "topics": "test-topic",
>   "name": "differentname"
>   },
>   "tasks": [
>   {
>   "connector": "test1",
>   "task": 0
>   }
>   ]
> }
> {code}
> I am not aware of any issues that this currently causes, but it is at least 
> confusing and probably not intended behavior and definitely bears potential 
> for bugs, if different functions take the name from different places.
> Would it make sense to add a check to reject requests that provide different 
> names in the request and the config section?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5637) Document compatibility and release policies

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sönke Liebau updated KAFKA-5637:

Description: 
We should document our compatibility and release policies in one place so that 
people have the correct expectations. This is generally important, but more so 
now that we are releasing 1.0.0.

I extracted the following topics from the mailing list thread as the ones that 
should be documented as a minimum: 

*Code stability*
* Explanation of stability annotations and their implications
* Explanation of what public apis are
* *Discussion point: * Do we want to keep the _unstable_ annotation or is 
_evolving_ sufficient going forward?

*Support duration*
* How long are versions supported?
* How far are bugfixes backported?
* How far are security fixes backported?
* How long are protocol versions supported by subsequent code versions?
* How long are older clients supported?
* How long are older brokers supported?

I will create an initial pull request to add a section to the documentation as 
basis for further discussion.

  was:
We should document our compatibility and release policies in one place so that 
people have the correct expectations. This is generally important, but more so 
now that we are releasing 1.0.0.

More details to come.

Component/s: documentation

> Document compatibility and release policies
> ---
>
> Key: KAFKA-5637
> URL: https://issues.apache.org/jira/browse/KAFKA-5637
> Project: Kafka
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Ismael Juma
>Assignee: Sönke Liebau
> Fix For: 1.0.0
>
>
> We should document our compatibility and release policies in one place so 
> that people have the correct expectations. This is generally important, but 
> more so now that we are releasing 1.0.0.
> I extracted the following topics from the mailing list thread as the ones 
> that should be documented as a minimum: 
> *Code stability*
> * Explanation of stability annotations and their implications
> * Explanation of what public apis are
> * *Discussion point: * Do we want to keep the _unstable_ annotation or is 
> _evolving_ sufficient going forward?
> *Support duration*
> * How long are versions supported?
> * How far are bugfixes backported?
> * How far are security fixes backported?
> * How long are protocol versions supported by subsequent code versions?
> * How long are older clients supported?
> * How long are older brokers supported?
> I will create an initial pull request to add a section to the documentation 
> as basis for further discussion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (KAFKA-5637) Document compatibility and release policies

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sönke Liebau reassigned KAFKA-5637:
---

Assignee: Sönke Liebau

> Document compatibility and release policies
> ---
>
> Key: KAFKA-5637
> URL: https://issues.apache.org/jira/browse/KAFKA-5637
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Sönke Liebau
> Fix For: 1.0.0
>
>
> We should document our compatibility and release policies in one place so 
> that people have the correct expectations. This is generally important, but 
> more so now that we are releasing 1.0.0.
> More details to come.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5634) Replica fetcher thread crashes due to OffsetOutOfRangeException

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-5634.

Resolution: Fixed

Issue resolved by pull request 3575
[https://github.com/apache/kafka/pull/3575]

> Replica fetcher thread crashes due to OffsetOutOfRangeException
> ---
>
> Key: KAFKA-5634
> URL: https://issues.apache.org/jira/browse/KAFKA-5634
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.0
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Critical
>  Labels: regression, reliability
> Fix For: 0.11.0.1
>
>
> We have seen the following exception recently:
> {code}
> kafka.common.KafkaException: error processing data for partition [foo,0] 
> offset 1459250
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:203)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:174)
> at scala.Option.foreach(Option.scala:257)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:174)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:171)
> at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply$mcV$sp(AbstractFetcherThread.scala:171)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:171)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:171)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213)
> at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:169)
> at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
> Caused by: org.apache.kafka.common.errors.OffsetOutOfRangeException: The 
> specified offset 1459250 is higher than the high watermark 1459032 of the 
> partition foo-0
> {code}
> The error check was added in the patch for KIP-107: 
> https://github.com/apache/kafka/commit/8b05ad406d4cba6a75d1683b6d8699c3ab28f9d6.
>  After investigation, we found that it is possible for the log start offset 
> on the leader to get ahead of the high watermark on the follower after 
> segment deletion. The check therefore seems incorrect. The impact of this bug 
> is that the fetcher thread crashes on the follower and the broker must be 
> restarted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5634) Replica fetcher thread crashes due to OffsetOutOfRangeException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103800#comment-16103800
 ] 

ASF GitHub Bot commented on KAFKA-5634:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3575


> Replica fetcher thread crashes due to OffsetOutOfRangeException
> ---
>
> Key: KAFKA-5634
> URL: https://issues.apache.org/jira/browse/KAFKA-5634
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.0
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Critical
>  Labels: regression, reliability
> Fix For: 0.11.0.1
>
>
> We have seen the following exception recently:
> {code}
> kafka.common.KafkaException: error processing data for partition [foo,0] 
> offset 1459250
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:203)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:174)
> at scala.Option.foreach(Option.scala:257)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:174)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:171)
> at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply$mcV$sp(AbstractFetcherThread.scala:171)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:171)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:171)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213)
> at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:169)
> at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
> Caused by: org.apache.kafka.common.errors.OffsetOutOfRangeException: The 
> specified offset 1459250 is higher than the high watermark 1459032 of the 
> partition foo-0
> {code}
> The error check was added in the patch for KIP-107: 
> https://github.com/apache/kafka/commit/8b05ad406d4cba6a75d1683b6d8699c3ab28f9d6.
>  After investigation, we found that it is possible for the log start offset 
> on the leader to get ahead of the high watermark on the follower after 
> segment deletion. The check therefore seems incorrect. The impact of this bug 
> is that the fetcher thread crashes on the follower and the broker must be 
> restarted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5666) Need feedback to user if consumption fails due to offsets.topic.replication.factor=3

[Apurva Mehta (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103644#comment-16103644
 ] 

Apurva Mehta commented on KAFKA-5666:
-

A solution for just the console consumer could be to have some precondition 
checks before entering the IO loop. 

In this case, we could check whether the offset topics exists using the admin 
client ([~yevabyzek]'s suggestion). 

If these preconditions are not met after a certain amount of time, the console 
consumer could exit with a proper error about which precondition failed. 

This seems like a better route for solving the developer experience problem: 
check for preconditions in the app itself because the environment for these 
applications is more likely to have problems relative to a prod environment.


> Need feedback to user if consumption fails due to 
> offsets.topic.replication.factor=3
> 
>
> Key: KAFKA-5666
> URL: https://issues.apache.org/jira/browse/KAFKA-5666
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Affects Versions: 0.11.0.0
>Reporter: Yeva Byzek
>  Labels: newbie
>
> Introduced in 0.11: The offsets.topic.replication.factor broker config is now 
> enforced upon auto topic creation. Internal auto topic creation will fail 
> with a GROUP_COORDINATOR_NOT_AVAILABLE error until the cluster size meets 
> this replication factor requirement.
> Issue: Default is setting offsets.topic.replication.factor=3, but in 
> development and docker environments where there is only 1 broker, the offsets 
> topic will fail to be created when a consumer tries to consume and no records 
> will be returned.  As a result, the user experience is bad.  The user may 
> have no idea about this setting change and enforcement, and they just see 
> that `kafka-console-consumer` hangs with ZERO output. It is true that the 
> broker log file will provide a message (e.g. {{ERROR [KafkaApi-1] Number of 
> alive brokers '1' does not meet the required replication factor '3' for the 
> offsets topic (configured via 'offsets.topic.replication.factor'). This error 
> can be ignored if the cluster is starting up and not all brokers are up yet. 
> (kafka.server.KafkaApis)}}) but many users do not have access to the log 
> files or know how to get them.
> Suggestion: give feedback to the user/app if offsets topic cannot be created. 
>  For example, after some timeout.
> Workaround:
> Set offsets.topic.replication.factor=3



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5667) kafka.api.LogDirFailureTest testProduceAfterLogDirFailure flaky test

[Ewen Cheslack-Postava (JIRA)]

Ewen Cheslack-Postava created KAFKA-5667:


 Summary: kafka.api.LogDirFailureTest testProduceAfterLogDirFailure 
flaky test
 Key: KAFKA-5667
 URL: https://issues.apache.org/jira/browse/KAFKA-5667
 Project: Kafka
  Issue Type: Bug
  Components: core
Reporter: Ewen Cheslack-Postava


We observed this on our Jenkins build against trunk:

{quote}
10:59:38 kafka.api.LogDirFailureTest > testProduceAfterLogDirFailure FAILED
10:59:38 java.lang.AssertionError
10:59:38 at org.junit.Assert.fail(Assert.java:86)
10:59:38 at org.junit.Assert.assertTrue(Assert.java:41)
10:59:38 at org.junit.Assert.assertTrue(Assert.java:52)
10:59:38 at 
kafka.api.LogDirFailureTest.testProduceAfterLogDirFailure(LogDirFailureTest.scala:76)
{quote}

Not sure yet how flaky the test is, we've only seen this once so far.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5667) kafka.api.LogDirFailureTest testProduceAfterLogDirFailure flaky test

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103632#comment-16103632
 ] 

Ewen Cheslack-Postava commented on KAFKA-5667:
--

[~lindong] I think this was added for the JBOD KIP

> kafka.api.LogDirFailureTest testProduceAfterLogDirFailure flaky test
> 
>
> Key: KAFKA-5667
> URL: https://issues.apache.org/jira/browse/KAFKA-5667
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Ewen Cheslack-Postava
>
> We observed this on our Jenkins build against trunk:
> {quote}
> 10:59:38 kafka.api.LogDirFailureTest > testProduceAfterLogDirFailure FAILED
> 10:59:38 java.lang.AssertionError
> 10:59:38 at org.junit.Assert.fail(Assert.java:86)
> 10:59:38 at org.junit.Assert.assertTrue(Assert.java:41)
> 10:59:38 at org.junit.Assert.assertTrue(Assert.java:52)
> 10:59:38 at 
> kafka.api.LogDirFailureTest.testProduceAfterLogDirFailure(LogDirFailureTest.scala:76)
> {quote}
> Not sure yet how flaky the test is, we've only seen this once so far.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5666) Need feedback to user if consumption fails due to offsets.topic.replication.factor=3

[Apurva Mehta (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103602#comment-16103602
 ] 

Apurva Mehta commented on KAFKA-5666:
-

I see the problem, but currently the {{GROUP_COORDINATOR_NOT_AVAILABLE}} is 
sort of an expected error code in normal operations: for instance, it is bound 
to happen when there is a rolling restart, a software upgrade, etc. 

Even if we logged it all the time, the current RPC framework doesn't really 
allow us to pass an error string between the server and client. So we would not 
be able to give any details which suggest that offsets topic can't be created. 

>From an application perspective, the consumer never returns any underlying 
>errors from {{poll}} unless they are fatal. This is actually desirable 
>behavior. 

All of this makes sense from the perspective of a system operating in 
production. I think the real solution here is to make sure that development 
images are configured correctly and work out of the box.

> Need feedback to user if consumption fails due to 
> offsets.topic.replication.factor=3
> 
>
> Key: KAFKA-5666
> URL: https://issues.apache.org/jira/browse/KAFKA-5666
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Affects Versions: 0.11.0.0
>Reporter: Yeva Byzek
>  Labels: newbie
>
> Introduced in 0.11: The offsets.topic.replication.factor broker config is now 
> enforced upon auto topic creation. Internal auto topic creation will fail 
> with a GROUP_COORDINATOR_NOT_AVAILABLE error until the cluster size meets 
> this replication factor requirement.
> Issue: Default is setting offsets.topic.replication.factor=3, but in 
> development and docker environments where there is only 1 broker, the offsets 
> topic will fail to be created when a consumer tries to consume and no records 
> will be returned.  As a result, the user experience is bad.  The user may 
> have no idea about this setting change and enforcement, and they just see 
> that `kafka-console-consumer` hangs with ZERO output. It is true that the 
> broker log file will provide a message (e.g. {{ERROR [KafkaApi-1] Number of 
> alive brokers '1' does not meet the required replication factor '3' for the 
> offsets topic (configured via 'offsets.topic.replication.factor'). This error 
> can be ignored if the cluster is starting up and not all brokers are up yet. 
> (kafka.server.KafkaApis)}}) but many users do not have access to the log 
> files or know how to get them.
> Suggestion: give feedback to the user/app if offsets topic cannot be created. 
>  For example, after some timeout.
> Workaround:
> Set offsets.topic.replication.factor=3



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-1944) Rename LogCleaner and related classes to LogCompactor

[Pranav Maniar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103571#comment-16103571
 ] 

Pranav Maniar commented on KAFKA-1944:
--

Is someone working on this?? If not I can take it up. 
I am new to kafka and this seems like a refactoring task, which will be easy to 
start with.. 

P.S. : This issue seems to be old, so this still needs to be refactored ??

> Rename LogCleaner and related classes to LogCompactor
> -
>
> Key: KAFKA-1944
> URL: https://issues.apache.org/jira/browse/KAFKA-1944
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Aravind Selvan
>  Labels: newbie
>
> Following a mailing list discussion:
> "the name LogCleaner is seriously misleading. Its more of a log compactor. 
> Deleting old logs happens elsewhere from what I've seen."
> Note that this may require renaming related classes, objects, configs and 
> metrics.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5666) Need feedback to user if consumption fails due to offsets.topic.replication.factor=3

[Yeva Byzek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yeva Byzek updated KAFKA-5666:
--
Component/s: consumer

> Need feedback to user if consumption fails due to 
> offsets.topic.replication.factor=3
> 
>
> Key: KAFKA-5666
> URL: https://issues.apache.org/jira/browse/KAFKA-5666
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Affects Versions: 0.11.0.0
>Reporter: Yeva Byzek
>  Labels: newbie
>
> Introduced in 0.11: The offsets.topic.replication.factor broker config is now 
> enforced upon auto topic creation. Internal auto topic creation will fail 
> with a GROUP_COORDINATOR_NOT_AVAILABLE error until the cluster size meets 
> this replication factor requirement.
> Issue: Default is setting offsets.topic.replication.factor=3, but in 
> development and docker environments where there is only 1 broker, the offsets 
> topic will fail to be created when a consumer tries to consume and no records 
> will be returned.  As a result, the user experience is bad.  The user may 
> have no idea about this setting change and enforcement, and they just see 
> that `kafka-console-consumer` hangs with ZERO output. It is true that the 
> broker log file will provide a message (e.g. {{ERROR [KafkaApi-1] Number of 
> alive brokers '1' does not meet the required replication factor '3' for the 
> offsets topic (configured via 'offsets.topic.replication.factor'). This error 
> can be ignored if the cluster is starting up and not all brokers are up yet. 
> (kafka.server.KafkaApis)}}) but many users do not have access to the log 
> files or know how to get them.
> Suggestion: give feedback to the user/app if offsets topic cannot be created. 
>  For example, after some timeout.
> Workaround:
> Set offsets.topic.replication.factor=3



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5666) Need feedback to user if consumption fails due to offsets.topic.replication.factor=3

[Yeva Byzek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yeva Byzek updated KAFKA-5666:
--
Component/s: clients

> Need feedback to user if consumption fails due to 
> offsets.topic.replication.factor=3
> 
>
> Key: KAFKA-5666
> URL: https://issues.apache.org/jira/browse/KAFKA-5666
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Affects Versions: 0.11.0.0
>Reporter: Yeva Byzek
>  Labels: newbie
>
> Introduced in 0.11: The offsets.topic.replication.factor broker config is now 
> enforced upon auto topic creation. Internal auto topic creation will fail 
> with a GROUP_COORDINATOR_NOT_AVAILABLE error until the cluster size meets 
> this replication factor requirement.
> Issue: Default is setting offsets.topic.replication.factor=3, but in 
> development and docker environments where there is only 1 broker, the offsets 
> topic will fail to be created when a consumer tries to consume and no records 
> will be returned.  As a result, the user experience is bad.  The user may 
> have no idea about this setting change and enforcement, and they just see 
> that `kafka-console-consumer` hangs with ZERO output. It is true that the 
> broker log file will provide a message (e.g. {{ERROR [KafkaApi-1] Number of 
> alive brokers '1' does not meet the required replication factor '3' for the 
> offsets topic (configured via 'offsets.topic.replication.factor'). This error 
> can be ignored if the cluster is starting up and not all brokers are up yet. 
> (kafka.server.KafkaApis)}}) but many users do not have access to the log 
> files or know how to get them.
> Suggestion: give feedback to the user/app if offsets topic cannot be created. 
>  For example, after some timeout.
> Workaround:
> Set offsets.topic.replication.factor=3



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5666) Need feedback to user if consumption fails due to offsets.topic.replication.factor=3

[Yeva Byzek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yeva Byzek updated KAFKA-5666:
--
Labels: newbie  (was: )

> Need feedback to user if consumption fails due to 
> offsets.topic.replication.factor=3
> 
>
> Key: KAFKA-5666
> URL: https://issues.apache.org/jira/browse/KAFKA-5666
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Affects Versions: 0.11.0.0
>Reporter: Yeva Byzek
>  Labels: newbie
>
> Introduced in 0.11: The offsets.topic.replication.factor broker config is now 
> enforced upon auto topic creation. Internal auto topic creation will fail 
> with a GROUP_COORDINATOR_NOT_AVAILABLE error until the cluster size meets 
> this replication factor requirement.
> Issue: Default is setting offsets.topic.replication.factor=3, but in 
> development and docker environments where there is only 1 broker, the offsets 
> topic will fail to be created when a consumer tries to consume and no records 
> will be returned.  As a result, the user experience is bad.  The user may 
> have no idea about this setting change and enforcement, and they just see 
> that `kafka-console-consumer` hangs with ZERO output. It is true that the 
> broker log file will provide a message (e.g. {{ERROR [KafkaApi-1] Number of 
> alive brokers '1' does not meet the required replication factor '3' for the 
> offsets topic (configured via 'offsets.topic.replication.factor'). This error 
> can be ignored if the cluster is starting up and not all brokers are up yet. 
> (kafka.server.KafkaApis)}}) but many users do not have access to the log 
> files or know how to get them.
> Suggestion: give feedback to the user/app if offsets topic cannot be created. 
>  For example, after some timeout.
> Workaround:
> Set offsets.topic.replication.factor=3



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5621) The producer should retry expired batches when retries are enabled

[Apurva Mehta (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103516#comment-16103516
 ] 

Apurva Mehta commented on KAFKA-5621:
-

[~sutambe] : We could name the second config {{batch.expiry.ms}}, I didn't 
remember the proposed name. 

I guess at this point we need to figure out which option we pick for satisfying 
a good out of the box experience for applications, while making tools like 
mirror maker still work optimally.

1. We have the option for reserving idempotence only for apps. So when 
idempotence is disabled KMM can be tuned just the way it is today.
2. We add the {{batch.expiry.ms}}. When it is defined by the user, we disable 
retries for expired batches. This will work for KMM. Applications which don't 
touch this setting will have their expired batches retried. 
3. ??? any other options for resolving these two use cases? 



> The producer should retry expired batches when retries are enabled
> --
>
> Key: KAFKA-5621
> URL: https://issues.apache.org/jira/browse/KAFKA-5621
> Project: Kafka
>  Issue Type: Bug
>Reporter: Apurva Mehta
> Fix For: 1.0.0
>
>
> Today, when a batch is expired in the accumulator, a {{TimeoutException}} is 
> raised to the user.
> It might be better the producer to retry the expired batch rather up to the 
> configured number of retries. This is more intuitive from the user's point of 
> view. 
> Further the proposed behavior makes it easier for applications like mirror 
> maker to provide ordering guarantees even when batches expire. Today, they 
> would resend the expired batch and it would get added to the back of the 
> queue, causing the output ordering to be different from the input ordering.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5666) Need feedback to user if consumption fails due to offsets.topic.replication.factor=3

[Yeva Byzek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yeva Byzek updated KAFKA-5666:
--
Description: 
Introduced in 0.11: The offsets.topic.replication.factor broker config is now 
enforced upon auto topic creation. Internal auto topic creation will fail with 
a GROUP_COORDINATOR_NOT_AVAILABLE error until the cluster size meets this 
replication factor requirement.

Issue: Default is setting offsets.topic.replication.factor=3, but in 
development and docker environments where there is only 1 broker, the offsets 
topic will fail to be created when a consumer tries to consume and no records 
will be returned.  As a result, the user experience is bad.  The user may have 
no idea about this setting change and enforcement, and they just see that 
`kafka-console-consumer` hangs with ZERO output. It is true that the broker log 
file will provide a message (e.g. {{ERROR [KafkaApi-1] Number of alive brokers 
'1' does not meet the required replication factor '3' for the offsets topic 
(configured via 'offsets.topic.replication.factor'). This error can be ignored 
if the cluster is starting up and not all brokers are up yet. 
(kafka.server.KafkaApis)}}) but many users do not have access to the log files 
or know how to get them.

Suggestion: give feedback to the user/app if offsets topic cannot be created.  
For example, after some timeout.

Workaround:
Set offsets.topic.replication.factor=3



  was:
Introduced in 0.11: The offsets.topic.replication.factor broker config is now 
enforced upon auto topic creation. Internal auto topic creation will fail with 
a GROUP_COORDINATOR_NOT_AVAILABLE error until the cluster size meets this 
replication factor requirement.

Result: Default is setting offsets.topic.replication.factor=3, but in 
development and docker environments where there is only 1 broker, the offsets 
topic will fail to be created when a consumer tries to consume and no records 
will be returned.

Issue: the user experience is bad.  The user may have no idea about this 
setting change and enforcement, and they just see that `kafka-console-consumer` 
hangs with ZERO output. It is true that the broker log file will provide a 
message (e.g. {{ERROR [KafkaApi-1] Number of alive brokers '1' does not meet 
the required replication factor '3' for the offsets topic (configured via 
'offsets.topic.replication.factor'). This error can be ignored if the cluster 
is starting up and not all brokers are up yet. (kafka.server.KafkaApis)}}) but 
many users do not have access to the log files or know how to get them.

Suggestion: give feedback to the user/app if offsets topic cannot be created.  
For example, after some timeout.

Workaround:
Set offsets.topic.replication.factor=3




> Need feedback to user if consumption fails due to 
> offsets.topic.replication.factor=3
> 
>
> Key: KAFKA-5666
> URL: https://issues.apache.org/jira/browse/KAFKA-5666
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Yeva Byzek
>
> Introduced in 0.11: The offsets.topic.replication.factor broker config is now 
> enforced upon auto topic creation. Internal auto topic creation will fail 
> with a GROUP_COORDINATOR_NOT_AVAILABLE error until the cluster size meets 
> this replication factor requirement.
> Issue: Default is setting offsets.topic.replication.factor=3, but in 
> development and docker environments where there is only 1 broker, the offsets 
> topic will fail to be created when a consumer tries to consume and no records 
> will be returned.  As a result, the user experience is bad.  The user may 
> have no idea about this setting change and enforcement, and they just see 
> that `kafka-console-consumer` hangs with ZERO output. It is true that the 
> broker log file will provide a message (e.g. {{ERROR [KafkaApi-1] Number of 
> alive brokers '1' does not meet the required replication factor '3' for the 
> offsets topic (configured via 'offsets.topic.replication.factor'). This error 
> can be ignored if the cluster is starting up and not all brokers are up yet. 
> (kafka.server.KafkaApis)}}) but many users do not have access to the log 
> files or know how to get them.
> Suggestion: give feedback to the user/app if offsets topic cannot be created. 
>  For example, after some timeout.
> Workaround:
> Set offsets.topic.replication.factor=3



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5666) Need feedback to user if consumption fails due to offsets.topic.replication.factor=3

[Yeva Byzek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yeva Byzek updated KAFKA-5666:
--
Description: 
Introduced in 0.11: The offsets.topic.replication.factor broker config is now 
enforced upon auto topic creation. Internal auto topic creation will fail with 
a GROUP_COORDINATOR_NOT_AVAILABLE error until the cluster size meets this 
replication factor requirement.

Result: Default is setting offsets.topic.replication.factor=3, but in 
development and docker environments where there is only 1 broker, the offsets 
topic will fail to be created when a consumer tries to consume and no records 
will be returned.

Issue: the user experience is bad.  The user may have no idea about this 
setting change and enforcement, and they just see that `kafka-console-consumer` 
hangs with ZERO output. It is true that the broker log file will provide a 
message (e.g. {{ERROR [KafkaApi-1] Number of alive brokers '1' does not meet 
the required replication factor '3' for the offsets topic (configured via 
'offsets.topic.replication.factor'). This error can be ignored if the cluster 
is starting up and not all brokers are up yet. (kafka.server.KafkaApis)}}) but 
many users do not have access to the log files or know how to get them.

Suggestion: give feedback to the user/app if offsets topic cannot be created.  
For example, after some timeout.

Workaround:
Set offsets.topic.replication.factor=3



  was:
Introduced in 0.11: The offsets.topic.replication.factor broker config is now 
enforced upon auto topic creation. Internal auto topic creation will fail with 
a GROUP_COORDINATOR_NOT_AVAILABLE error until the cluster size meets this 
replication factor requirement.

Result: Default is setting offsets.topic.replication.factor=3, but in 
development and docker environments where there is only 1 broker, the offsets 
topic will fail to be created when a consumer tries to consume and no records 
will be returned.

Issue: the user experience is bad.  The user may have no idea about this 
setting change and enforcement, and they just see that `kafka-console-consumer` 
hangs with ZERO output. 

Suggestion: consider giving feedback to the user/app if offsets topic cannot be 
created.  For example, after some timeout.

Workaround:
Set offsets.topic.replication.factor=3




> Need feedback to user if consumption fails due to 
> offsets.topic.replication.factor=3
> 
>
> Key: KAFKA-5666
> URL: https://issues.apache.org/jira/browse/KAFKA-5666
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Yeva Byzek
>
> Introduced in 0.11: The offsets.topic.replication.factor broker config is now 
> enforced upon auto topic creation. Internal auto topic creation will fail 
> with a GROUP_COORDINATOR_NOT_AVAILABLE error until the cluster size meets 
> this replication factor requirement.
> Result: Default is setting offsets.topic.replication.factor=3, but in 
> development and docker environments where there is only 1 broker, the offsets 
> topic will fail to be created when a consumer tries to consume and no records 
> will be returned.
> Issue: the user experience is bad.  The user may have no idea about this 
> setting change and enforcement, and they just see that 
> `kafka-console-consumer` hangs with ZERO output. It is true that the broker 
> log file will provide a message (e.g. {{ERROR [KafkaApi-1] Number of alive 
> brokers '1' does not meet the required replication factor '3' for the offsets 
> topic (configured via 'offsets.topic.replication.factor'). This error can be 
> ignored if the cluster is starting up and not all brokers are up yet. 
> (kafka.server.KafkaApis)}}) but many users do not have access to the log 
> files or know how to get them.
> Suggestion: give feedback to the user/app if offsets topic cannot be created. 
>  For example, after some timeout.
> Workaround:
> Set offsets.topic.replication.factor=3



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5666) Need feedback to user if consumption fails due to offsets.topic.replication.factor=3

[Yeva Byzek (JIRA)]

Yeva Byzek created KAFKA-5666:
-

 Summary: Need feedback to user if consumption fails due to 
offsets.topic.replication.factor=3
 Key: KAFKA-5666
 URL: https://issues.apache.org/jira/browse/KAFKA-5666
 Project: Kafka
  Issue Type: Improvement
Reporter: Yeva Byzek


Introduced in 0.11: The offsets.topic.replication.factor broker config is now 
enforced upon auto topic creation. Internal auto topic creation will fail with 
a GROUP_COORDINATOR_NOT_AVAILABLE error until the cluster size meets this 
replication factor requirement.

Result: Default is setting offsets.topic.replication.factor=3, but in 
development and docker environments where there is only 1 broker, the offsets 
topic will fail to be created when a consumer tries to consume and no records 
will be returned.

Issue: the user experience is bad.  The user may have no idea about this 
setting change and enforcement, and they just see that `kafka-console-consumer` 
hangs with ZERO output. 

Suggestion: consider giving feedback to the user/app if offsets topic cannot be 
created.  For example, after some timeout.

Workaround:
Set offsets.topic.replication.factor=3





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (KAFKA-5576) Support Power platform by updating rocksdb

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang reassigned KAFKA-5576:


Assignee: Yussuf Shaikh

> Support Power platform by updating rocksdb
> --
>
> Key: KAFKA-5576
> URL: https://issues.apache.org/jira/browse/KAFKA-5576
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: $ cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=14.04
> DISTRIB_CODENAME=trusty
> DISTRIB_DESCRIPTION="Ubuntu 14.04.2 LTS"
> $ uname -a
> Linux pts00432-vm20 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 
> 17:42:36 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Yussuf Shaikh
>Assignee: Yussuf Shaikh
>Priority: Minor
> Fix For: 1.0.0
>
> Attachments: KAFKA-5576.patch, kafka-stream.txt
>
>
> Many test cases are failing with one to the following exceptions related to 
> rocksdb.
> 1. java.lang.NoClassDefFoundError: Could not initialize class 
> org.rocksdb.Options
> at 
> org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:119)
> at 
> org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:40)
> 2. java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni4427030040392983276.so: 
> /lib64/ld64.so.2: version `GLIBC_2.22' not found (required by 
> /tmp/librocksdbjni4427030040392983276.so)
> at java.lang.ClassLoader$NativeLibrary.load(Native Method)
> at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
> at java.lang.Runtime.load0(Runtime.java:809)
> at java.lang.System.load(System.java:1086)
> at 
> org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
> 3. java.lang.AssertionError: Condition not met within timeout 3. 
> Expecting 3 records from topic output-topic-2 while only received 0: []
> at 
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:274)
> at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:160)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5621) The producer should retry expired batches when retries are enabled

[Sumant Tambe (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103460#comment-16103460
 ] 

Sumant Tambe commented on KAFKA-5621:
-

{{queue.time.ms}} sound a lot like {{linger.ms}} to me. It may be confusing for 
those who are new to Kafka. I naturally biased towards {{batch.expiry.ms}} 
which  proposed in KIP-91. For backwards compatibility, the default should be 
equal to {{request.timeout.ms}}, I think. 

> The producer should retry expired batches when retries are enabled
> --
>
> Key: KAFKA-5621
> URL: https://issues.apache.org/jira/browse/KAFKA-5621
> Project: Kafka
>  Issue Type: Bug
>Reporter: Apurva Mehta
> Fix For: 1.0.0
>
>
> Today, when a batch is expired in the accumulator, a {{TimeoutException}} is 
> raised to the user.
> It might be better the producer to retry the expired batch rather up to the 
> configured number of retries. This is more intuitive from the user's point of 
> view. 
> Further the proposed behavior makes it easier for applications like mirror 
> maker to provide ordering guarantees even when batches expire. Today, they 
> would resend the expired batch and it would get added to the back of the 
> queue, causing the output ordering to be different from the input ordering.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5648) make Merger extend Aggregator

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103319#comment-16103319
 ] 

Matthias J. Sax commented on KAFKA-5648:


The issue is, if one implements old {{Merger}} and does a hotswap of the 
Streams library without recompiling the code. The class ID will not match as 
the class hierarchy did change. The existing class would only implement 
{{Merger}} but not {{Merger extends Aggregator}}.

You are right, that if one recompiles the code, not code change is required to 
pass the build.

> make Merger extend Aggregator
> -
>
> Key: KAFKA-5648
> URL: https://issues.apache.org/jira/browse/KAFKA-5648
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Clemens Valiente
>Assignee: Clemens Valiente
>Priority: Minor
>
> Hi,
> I suggest that Merger should extend Aggregator.
> reason:
> Both classes usually do very similar things. A merger takes two sessions and 
> combines them, an aggregator takes an existing session and aggregates new 
> values into it.
> in some use cases it is actually the same thing, e.g.:
>  -> .map() to  -> 
> .groupByKey().aggregate() to 
> In this case both merger and aggregator do the same thing: take two lists and 
> combine them into one.
> With the proposed change we could pass the Merger as both the merger and 
> aggregator to the .aggregate() method and keep our business logic within one 
> merger class.
> Or in other words: The Merger is simply an Aggregator that happens to 
> aggregate two objects of the same class



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5549) Explain that `client.id` is just used as a prefix within Streams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103311#comment-16103311
 ] 

ASF GitHub Bot commented on KAFKA-5549:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3544


> Explain that `client.id` is just used as a prefix within Streams
> 
>
> Key: KAFKA-5549
> URL: https://issues.apache.org/jira/browse/KAFKA-5549
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Priority: Trivial
>  Labels: beginner, newbie
> Fix For: 1.0.0
>
>
> We should explain, that {{client.id}} is used as a prefix for internal 
> consumer, producer, and restore-consumer and not reuse 
> {{CommonClientConfigs.CLIENT_ID_DOC}} within {{StreamsConfig}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5549) Explain that `client.id` is just used as a prefix within Streams

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy resolved KAFKA-5549.
---
   Resolution: Fixed
Fix Version/s: 1.0.0

Issue resolved by pull request 3544
[https://github.com/apache/kafka/pull/3544]

> Explain that `client.id` is just used as a prefix within Streams
> 
>
> Key: KAFKA-5549
> URL: https://issues.apache.org/jira/browse/KAFKA-5549
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Priority: Trivial
>  Labels: beginner, newbie
> Fix For: 1.0.0
>
>
> We should explain, that {{client.id}} is used as a prefix for internal 
> consumer, producer, and restore-consumer and not reuse 
> {{CommonClientConfigs.CLIENT_ID_DOC}} within {{StreamsConfig}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-1595) Remove deprecated and slower scala JSON parser

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103191#comment-16103191
 ] 

ASF GitHub Bot commented on KAFKA-1595:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/83


> Remove deprecated and slower scala JSON parser
> --
>
> Key: KAFKA-1595
> URL: https://issues.apache.org/jira/browse/KAFKA-1595
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.1.1
>Reporter: Jagbir
>Assignee: Ismael Juma
>  Labels: newbie
> Fix For: 1.0.0
>
>
> The following issue is created as a follow up suggested by Jun Rao
> in a kafka news group message with the Subject
> "Blocking Recursive parsing from 
> kafka.consumer.TopicCount$.constructTopicCount"
> SUMMARY:
> An issue was detected in a typical cluster of 3 kafka instances backed
> by 3 zookeeper instances (kafka version 0.8.1.1, scala version 2.10.3,
> java version 1.7.0_65). On consumer end, when consumers get recycled,
> there is a troubling JSON parsing recursion which takes a busy lock and
> blocks consumers thread pool.
> In 0.8.1.1 scala client library ZookeeperConsumerConnector.scala:355 takes
> a global lock (0xd3a7e1d0) during the rebalance, and fires an
> expensive JSON parsing, while keeping the other consumers from shutting
> down, see, e.g,
> at 
> kafka.consumer.ZookeeperConsumerConnector.shutdown(ZookeeperConsumerConnector.scala:161)
> The deep recursive JSON parsing should be deprecated in favor
> of a better JSON parser, see, e.g,
> http://engineering.ooyala.com/blog/comparing-scala-json-libraries?
> DETAILS:
> The first dump is for a recursive blocking thread holding the lock for 
> 0xd3a7e1d0
> and the subsequent dump is for a waiting thread.
> (Please grep for 0xd3a7e1d0 to see the locked object.)
> Â 
> -8<-
> "Sa863f22b1e5hjh6788991800900b34545c_profile-a-prod1-s-140789080845312-c397945e8_watcher_executor"
> prio=10 tid=0x7f24dc285800 nid=0xda9 runnable [0x7f249e40b000]
> java.lang.Thread.State: RUNNABLE
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.p$7(Parsers.scala:722)
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.continue$1(Parsers.scala:726)
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.apply(Parsers.scala:737)
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.apply(Parsers.scala:721)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Success.flatMapWithNext(Parsers.scala:142)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$flatMap$1.apply(Parsers.scala:239)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$flatMap$1.apply(Parsers.scala:239)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$flatMap$1.apply(Parsers.scala:239)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$flatMap$1.apply(Parsers.scala:239)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at 

[jira] [Updated] (KAFKA-3806) Adjust default values of log.retention.hours and offsets.retention.minutes

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3806:
---
Fix Version/s: 1.0.0

> Adjust default values of log.retention.hours and offsets.retention.minutes
> --
>
> Key: KAFKA-3806
> URL: https://issues.apache.org/jira/browse/KAFKA-3806
> Project: Kafka
>  Issue Type: Improvement
>  Components: config
>Affects Versions: 0.9.0.1, 0.10.0.0
>Reporter: Michal Turek
>Priority: Minor
> Fix For: 1.0.0
>
>
> Combination of default values of log.retention.hours (168 hours = 7 days) and 
> offsets.retention.minutes (1440 minutes = 1 day) may be dangerous in special 
> cases. Offset retention should be always greater than log retention.
> We have observed the following scenario and issue:
> - Producing of data to a topic was disabled two days ago by producer update, 
> topic wasn't deleted.
> - Consumer consumed all data and properly committed offsets to Kafka.
> - Consumer made no more offset commits for that topic because there was no 
> more incoming data and there was nothing to confirm. (We have auto-commit 
> disabled, I'm not sure how behaves enabled auto-commit.)
> - After one day: Kafka cleared too old offsets according to 
> offsets.retention.minutes.
> - After two days: Long-term running consumer was restarted after update, it 
> didn't find any committed offsets for that topic since they were deleted by 
> offsets.retention.minutes so it started consuming from the beginning.
> - The messages were still in Kafka due to larger log.retention.hours, about 5 
> days of messages were read again.
> Known workaround to solve this issue:
> - Explicitly configure log.retention.hours and offsets.retention.minutes, 
> don't use defaults.
> Proposals:
> - Prolong default value of offsets.retention.minutes to be at least twice 
> larger than log.retention.hours.
> - Check these values during Kafka startup and log a warning if 
> offsets.retention.minutes is smaller than log.retention.hours.
> - Add a note to migration guide about differences between storing of offsets 
> in ZooKeeper and Kafka (http://kafka.apache.org/documentation.html#upgrade).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (KAFKA-4827) Kafka connect: error with special characters in connector name

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sönke Liebau reassigned KAFKA-4827:
---

Assignee: Sönke Liebau

> Kafka connect: error with special characters in connector name
> --
>
> Key: KAFKA-4827
> URL: https://issues.apache.org/jira/browse/KAFKA-4827
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.1.0
>Reporter: Aymeric Bouvet
>Assignee: Sönke Liebau
>Priority: Minor
>
> When creating a connector, if the connector name (and possibly other 
> properties) end with a carriage return, kafka-connect will create the config 
> but report error
> {code}
> cat << EOF > file-connector.json
> {
>   "name": "file-connector\r",
>   "config": {
> "topic": "kafka-connect-logs\r",
> "tasks.max": "1",
> "file": "/var/log/ansible-confluent/connect.log",
> "connector.class": 
> "org.apache.kafka.connect.file.FileStreamSourceConnector"
>   }
> }
> EOF
> curl -X POST -H "Content-Type: application/json" -H "Accept: 
> application/json" -d @file-connector.json localhost:8083/connectors 
> {code}
> returns an error 500  and log the following
> {code}
> [2017-03-01 18:25:23,895] WARN  (org.eclipse.jetty.servlet.ServletHandler)
> javax.servlet.ServletException: java.lang.IllegalArgumentException: Illegal 
> character in path at index 27: /connectors/file-connector4
> at 
> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:489)
> at 
> org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
> at 
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
> at 
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
> at 
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
> at 
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at 
> org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
> at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
> at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at 
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
> at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Illegal character in path at 
> index 27: /connectors/file-connector4
> at java.net.URI.create(URI.java:852)
> at 
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.createConnector(ConnectorsResource.java:100)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
> at 
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
> at 
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
> at 
> 

[jira] [Commented] (KAFKA-4827) Kafka connect: error with special characters in connector name

[JIRA]

[ 
https://issues.apache.org/jira/browse/KAFKA-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103087#comment-16103087
 ] 

Sönke Liebau commented on KAFKA-4827:
-

All true :)

I'll create a dedicated jira and kick off a discussion on the mailing list, as 
I think this jira is too narrow and should be solved shortly when the fix for 
KAFKA-4930 is merged.

> Kafka connect: error with special characters in connector name
> --
>
> Key: KAFKA-4827
> URL: https://issues.apache.org/jira/browse/KAFKA-4827
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.1.0
>Reporter: Aymeric Bouvet
>Priority: Minor
>
> When creating a connector, if the connector name (and possibly other 
> properties) end with a carriage return, kafka-connect will create the config 
> but report error
> {code}
> cat << EOF > file-connector.json
> {
>   "name": "file-connector\r",
>   "config": {
> "topic": "kafka-connect-logs\r",
> "tasks.max": "1",
> "file": "/var/log/ansible-confluent/connect.log",
> "connector.class": 
> "org.apache.kafka.connect.file.FileStreamSourceConnector"
>   }
> }
> EOF
> curl -X POST -H "Content-Type: application/json" -H "Accept: 
> application/json" -d @file-connector.json localhost:8083/connectors 
> {code}
> returns an error 500  and log the following
> {code}
> [2017-03-01 18:25:23,895] WARN  (org.eclipse.jetty.servlet.ServletHandler)
> javax.servlet.ServletException: java.lang.IllegalArgumentException: Illegal 
> character in path at index 27: /connectors/file-connector4
> at 
> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:489)
> at 
> org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
> at 
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
> at 
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
> at 
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
> at 
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at 
> org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
> at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
> at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at 
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
> at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Illegal character in path at 
> index 27: /connectors/file-connector4
> at java.net.URI.create(URI.java:852)
> at 
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.createConnector(ConnectorsResource.java:100)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
> at 
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
> at 
> 

[jira] [Updated] (KAFKA-5576) Support Power platform by updating rocksdb

[Yussuf Shaikh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yussuf Shaikh updated KAFKA-5576:
-
Attachment: kafka-stream.txt

> Support Power platform by updating rocksdb
> --
>
> Key: KAFKA-5576
> URL: https://issues.apache.org/jira/browse/KAFKA-5576
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: $ cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=14.04
> DISTRIB_CODENAME=trusty
> DISTRIB_DESCRIPTION="Ubuntu 14.04.2 LTS"
> $ uname -a
> Linux pts00432-vm20 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 
> 17:42:36 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Yussuf Shaikh
>Priority: Minor
> Fix For: 1.0.0
>
> Attachments: KAFKA-5576.patch, kafka-stream.txt
>
>
> Many test cases are failing with one to the following exceptions related to 
> rocksdb.
> 1. java.lang.NoClassDefFoundError: Could not initialize class 
> org.rocksdb.Options
> at 
> org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:119)
> at 
> org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:40)
> 2. java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni4427030040392983276.so: 
> /lib64/ld64.so.2: version `GLIBC_2.22' not found (required by 
> /tmp/librocksdbjni4427030040392983276.so)
> at java.lang.ClassLoader$NativeLibrary.load(Native Method)
> at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
> at java.lang.Runtime.load0(Runtime.java:809)
> at java.lang.System.load(System.java:1086)
> at 
> org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
> 3. java.lang.AssertionError: Condition not met within timeout 3. 
> Expecting 3 records from topic output-topic-2 while only received 0: []
> at 
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:274)
> at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:160)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Issue Comment Deleted] (KAFKA-5016) Consumer hang in poll method while rebalancing is in progress

[Domenico Di Giulio (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Domenico Di Giulio updated KAFKA-5016:
--
Comment: was deleted

(was: I am currently out of the office, with no access to my e-mail.
I will be back at work on July 27.

** Le e-mail provenienti dalla Banca d'Italia sono trasmesse in buona fede e 
non comportano alcun vincolo nè creano obblighi per la Banca stessa, salvo che 
ciò non sia espressamente previsto da un accordo scritto. Questa e-mail è 
confidenziale. Qualora l'avesse ricevuta per errore, La preghiamo di 
comunicarne via e-mail la ricezione al mittente e di distruggere il contenuto. 
La informiamo inoltre che l'utilizzo non autorizzato del messaggio o dei suoi 
allegati potrebbe costituire reato. Grazie per la collaborazione.
-- E-mail from Bank of Italy are sent in good faith but they are neither 
binding on the Bank nor to be understood as creating any obligation on its part 
except where provided for in a written agreement. This e-mail is confidential. 
If you have received it by mistake, please inform the sender by reply e-mail 
and delete it from your system. Please also note that the unauthorized 
disclosure or use of the message or any attachments could be an offence. Thank 
you for your cooperation. **
)

> Consumer hang in poll method while rebalancing is in progress
> -
>
> Key: KAFKA-5016
> URL: https://issues.apache.org/jira/browse/KAFKA-5016
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.1.0, 0.10.2.0
>Reporter: Domenico Di Giulio
>Assignee: Vahid Hashemian
> Attachments: Kafka 0.10.2.0 Issue (TRACE) - Server + Client.txt, 
> Kafka 0.10.2.0 Issue (TRACE).txt, KAFKA_5016.java
>
>
> After moving to Kafka 0.10.2.0, it looks like I'm experiencing a hang in the 
> rebalancing code. 
> This is a test case, not (still) production code. It does the following with 
> a single-partition topic and two consumers in the same group:
> 1) a topic with one partition is forced to be created (auto-created)
> 2) a producer is used to write 10 messages
> 3) the first consumer reads all the messages and commits
> 4) the second consumer attempts a poll() and hangs indefinitely
> The same issue can't be found with 0.10.0.0.
> See the attached logs at TRACE level. Look for "SERVER HANGS" to see where 
> the hang is found: when this happens, the client keeps failing any hearbeat 
> attempt, as the rebalancing is in progress, and the poll method hangs 
> indefinitely.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5611) One or more consumers in a consumer-group stop consuming after rebalancing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102758#comment-16102758
 ] 

ASF GitHub Bot commented on KAFKA-5611:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3571


> One or more consumers in a consumer-group stop consuming after rebalancing
> --
>
> Key: KAFKA-5611
> URL: https://issues.apache.org/jira/browse/KAFKA-5611
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.0
>Reporter: Panos Skianis
>Assignee: Jason Gustafson
>  Labels: reliability
> Fix For: 0.11.0.1
>
> Attachments: bad-server-with-more-logging-1.tar.gz, kka02, Server 1, 
> Server 2, Server 3
>
>
> Scenario: 
>   - 3 zookeepers, 4 Kafkas. 0.10.2.0, with 0.9.0 compatibility still on 
> (other apps need it but the one mentioned below is already on kafka 0.10.2.0  
> client).
>   - 3 servers running 1 consumer each under the same consumer groupId. 
>   - Servers seem to be consuming messages happily but then there is a timeout 
> to an external service that causes our app to restart the Kafka Consumer on 
> one of the servers (this is by design). That causes rebalancing of the group 
> and upon restart of one of the Consumers seem to "block".
>   - Server 3 is where the problems occur.
>   - Problem fixes itself either by restarting one of the 3 servers or cause 
> the group to rebalance again by using the console consumer with the 
> autocommit set to false and using the same group.
>  
> Note: 
>  - Haven't managed to recreate it at will yet.
>  - Mainly happens in production environment, often enough. Hence I do not 
> have any logs with DEBUG/TRACE statements yet.
>  - Extracts from log of each app server are attached. Also the log of the 
> kafka that seems to be dealing with the related group and generations.
>  - See COMMENT lines in the files for further info.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)