[jira] [Commented] (KAFKA-3994) Deadlock between consumer heartbeat expiration and offset commit.

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678587#comment-15678587
 ] 

Jiangjie Qin commented on KAFKA-3994:
-

[~mjuarez] Thanks for the update. Good to know. We are still working on the 
patch, though :)

> Deadlock between consumer heartbeat expiration and offset commit.
> -
>
> Key: KAFKA-3994
> URL: https://issues.apache.org/jira/browse/KAFKA-3994
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Jiangjie Qin
>Assignee: Jason Gustafson
>Priority: Critical
> Fix For: 0.10.1.1
>
>
> I got the following stacktraces from ConsumerBounceTest
> {code}
> ...
> "Test worker" #12 prio=5 os_prio=0 tid=0x7fbb28b7f000 nid=0x427c runnable 
> [0x7fbb06445000]
>java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0003d48bcbc0> (a sun.nio.ch.Util$2)
> - locked <0x0003d48bcbb0> (a 
> java.util.Collections$UnmodifiableSet)
> - locked <0x0003d48bbd28> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.kafka.common.network.Selector.select(Selector.java:454)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:277)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:179)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:411)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1086)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1054)
> at 
> kafka.api.ConsumerBounceTest.consumeWithBrokerFailures(ConsumerBounceTest.scala:103)
> at 
> kafka.api.ConsumerBounceTest.testConsumptionWithBrokerFailures(ConsumerBounceTest.scala:70)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
> at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
> at 
> 

Build failed in Jenkins: kafka-trunk-jdk8 #1052

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-4420; Group StopReplicaRequests for partitions on the same broker

--
[...truncated 14380 lines...]

org.apache.kafka.streams.KafkaStreamsTest > 
shouldNotGetTaskWithKeyAndSerializerWhenNotRunning PASSED

org.apache.kafka.streams.KafkaStreamsTest > 
shouldReturnFalseOnCloseWhenThreadsHaventTerminated STARTED

org.apache.kafka.streams.KafkaStreamsTest > 
shouldReturnFalseOnCloseWhenThreadsHaventTerminated PASSED

org.apache.kafka.streams.KafkaStreamsTest > 
shouldNotGetAllTasksWithStoreWhenNotRunning STARTED

org.apache.kafka.streams.KafkaStreamsTest > 
shouldNotGetAllTasksWithStoreWhenNotRunning PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCannotStartOnceClosed STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCannotStartOnceClosed PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCleanup STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCleanup PASSED

org.apache.kafka.streams.KafkaStreamsTest > testStartAndClose STARTED

org.apache.kafka.streams.KafkaStreamsTest > testStartAndClose PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCloseIsIdempotent STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCloseIsIdempotent PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCannotCleanupWhileRunning 
STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCannotCleanupWhileRunning PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCannotStartTwice STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCannotStartTwice PASSED

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[0] STARTED

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[0] PASSED

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[1] STARTED

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[0] FAILED
java.lang.AssertionError: Condition not met within timeout 3. waiting 
for store count-by-key
at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:279)
at 
org.apache.kafka.streams.integration.QueryableStateIntegrationTest.shouldNotMakeStoreAvailableUntilAllStoresAvailable(QueryableStateIntegrationTest.java:488)

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[1] PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testShouldReadFromRegexAndNamedTopics STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testShouldReadFromRegexAndNamedTopics PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenCreated STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenCreated PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testMultipleConsumersCanReadFromPartitionedTopic STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testMultipleConsumersCanReadFromPartitionedTopic PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenDeleted STARTED

Build failed in Jenkins: kafka-trunk-jdk7 #1701

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-4420; Group StopReplicaRequests for partitions on the same broker

--
[...truncated 14359 lines...]
org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest > 
shouldGetInstanceWithKeyWithMergedStreams PASSED

org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest > 
shouldNotThrowNPEWhenOnChangeNotCalled STARTED

org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest > 
shouldNotThrowNPEWhenOnChangeNotCalled PASSED

org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest > 
shouldThrowIfStoreNameIsNull STARTED

org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest > 
shouldThrowIfStoreNameIsNull PASSED

org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest > 
shouldReturnNotAvailableWhenClusterIsEmpty STARTED

org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest > 
shouldReturnNotAvailableWhenClusterIsEmpty PASSED

org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest > 
shouldReturnEmptyCollectionOnGetAllInstancesWithStoreWhenStoreDoesntExist 
STARTED

org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest > 
shouldReturnEmptyCollectionOnGetAllInstancesWithStoreWhenStoreDoesntExist PASSED

org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest > 
shouldReturnNullOnGetWithKeyWhenStoreDoesntExist STARTED

org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest > 
shouldReturnNullOnGetWithKeyWhenStoreDoesntExist PASSED

org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest > 
shouldThrowIfStreamPartitionerIsNull STARTED

org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest > 
shouldThrowIfStreamPartitionerIsNull PASSED

org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest > 
shouldGetInstanceWithKeyAndCustomPartitioner STARTED

org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest > 
shouldGetInstanceWithKeyAndCustomPartitioner PASSED

org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest > 
shouldThrowIfStoreNameIsNullOnGetAllInstancesWithStore STARTED

org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest > 
shouldThrowIfStoreNameIsNullOnGetAllInstancesWithStore PASSED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowProcessorStateExceptionOnInitializeOffsetsWhenAuthorizationException 
STARTED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowProcessorStateExceptionOnInitializeOffsetsWhenAuthorizationException 
PASSED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowProcessorStateExceptionOnInitializeOffsetsWhenKafkaException STARTED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowProcessorStateExceptionOnInitializeOffsetsWhenKafkaException PASSED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowWakeupExceptionOnInitializeOffsetsWhenWakeupException STARTED

org.apache.kafka.streams.processor.internals.AbstractTaskTest > 
shouldThrowWakeupExceptionOnInitializeOffsetsWhenWakeupException PASSED

org.apache.kafka.streams.processor.internals.InternalTopicConfigTest > 
shouldHaveCompactionPropSetIfSupplied STARTED

org.apache.kafka.streams.processor.internals.InternalTopicConfigTest > 
shouldHaveCompactionPropSetIfSupplied PASSED

org.apache.kafka.streams.processor.internals.InternalTopicConfigTest > 
shouldThrowIfNameIsNull STARTED

org.apache.kafka.streams.processor.internals.InternalTopicConfigTest > 
shouldThrowIfNameIsNull PASSED

org.apache.kafka.streams.processor.internals.InternalTopicConfigTest > 
shouldConfigureRetentionMsWithAdditionalRetentionWhenCompactAndDelete STARTED

org.apache.kafka.streams.processor.internals.InternalTopicConfigTest > 
shouldConfigureRetentionMsWithAdditionalRetentionWhenCompactAndDelete PASSED

org.apache.kafka.streams.processor.internals.InternalTopicConfigTest > 
shouldBeCompactedIfCleanupPolicyCompactOrCompactAndDelete STARTED

org.apache.kafka.streams.processor.internals.InternalTopicConfigTest > 
shouldBeCompactedIfCleanupPolicyCompactOrCompactAndDelete PASSED

org.apache.kafka.streams.processor.internals.InternalTopicConfigTest > 
shouldNotBeCompactedWhenCleanupPolicyIsDelete STARTED

org.apache.kafka.streams.processor.internals.InternalTopicConfigTest > 
shouldNotBeCompactedWhenCleanupPolicyIsDelete PASSED

org.apache.kafka.streams.processor.internals.InternalTopicConfigTest > 
shouldHavePropertiesSuppliedByUser STARTED

org.apache.kafka.streams.processor.internals.InternalTopicConfigTest > 
shouldHavePropertiesSuppliedByUser PASSED

org.apache.kafka.streams.processor.internals.InternalTopicConfigTest > 
shouldUseCleanupPolicyFromConfigIfSupplied STARTED

org.apache.kafka.streams.processor.internals.InternalTopicConfigTest > 

[jira] [Updated] (KAFKA-3866) KerberosLogin refresh time bug and other improvements

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3866:
---
Fix Version/s: (was: 0.10.1.1)
   0.10.2.0

> KerberosLogin refresh time bug and other improvements
> -
>
> Key: KAFKA-3866
> URL: https://issues.apache.org/jira/browse/KAFKA-3866
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.10.0.0
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.10.2.0
>
>
> ZOOKEEPER-2295 describes a bug in the Kerberos refresh time logic that is 
> also present in our KerberosLogin class. While looking at the code, I found a 
> number of things that could be improved. More details in the PR.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4362) Consumer can fail after reassignment of the offsets topic partition

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4362:
---
Fix Version/s: 0.10.1.1

> Consumer can fail after reassignment of the offsets topic partition
> ---
>
> Key: KAFKA-4362
> URL: https://issues.apache.org/jira/browse/KAFKA-4362
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Joel Koshy
>Assignee: Mayuresh Gharat
> Fix For: 0.10.1.1
>
>
> When a consumer offsets topic partition reassignment completes, an offset 
> commit shows this:
> {code}
> java.lang.IllegalArgumentException: Message format version for partition 100 
> not found
> at 
> kafka.coordinator.GroupMetadataManager$$anonfun$14.apply(GroupMetadataManager.scala:633)
>  ~[kafka_2.10.jar:?]
> at 
> kafka.coordinator.GroupMetadataManager$$anonfun$14.apply(GroupMetadataManager.scala:633)
>  ~[kafka_2.10.jar:?]
> at scala.Option.getOrElse(Option.scala:120) ~[scala-library-2.10.4.jar:?]
> at 
> kafka.coordinator.GroupMetadataManager.kafka$coordinator$GroupMetadataManager$$getMessageFormatVersionAndTimestamp(GroupMetadataManager.scala:632)
>  ~[kafka_2.10.jar:?]
> at 
> ...
> {code}
> The issue is that the replica has been deleted so the 
> {{GroupMetadataManager.getMessageFormatVersionAndTimestamp}} throws this 
> exception instead which propagates as an unknown error.
> Unfortunately consumers don't respond to this and will fail their offset 
> commits.
> One workaround in the above situation is to bounce the cluster - the consumer 
> will be forced to rediscover the group coordinator.
> (Incidentally, the message incorrectly prints the number of partitions 
> instead of the actual partition.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4399) Deadlock between cleanupGroupMetadata and offset commit

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4399:
---
Fix Version/s: 0.10.1.1

> Deadlock between cleanupGroupMetadata and offset commit
> ---
>
> Key: KAFKA-4399
> URL: https://issues.apache.org/jira/browse/KAFKA-4399
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Alexey Ozeritskiy
>Priority: Blocker
> Fix For: 0.10.1.1
>
> Attachments: deadlock-stack
>
>
> We have upgraded our clusters to 0.10.1.0 and got deadlock issue.
> We thought it smth like https://issues.apache.org/jira/browse/KAFKA-3994, but 
> patch did not help us and our stacks is different. I think it is other issue.
> Stack traces attached



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4420) Group StopReplicaRequests for partitions on the same broker into one StopReplicaRequest

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-4420.

Resolution: Fixed

> Group StopReplicaRequests for partitions on the same broker into one 
> StopReplicaRequest
> ---
>
> Key: KAFKA-4420
> URL: https://issues.apache.org/jira/browse/KAFKA-4420
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 0.10.2.0
>
>
> Currently if a broker of 1000 partitions want to do controlled shutdown, 
> controller will send 1000 StopReplicaRequest to that broker. This is 
> inefficient. Controller only need to send one StopReplicaRequest and include 
> those 1000 partitions in this request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4420) Group StopReplicaRequests for partitions on the same broker into one StopReplicaRequest

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4420:
---
Fix Version/s: 0.10.2.0

> Group StopReplicaRequests for partitions on the same broker into one 
> StopReplicaRequest
> ---
>
> Key: KAFKA-4420
> URL: https://issues.apache.org/jira/browse/KAFKA-4420
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 0.10.2.0
>
>
> Currently if a broker of 1000 partitions want to do controlled shutdown, 
> controller will send 1000 StopReplicaRequest to that broker. This is 
> inefficient. Controller only need to send one StopReplicaRequest and include 
> those 1000 partitions in this request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4420) Group StopReplicaRequests for partitions on the same broker into one StopReplicaRequest

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4420:
---
Issue Type: Improvement  (was: Bug)

> Group StopReplicaRequests for partitions on the same broker into one 
> StopReplicaRequest
> ---
>
> Key: KAFKA-4420
> URL: https://issues.apache.org/jira/browse/KAFKA-4420
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 0.10.2.0
>
>
> Currently if a broker of 1000 partitions want to do controlled shutdown, 
> controller will send 1000 StopReplicaRequest to that broker. This is 
> inefficient. Controller only need to send one StopReplicaRequest and include 
> those 1000 partitions in this request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4420) Group StopReplicaRequests for partitions on the same broker into one StopReplicaRequest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678316#comment-15678316
 ] 

ASF GitHub Bot commented on KAFKA-4420:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2148


> Group StopReplicaRequests for partitions on the same broker into one 
> StopReplicaRequest
> ---
>
> Key: KAFKA-4420
> URL: https://issues.apache.org/jira/browse/KAFKA-4420
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>
> Currently if a broker of 1000 partitions want to do controlled shutdown, 
> controller will send 1000 StopReplicaRequest to that broker. This is 
> inefficient. Controller only need to send one StopReplicaRequest and include 
> those 1000 partitions in this request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2148: KAFKA-4420; Group StopReplicaRequests for partitio...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2148


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3994) Deadlock between consumer heartbeat expiration and offset commit.

[mjuarez (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678062#comment-15678062
 ] 

mjuarez commented on KAFKA-3994:


Turns out I was wrong.  The deadlock we saw in our kafka cluster was because a 
single broker had been patched incorrectly, every other broker experienced no 
issues at all for roughly a week after we patched them.  After performing 
multiple tests, I'd say we're 99% confident this patch will fix the deadlock 
issue we're seeing.

> Deadlock between consumer heartbeat expiration and offset commit.
> -
>
> Key: KAFKA-3994
> URL: https://issues.apache.org/jira/browse/KAFKA-3994
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Jiangjie Qin
>Assignee: Jason Gustafson
>Priority: Critical
> Fix For: 0.10.1.1
>
>
> I got the following stacktraces from ConsumerBounceTest
> {code}
> ...
> "Test worker" #12 prio=5 os_prio=0 tid=0x7fbb28b7f000 nid=0x427c runnable 
> [0x7fbb06445000]
>java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0003d48bcbc0> (a sun.nio.ch.Util$2)
> - locked <0x0003d48bcbb0> (a 
> java.util.Collections$UnmodifiableSet)
> - locked <0x0003d48bbd28> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.kafka.common.network.Selector.select(Selector.java:454)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:277)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:179)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:411)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1086)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1054)
> at 
> kafka.api.ConsumerBounceTest.consumeWithBrokerFailures(ConsumerBounceTest.scala:103)
> at 
> kafka.api.ConsumerBounceTest.testConsumptionWithBrokerFailures(ConsumerBounceTest.scala:70)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at 
> 

[jira] [Comment Edited] (KAFKA-4270) ClassCast for Agregation

[Damian Guy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677990#comment-15677990
 ] 

Damian Guy edited comment on KAFKA-4270 at 11/18/16 10:41 PM:
--

Hi Mykola,

I'm not entirely sure i follow what you are saying. I think that all you need 
to do is provide the serdes to the {{groupBy}} method, i.e.,
{{table.groupBy((key, value) -> value.thing, keySerde, valueSerde)}}

Thanks,
Damian


was (Author: damianguy):
Hi Mykola,

I'm not entirely sure i follow what you are saying. I think that all you need 
to do is provide the serdes to the `groupBy` method, i.e.,
`table.groupBy((key, value) -> value.thing, keySerde, valueSerde)`

Thanks,
Damian

> ClassCast for Agregation
> 
>
> Key: KAFKA-4270
> URL: https://issues.apache.org/jira/browse/KAFKA-4270
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Mykola Polonskyi
>Assignee: Damian Guy
>Priority: Critical
>  Labels: architecture
>
> With defined serdes for intermediate topic in aggregation catch the 
> ClassCastException: from custom class to the ByteArray.
> In debug I saw that defined serde isn't used for creation sinkNode (incide 
> `org.apache.kafka.streams.kstream.internals.KGroupedTableImpl#doAggregate`) 
> Instead defined serde inside aggregation call is used default Impl with empty 
> plugs instead of implementations 
> {code:koltin} 
> userTable.join(
> skicardsTable.groupBy { key, value -> 
> KeyValue(value.skicardInfo.ownerId, value.skicardInfo) }
> .aggregate(
> { mutableSetOf() }, 
> { ownerId, skicardInfo, accumulator -> 
> accumulator.put(skicardInfo) },
> { ownerId, skicardInfo, accumulator -> 
> accumulator }, 
> skicardByOwnerIdSerde,
> skicardByOwnerIdTopicName
> ),
> { userCreatedOrUpdated, skicardInfoSet -> 
> UserWithSkicardsCreatedOrUpdated(userCreatedOrUpdated.user, skicardInfoSet) }
> ).to(
> userWithSkicardsTable
> )
> {code}
> I think current behavior of `doAggregate` with serdes and/or stores setting 
> up should be changed because that is incorrect in release 0.10.0.1-cp1 to.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4270) ClassCast for Agregation

[Damian Guy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677990#comment-15677990
 ] 

Damian Guy commented on KAFKA-4270:
---

Hi Mykola,

I'm not entirely sure i follow what you are saying. I think that all you need 
to do is provide the serdes to the `groupBy` method, i.e.,
`table.groupBy((key, value) -> value.thing, keySerde, valueSerde)`

Thanks,
Damian

> ClassCast for Agregation
> 
>
> Key: KAFKA-4270
> URL: https://issues.apache.org/jira/browse/KAFKA-4270
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Mykola Polonskyi
>Assignee: Damian Guy
>Priority: Critical
>  Labels: architecture
>
> With defined serdes for intermediate topic in aggregation catch the 
> ClassCastException: from custom class to the ByteArray.
> In debug I saw that defined serde isn't used for creation sinkNode (incide 
> `org.apache.kafka.streams.kstream.internals.KGroupedTableImpl#doAggregate`) 
> Instead defined serde inside aggregation call is used default Impl with empty 
> plugs instead of implementations 
> {code:koltin} 
> userTable.join(
> skicardsTable.groupBy { key, value -> 
> KeyValue(value.skicardInfo.ownerId, value.skicardInfo) }
> .aggregate(
> { mutableSetOf() }, 
> { ownerId, skicardInfo, accumulator -> 
> accumulator.put(skicardInfo) },
> { ownerId, skicardInfo, accumulator -> 
> accumulator }, 
> skicardByOwnerIdSerde,
> skicardByOwnerIdTopicName
> ),
> { userCreatedOrUpdated, skicardInfoSet -> 
> UserWithSkicardsCreatedOrUpdated(userCreatedOrUpdated.user, skicardInfoSet) }
> ).to(
> userWithSkicardsTable
> )
> {code}
> I think current behavior of `doAggregate` with serdes and/or stores setting 
> up should be changed because that is incorrect in release 0.10.0.1-cp1 to.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4416) Add a '--group' option to the console consumer

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian updated KAFKA-4416:
---
Status: Patch Available  (was: Open)

> Add a '--group' option to the console consumer
> --
>
> Key: KAFKA-4416
> URL: https://issues.apache.org/jira/browse/KAFKA-4416
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>Priority: Minor
>
> Add a {{--group}} option to the console consumer to simplify associating 
> consumers to consumer groups. The command line option would overwrite any 
> {{group.id}} property that may be specified in the consumer config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Deprecating the old consumers in trunk

[Onur Karaman]

So my earlier stated suboptimal migration plans and Joel's idea all suffer
from either downtime or dual partition ownership and consumption.

But I think there's a bigger problem: they assume users are willing to do
the full migration immediately. I'm not convinced that this is realistic.
Some teams may be okay with this (and the earlier stated consequences of
the existing approaches), but others want to "canary" new code. That is,
they want to deploy a single instance of the new code to test the waters
while all the other instances run old code. It's not unreasonable for this
to span days. In this world, earlier alternatives would have the canary
under heavy load since it is the sole new consumer in the group and it is
guaranteed to own every partition the group is interested in. So the canary
is likely going to look unhealthy and the consumer can fall behind.

Here's a not-fully-thought-out idea:
Suppose we roll out a ZookeeperConsumerConnector that uses an embedded
KafkaConsumer to passively participate in kafka-based coordination while
still participating in zookeeper-based coordination. For now, the
ZookeeperConsumerConnectors just uses the partition assignment as decided
in zookeeper. Now suppose an outside KafkaConsumer comes up. Kafka-based
coordination allows arbitrary metadata to get broadcasted to the group.
Maybe we can somehow broadcast a flag saying a new consumer is running
during this migration. If the KafkaConsumers embedded in the
ZookeeperConsumerConnector see this flag, then they can notify the
ZookeeperConsumerConnector's fetchers to fetch the partitions defined by
the kafka-based coordination rebalance result. The
ZookeeperConsumerConnector's embedded KafkaConsumer's fetchers never get
used at any point in time.

The benefits of this approach would be:
1. no downtime
2. minimal window of dual partition ownership
3. even partition distribution upon canary arrival.
ZookeeperConsumerConnector instances can claim some partition ownership, so
the canaried KafkaConsumer doesn't get overwhelmed by all of the partitions.

On Thu, Nov 17, 2016 at 9:17 PM, Joel Koshy  wrote:

> Not sure it is worth doing, but a simple migration approach that avoids
> *service* downtime could be as follows:
>
>- Add a “migration mode” to the old consumer that disables its fetchers
>and disables offset commits. i.e., the consumers rebalance and own
>partitions but do basically nothing.
>- So assuming the old consumer is already committing offsets to Kafka,
>the process would be:
>- Bounce the consumer group (still on the old consumer) with:
>  - Migration mode on
>  - consumer.timeout.ms -1
>   - Bounce the consumer group to switch to the new consumer
>- i.e., effectively pause and resume the entire group without real
>downtime of the services.
>
>
>
> On Thu, Nov 17, 2016 at 7:30 PM, Ismael Juma  wrote:
>
> > Thanks James. I had read your post and was planning to find it in order
> to
> > share it here so you saved me some work. :)
> >
> > Ismael
> >
> > On Fri, Nov 18, 2016 at 3:21 AM, James Cheng 
> wrote:
> >
> > > Sorry to self-plug, but I wrote a blog post that talks about this, with
> > > respect to mirrormaker. I came to the same 3 solutions that Onur
> > described.
> > >
> > > https://logallthethings.com/2016/10/07/mirrormaker-
> > > gotchas-when-moving-from-the-old-consumer-to-the-new-consumer/ <
> > > https://logallthethings.com/2016/10/07/mirrormaker-
> > > gotchas-when-moving-from-the-old-consumer-to-the-new-consumer/>
> > >
> > > -James
> > >
> > > > On Nov 17, 2016, at 7:37 AM, Ismael Juma  wrote:
> > > >
> > > > Hi Onur,
> > > >
> > > > It is a good point that there is currently no out of the box solution
> > for
> > > > migrating from the old consumer to the new consumer where neither
> > > downtime
> > > > or duplicate consumption are acceptable. As I understand, this is
> > > important
> > > > for some of the usages at LinkedIn. Do you have any plans to tackle
> > this
> > > > issue?
> > > >
> > > > Jason, any thoughts on this?
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Oct 31, 2016 at 11:03 PM, Onur Karaman <
> > > > okara...@linkedin.com.invalid> wrote:
> > > >
> > > >> Does this make sense given that we still don't have a graceful
> > migration
> > > >> plan from the old to new consumer?
> > > >>
> > > >> Different suboptimal migration plans that I can think of are:
> > > >> 1. shutdown all the old consumers of a group first and start them
> back
> > > up
> > > >> with the new consumer, causing downtime.
> > > >> 2. have a mix of old and new consumers in the same group, causing
> > > duplicate
> > > >> partition ownership and consumption as each rebalance protocol
> ignores
> > > the
> > > >> other.
> > > >> 3. form a brand new group for the new consumers doing the same work
> as
> > > the
> > > >> old consumer group, still causing duplicate partition 

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

[Jun Rao]

Hi, Radai,

3. Having a gauge of MemoryAvailable is useful. One issue with that though
is that if one only collects the metrics say every minute, one doesn't know
what has happened in between. We could additionally track the fraction of
the time when requested memory can't be served. Every time a request can't
be honored, we mark the starting time in memory pool. Every time a request
is honored, we end the time. We can then expose that accumulated fraction
of time as a Rate (similar to RequestHandlerAvgIdlePercent
in KafkaRequestHandlerPool). This will be a value between 0 and 1. The
higher the value, the more memory pressure.

Thanks,

Jun

On Fri, Nov 18, 2016 at 8:35 AM, radai  wrote:

> Hi Jun,
>
> 3. will (also :-) ) do. do you have ideas for appropriate names/metrics?
> I'm thinking along the lines of "MemoryAvailable" (current snapshot value
> from pool) and "Throttles" (some moving-avg of how often does throttling
> due to no mem kicks in). maybe also "BuffersOutstanding" ?
>
> On Thu, Nov 17, 2016 at 7:01 PM, Jun Rao  wrote:
>
> > Hi, Radai,
> >
> > 2. Yes, on the server side, the timeout is hardcoded at 300ms. That's not
> > too bad. We can just leave it as it is.
> >
> > 3. Another thing. Do we plan to expose some JMX metrics so that we can
> > monitor if there is any memory pressure in the pool?
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Nov 17, 2016 at 8:57 AM, radai 
> wrote:
> >
> > > Hi Jun,
> > >
> > > 1. will do.
> > >
> > > 2. true. for several reasons:
> > >2.1. which selector? there's a single pool but 16 selectors
> (linkedin
> > > typical, num.network.threads defaults to 3)
> > >2.2. even if i could figure out which selector (all?) the better
> thing
> > > to do would be resume reading not when any memory becomes available
> > > (because worst case its not enough for anything) but when some "low
> > > watermark" of available memory is hit - so mute when @100% mem, unmute
> > when
> > > back down to 90%?
> > >2.3. on the broker side (which is the current concern for my patch)
> > this
> > > max wait time is a hardcoded 300 ms (SocketServer.Processor.poll()),
> > which
> > > i think is acceptable and definitely not arbitrary or configurable.
> > >
> > >if you still think this needs to be addressed (and you are right
> that
> > in
> > > the general case the timeout param could be arbitrary) i can implement
> > the
> > > watermark approach + pool.waitForLowWatermark(timeout) or something,
> and
> > > make Selector.poll() wait for low watermark at the end of poll() if no
> > work
> > > has been done (so as not to wait on memory needlessly for requests that
> > > done require it, as rajini suggested earlier)
> > >
> > > On Wed, Nov 16, 2016 at 9:04 AM, Jun Rao  wrote:
> > >
> > > > Hi, Radai,
> > > >
> > > > Thanks for the updated proposal. +1 overall. A couple of comments
> > below.
> > > >
> > > > 1. Our current convention is to avoid using getters. Could you change
> > > > getSize and getAvailableMemory accordingly? Also, size is bit
> > ambiguous,
> > > > could we use sth like capacity?
> > > >
> > > > 2. This is more on the implementation details. I didn't see any code
> to
> > > > wake up the selector when memory is released from the pool. For
> > example,
> > > > suppose that all socket keys are muted since the pool is full. The
> > > > selector.poll() call will wait for the timeout, which could be
> > > arbitrarily
> > > > long. Now, if some memory is released, it seems that we should wake
> up
> > > the
> > > > selector early instead of waiting for the timeout.
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Mon, Nov 14, 2016 at 11:41 AM, Rajini Sivaram <
> > > > rajinisiva...@googlemail.com> wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Thank you for the KIP, Radai.
> > > > >
> > > > > On Mon, Nov 14, 2016 at 6:07 PM, Mickael Maison <
> > > > mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > +1. We've also been hit by OOMs on the broker because we were not
> > > able
> > > > > > to properly bound its memory usage.
> > > > > >
> > > > > > On Mon, Nov 14, 2016 at 5:56 PM, radai <
> radai.rosenbl...@gmail.com
> > >
> > > > > wrote:
> > > > > > > @rajini - fixed the hasBytesBuffered() method. also updated
> > poll()
> > > so
> > > > > > that
> > > > > > > no latency is added for picking up data stuck in ssl buffers
> > > (timeout
> > > > > is
> > > > > > > set to 0, just like with immediately connected keys and staged
> > > > > receives).
> > > > > > > thank you for pointing these out.
> > > > > > > added ssl (re) testing to the KIP testing plan.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Nov 14, 2016 at 7:24 AM, Rajini Sivaram <
> > > > > > > rajinisiva...@googlemail.com> wrote:
> > > > > > >
> > > > > > >> Open point 1. I would just retain the current long value that
> > > > > specifies
> > > > > > >> queued.max.bytes as 

[GitHub] kafka pull request #2147: revert corrupted commit 10cfc1628df024f7596d3af5c1...

[mjsax]

Github user mjsax closed the pull request at:

https://github.com/apache/kafka/pull/2147


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-82 - Add Record Headers

[Ignacio Solis]

Summary:

3) Yes - Header value as byte[]

4a) Int,Int - No
4b) Int - Yes
4c) String - Reluctant maybe

5) I believe the header system should take a single int.  I think 32bits is
a good size, if you want to interpret this as to 16bit numbers in the layer
above go right ahead.  If somebody wants to argue for 16 bits or 64 bits of
header key space I would listen.


Discussion:
Dividing the key space into sub_key_1 and sub_key_2 makes no sense to me at
this layer.  Are we going to start providing APIs to get all the
sub_key_1s? or all the sub_key_2s?  If there is no distinguishing functions
that are applied to each one then they should be a single value.  At this
layer all we're doing is equality.
If the above layer wants to interpret this as 2, 3 or more values that's a
different question.  I personally think it's all one keyspace that is
getting assigned using some structure, but if you want to sub-assign parts
of it then that's fine.

The same discussion applies to strings.  If somebody argued for strings,
would we be arguing to divide the strings with dots ('.') as a requirement?
Would we want them to give us the different name segments separately?
Would we be performing any actions on this key other than matching?

Nacho



On Fri, Nov 18, 2016 at 9:30 AM, Michael Pearce 
wrote:

> #jay #jun any concerns on 1 and 2 still?
>
> @all
> To get this moving along a bit more I'd also like to ask to get clarity on
> the below last points:
>
> 3) I believe we're all roughly happy with the header value being a byte[]?
>
> 4) I believe consensus has been for an namespace based int approach
> {int,int} for the key. Any objections if this is what we go with?
>
> 5) as we have if assumption in (4)  is correct, {int,int} keys.
> Should both int's be int16 or int32?
> I'm for them being int16(2 bytes) as combined is space of 4bytes as per
> original and gives plenty of combinations for the foreseeable, and keeps
> the overhead small.
>
> Do we see any benefit in another kip call to discuss these at all?
>
> Cheers
> Mike
> 
> From: K Burstev 
> Sent: Friday, November 18, 2016 7:07:07 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> For what it is worth also i agree. As a user:
>
>  1) Yes - Headers are worthwhile
>  2) Yes - Headers should be a top level option
>
> 14.11.2016, 21:15, "Ignacio Solis" :
> > 1) Yes - Headers are worthwhile
> > 2) Yes - Headers should be a top level option
> >
> > On Mon, Nov 14, 2016 at 9:16 AM, Michael Pearce 
> > wrote:
> >
> >>  Hi Roger,
> >>
> >>  The kip details/examples the original proposal for key spacing , not
> the
> >>  new mentioned as per discussion namespace idea.
> >>
> >>  We will need to update the kip, when we get agreement this is a better
> >>  approach (which seems to be the case if I have understood the general
> >>  feeling in the conversation)
> >>
> >>  Re the variable ints, at very early stage we did think about this. I
> think
> >>  the added complexity for the saving isn't worth it. I'd rather go
> with, if
> >>  we want to reduce overheads and size int16 (2bytes) keys as it keeps it
> >>  simple.
> >>
> >>  On the note of no headers, there is as per the kip as we use an
> attribute
> >>  bit to denote if headers are present or not as such provides a zero
> >>  overhead currently if headers are not used.
> >>
> >>  I think as radai mentions would be good first if we can get clarity if
> do
> >>  we now have general consensus that (1) headers are worthwhile and
> useful,
> >>  and (2) we want it as a top level entity.
> >>
> >>  Just to state the obvious i believe (1) headers are worthwhile and (2)
> >>  agree as a top level entity.
> >>
> >>  Cheers
> >>  Mike
> >>  
> >>  From: Roger Hoover 
> >>  Sent: Wednesday, November 9, 2016 9:10:47 PM
> >>  To: dev@kafka.apache.org
> >>  Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >>
> >>  Sorry for going a little in the weeds but thanks for the replies
> regarding
> >>  varint.
> >>
> >>  Agreed that a prefix and {int, int} can be the same. It doesn't look
> like
> >>  that's what the KIP is saying the "Open" section. The example shows
> >>  211
> >>  for New Relic and 210002 for App Dynamics implying that the New Relic
> >>  organization will have only a single header id to work with. Or is
> 211
> >>  a prefix? The main point of a namespace or prefix is to reduce the
> >>  overhead of config mapping or registration depending on how
> >>  namespaces/prefixes are managed.
> >>
> >>  Would love to hear more feedback on the higher-level questions
> though...
> >>
> >>  Cheers,
> >>
> >>  Roger
> >>
> >>  On Wed, Nov 9, 2016 at 11:38 AM, radai 
> wrote:
> >>
> >>  > I think this discussion is getting a bit into the weeds on technical
> >>  > 

[DISCUSS] KIP-93: Improve invalid timestamp handling in Kafka Streams

[Matthias J. Sax]

Hi all,

I want to start a discussion about KIP-93:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-93%3A+Improve+invalid+timestamp+handling+in+Kafka+Streams

Looking forward to your feedback.


-Matthias



signature.asc
Description: OpenPGP digital signature

[jira] [Commented] (KAFKA-4416) Add a '--group' option to the console consumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677642#comment-15677642
 ] 

ASF GitHub Bot commented on KAFKA-4416:
---

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/2150

KAFKA-4416: Add a `--group` option to console consumer

The value provided for this option overwrites any `group.id` value 
indirectly provided via `consumer-property` or `consumer.config` options.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-4416

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2150.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2150


commit 87a9cf74765ea106d9708745f62a935fd917a866
Author: Vahid Hashemian 
Date:   2016-11-18T19:29:25Z

KAFKA-4416: Add a `--group` option to console consumer

The value provided for this option overwrites any `group.id` value 
indirectly provided via `consumer-property` or `consumer.config` options.




> Add a '--group' option to the console consumer
> --
>
> Key: KAFKA-4416
> URL: https://issues.apache.org/jira/browse/KAFKA-4416
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>Priority: Minor
>
> Add a {{--group}} option to the console consumer to simplify associating 
> consumers to consumer groups. The command line option would overwrite any 
> {{group.id}} property that may be specified in the consumer config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2150: KAFKA-4416: Add a `--group` option to console cons...

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/2150

KAFKA-4416: Add a `--group` option to console consumer

The value provided for this option overwrites any `group.id` value 
indirectly provided via `consumer-property` or `consumer.config` options.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-4416

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2150.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2150


commit 87a9cf74765ea106d9708745f62a935fd917a866
Author: Vahid Hashemian 
Date:   2016-11-18T19:29:25Z

KAFKA-4416: Add a `--group` option to console consumer

The value provided for this option overwrites any `group.id` value 
indirectly provided via `consumer-property` or `consumer.config` options.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (KAFKA-4270) ClassCast for Agregation

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy reassigned KAFKA-4270:
-

Assignee: Damian Guy

> ClassCast for Agregation
> 
>
> Key: KAFKA-4270
> URL: https://issues.apache.org/jira/browse/KAFKA-4270
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Mykola Polonskyi
>Assignee: Damian Guy
>Priority: Critical
>  Labels: architecture
>
> With defined serdes for intermediate topic in aggregation catch the 
> ClassCastException: from custom class to the ByteArray.
> In debug I saw that defined serde isn't used for creation sinkNode (incide 
> `org.apache.kafka.streams.kstream.internals.KGroupedTableImpl#doAggregate`) 
> Instead defined serde inside aggregation call is used default Impl with empty 
> plugs instead of implementations 
> {code:koltin} 
> userTable.join(
> skicardsTable.groupBy { key, value -> 
> KeyValue(value.skicardInfo.ownerId, value.skicardInfo) }
> .aggregate(
> { mutableSetOf() }, 
> { ownerId, skicardInfo, accumulator -> 
> accumulator.put(skicardInfo) },
> { ownerId, skicardInfo, accumulator -> 
> accumulator }, 
> skicardByOwnerIdSerde,
> skicardByOwnerIdTopicName
> ),
> { userCreatedOrUpdated, skicardInfoSet -> 
> UserWithSkicardsCreatedOrUpdated(userCreatedOrUpdated.user, skicardInfoSet) }
> ).to(
> userWithSkicardsTable
> )
> {code}
> I think current behavior of `doAggregate` with serdes and/or stores setting 
> up should be changed because that is incorrect in release 0.10.0.1-cp1 to.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4373) Kafka Consumer API jumping offsets

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677456#comment-15677456
 ] 

Guozhang Wang commented on KAFKA-4373:
--

This may be resulted from the intermediate topic that is written by topology 1 
and read by topology 2 was truncated due to log retention policy. To validate, 
could you check the streams client log and see if there are entries mentioning 
"... is out of range for partition ..."

> Kafka Consumer API jumping offsets
> --
>
> Key: KAFKA-4373
> URL: https://issues.apache.org/jira/browse/KAFKA-4373
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Srinivasan Venkatraman
>
> Hi,
> I am using Kafka Version 0.10.0.1 and java consumer API to consume messages 
> from a topic. We are using a single node kafka and zookeeper. It is sometimes 
> observed that the consumer is losing a bulk of message. We are unable to find 
> the exact reason to replicate the issue.
> The scenario is:
> Consumer polls the topic.
> Fetches the messages and gives it to a thread pool to handle the message.
> Waits for the threads to return and then commits the offsets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-3701) Expose KafkaStreams metrics in public API

[Mitch Seymour (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mitch Seymour reassigned KAFKA-3701:


Assignee: Mitch Seymour

> Expose KafkaStreams metrics in public API
> -
>
> Key: KAFKA-3701
> URL: https://issues.apache.org/jira/browse/KAFKA-3701
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Jeff Klukas
>Assignee: Mitch Seymour
>Priority: Minor
>  Labels: user-experience
> Fix For: 0.10.2.0
>
>
> The Kafka clients expose their metrics registries through a `metrics` method 
> presenting an unmodifiable collection, but `KafkaStreams` does not expose its 
> registry. Currently, applications can access a StreamsMetrics instance via 
> the ProcessorContext within a Processor, but this limits flexibility.
> Having read-only access to a KafkaStreams.metrics() method would allow a 
> developer to define a health check for their application based on the metrics 
> that KafkaStreams is collecting. Or a developer might want to define a metric 
> in some other framework based on KafkaStreams' metrics.
> I am imagining that an application would build and register 
> KafkaStreams-based health checks after building a KafkaStreams instance but 
> before calling the start() method. Are metrics added to the registry at the 
> time a KafkaStreams instance is constructed, or only after calling the 
> start() method? If metrics are registered only after application startup, 
> then this approach may not be sufficient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-82 - Add Record Headers

[Michael Pearce]

#jay #jun any concerns on 1 and 2 still?

@all
To get this moving along a bit more I'd also like to ask to get clarity on the 
below last points:

3) I believe we're all roughly happy with the header value being a byte[]?

4) I believe consensus has been for an namespace based int approach {int,int} 
for the key. Any objections if this is what we go with?

5) as we have if assumption in (4)  is correct, {int,int} keys.
Should both int's be int16 or int32?
I'm for them being int16(2 bytes) as combined is space of 4bytes as per 
original and gives plenty of combinations for the foreseeable, and keeps the 
overhead small.

Do we see any benefit in another kip call to discuss these at all?

Cheers
Mike

From: K Burstev 
Sent: Friday, November 18, 2016 7:07:07 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers

For what it is worth also i agree. As a user:

 1) Yes - Headers are worthwhile
 2) Yes - Headers should be a top level option

14.11.2016, 21:15, "Ignacio Solis" :
> 1) Yes - Headers are worthwhile
> 2) Yes - Headers should be a top level option
>
> On Mon, Nov 14, 2016 at 9:16 AM, Michael Pearce 
> wrote:
>
>>  Hi Roger,
>>
>>  The kip details/examples the original proposal for key spacing , not the
>>  new mentioned as per discussion namespace idea.
>>
>>  We will need to update the kip, when we get agreement this is a better
>>  approach (which seems to be the case if I have understood the general
>>  feeling in the conversation)
>>
>>  Re the variable ints, at very early stage we did think about this. I think
>>  the added complexity for the saving isn't worth it. I'd rather go with, if
>>  we want to reduce overheads and size int16 (2bytes) keys as it keeps it
>>  simple.
>>
>>  On the note of no headers, there is as per the kip as we use an attribute
>>  bit to denote if headers are present or not as such provides a zero
>>  overhead currently if headers are not used.
>>
>>  I think as radai mentions would be good first if we can get clarity if do
>>  we now have general consensus that (1) headers are worthwhile and useful,
>>  and (2) we want it as a top level entity.
>>
>>  Just to state the obvious i believe (1) headers are worthwhile and (2)
>>  agree as a top level entity.
>>
>>  Cheers
>>  Mike
>>  
>>  From: Roger Hoover 
>>  Sent: Wednesday, November 9, 2016 9:10:47 PM
>>  To: dev@kafka.apache.org
>>  Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>>
>>  Sorry for going a little in the weeds but thanks for the replies regarding
>>  varint.
>>
>>  Agreed that a prefix and {int, int} can be the same. It doesn't look like
>>  that's what the KIP is saying the "Open" section. The example shows
>>  211
>>  for New Relic and 210002 for App Dynamics implying that the New Relic
>>  organization will have only a single header id to work with. Or is 211
>>  a prefix? The main point of a namespace or prefix is to reduce the
>>  overhead of config mapping or registration depending on how
>>  namespaces/prefixes are managed.
>>
>>  Would love to hear more feedback on the higher-level questions though...
>>
>>  Cheers,
>>
>>  Roger
>>
>>  On Wed, Nov 9, 2016 at 11:38 AM, radai  wrote:
>>
>>  > I think this discussion is getting a bit into the weeds on technical
>>  > implementation details.
>>  > I'd liek to step back a minute and try and establish where we are in the
>>  > larger picture:
>>  >
>>  > (re-wording nacho's last paragraph)
>>  > 1. are we all in agreement that headers are a worthwhile and useful
>>  > addition to have? this was contested early on
>>  > 2. are we all in agreement on headers as top level entity vs headers
>>  > squirreled-away in V?
>>  >
>>  > if there are still concerns around these #2 points (#jay? #jun?)?
>>  >
>>  > (and now back to our normal programming ...)
>>  >
>>  > varints are nice. having said that, its adding complexity (see
>>  > https://github.com/addthis/stream-lib/blob/master/src/
>>  > main/java/com/clearspring/analytics/util/Varint.java
>>  > as 1st google result) and would require anyone writing other clients (C?
>>  > Python? Go? Bash? ;-) ) to get/implement the same, and for relatively
>>  > little gain (int vs string is order of magnitude, this isnt).
>>  >
>>  > int namespacing vs {int, int} namespacing are basically the same thing -
>>  > youre just namespacing an int64 and giving people while 2^32 ranges at a
>>  > time. the part i like about this is letting people have a large swath of
>>  > numbers with one registration so they dont have to come back for every
>>  > single plugin/header they want to "reserve".
>>  >
>>  >
>>  > On Wed, Nov 9, 2016 at 11:01 AM, Roger Hoover 
>>  > wrote:
>>  >
>>  > > Since some of the debate has been about overhead + performance, I'm
>>  > > wondering if we 

Build failed in Jenkins: kafka-trunk-jdk8 #1051

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-4377; remove deprecated scala.collection.JavaConversions calls

--
[...truncated 14207 lines...]
org.apache.kafka.streams.kstream.JoinWindowsTest > beforeOverUpper STARTED

org.apache.kafka.streams.kstream.JoinWindowsTest > beforeOverUpper PASSED

org.apache.kafka.streams.kstream.JoinWindowsTest > 
shouldHaveSaneEqualsAndHashCode STARTED

org.apache.kafka.streams.kstream.JoinWindowsTest > 
shouldHaveSaneEqualsAndHashCode PASSED

org.apache.kafka.streams.kstream.JoinWindowsTest > validWindows STARTED

org.apache.kafka.streams.kstream.JoinWindowsTest > validWindows PASSED

org.apache.kafka.streams.kstream.JoinWindowsTest > 
timeDifferenceMustNotBeNegative STARTED

org.apache.kafka.streams.kstream.JoinWindowsTest > 
timeDifferenceMustNotBeNegative PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldHaveCorrectSourceTopicsForTableFromMergedStream STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldHaveCorrectSourceTopicsForTableFromMergedStream PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldHaveCorrectSourceTopicsForTableFromMergedStreamWithProcessors STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldHaveCorrectSourceTopicsForTableFromMergedStreamWithProcessors PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testMerge STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testMerge PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testFrom STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testFrom PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldThrowExceptionWhenTopicNamesAreNull STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldThrowExceptionWhenTopicNamesAreNull PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldThrowExceptionWhenNoTopicPresent STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldThrowExceptionWhenNoTopicPresent PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testNewName STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testNewName PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
shouldHaveSaneEqualsAndHashCode STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
shouldHaveSaneEqualsAndHashCode PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeZero 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeZero 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > advanceIntervalMustNotBeZero 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > advanceIntervalMustNotBeZero 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeNegative 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeNegative 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeNegative STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeNegative PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeLargerThanWindowSize STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeLargerThanWindowSize PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForTumblingWindows 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForTumblingWindows 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForHoppingWindows 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForHoppingWindows 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
windowsForBarelyOverlappingHoppingWindows STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
windowsForBarelyOverlappingHoppingWindows PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
STARTED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfValueSerdeConfigFails STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfValueSerdeConfigFails PASSED

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

[Mayuresh Gharat]

Hi Michael,

That whilst sending tombstone and non null value, the consumer can expect
only to receive the non-null message only in step (3) is this correct?
---> I do agree with you here.

Becket, Ismael : can you guys review the migration plan listed above using
magic byte?

Thanks,

Mayuresh

On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce 
wrote:

> Many thanks for this Mayuresh. I don't have any objections.
>
> I assume we should state:
>
> That whilst sending tombstone and non null value, the consumer can expect
> only to receive the non-null message only in step (3) is this correct?
>
> Cheers
> Mike
>
>
>
> Sent using OWA for iPhone
> 
> From: Mayuresh Gharat 
> Sent: Thursday, November 17, 2016 5:18:41 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
>
> Hi Ismael,
>
> Thanks for the explanation.
> Specially I like this part where in you mentioned we can get rid of the
> older null value support for log compaction later on, here :
> We can't change semantics of the message format without having a long
> transition period. And we can't rely
> on people reading documentation or acting on a warning for something so
> fundamental. As such, my take is that we need to bump the magic byte. The
> good news is
> that we don't have to support all versions forever. We have said that we
> will support direct upgrades for 2 years. That means that message format
> version n could, in theory, be removed 2 years after the it's introduced.
>
> Just a heads up, I would like to mention that even without bumping magic
> byte, we will *NOT* loose zero copy as in the client(x+1) in my explanation
> above will convert internally a null value to have a tombstone bit set and
> a tombstone bit set to have a null value automatically internally and by
> the time we move to version (x+2), the clients would have upgraded.
> Obviously if we support a request from consumer(x), we will loose zero copy
> but that is the same case with magic byte.
>
> But if magic byte bump makes life easier for transition for the above
> reasons that you explained, I am OK with it since we are going to meet the
> end goal down the road :)
>
> On a side note can we update the doc here on magic byte to say that "*it
> should be bumped whenever the message format is changed or the
> interpretation of message format (usage of the reserved bits as well) is
> changed*".
>
>
> Hi Michael,
>
> Here is the update plan that we discussed offline yesterday :
>
> Currently the magic-byte which corresponds to the "message.format.version"
> is set to 1.
>
> 1) On broker it will be set to 1 initially.
>
> 2) When a producer client sends a message with magic-byte = 2, since the
> broker is on magic-byte = 1, we will down convert it, which means if the
> tombstone bit is set, the value will be set to null. A consumer
> understanding magic-byte = 1, will still work with this. A consumer working
> with magic-byte =2 will also be able to understand this, since it
> understands the tombstone.
> Now there is still the question of supporting a non-tombstone and null
> value from producer client with magic-byte = 2.* (I am not sure if we
> should support this. Ismael/Becket can comment here)*
>
> 3) When almost all the clients have upgraded, the message.format.version on
> the broker can be changed to 2, where in the down conversion in the above
> step will not happen. If at this point we get a consumer request from a
> older consumer, we might have to down convert where in we loose zero copy,
> but these cases should be rare.
>
> Becket can you review this plan and add more details if I have
> missed/wronged something, before we put it on KIP.
>
> Thanks,
>
> Mayuresh
>
> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce 
> wrote:
>
> > Thanks guys, for discussing this offline and getting some consensus.
> >
> > So its clear for myself and others what is proposed now (i think i
> > understand, but want to make sure)
> >
> > Could i ask either directly update the kip to detail the migration
> > strategy, or (re-)state your offline discussed and agreed migration
> > strategy based on a magic byte is in this thread.
> >
> >
> > The main original driver for the KIP was to support compaction where
> value
> > isn't null, based off the discussions on KIP-82 thread.
> >
> > We should be able to support non-tombstone + null value by the completion
> > of the KIP, as we noted when discussing this kip, having logic based on a
> > null value isn't very clean and also separates the concerns.
> >
> > As discussed already though we can split this into KIP-87a and KIP-87b
> >
> > Where we look to deliver KIP-87a on a compacted topic (to address the
> > immediate issues)
> > * tombstone + null value
> > * tombstone + non-null value
> > * non-tombstone + non-null value
> >
> > Then we can discuss once KIP-87a is completed options 

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

[Michael Pearce]

Many thanks for this Mayuresh. I don't have any objections.

I assume we should state:

That whilst sending tombstone and non null value, the consumer can expect only 
to receive the non-null message only in step (3) is this correct?

Cheers
Mike



Sent using OWA for iPhone

From: Mayuresh Gharat 
Sent: Thursday, November 17, 2016 5:18:41 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Hi Ismael,

Thanks for the explanation.
Specially I like this part where in you mentioned we can get rid of the
older null value support for log compaction later on, here :
We can't change semantics of the message format without having a long
transition period. And we can't rely
on people reading documentation or acting on a warning for something so
fundamental. As such, my take is that we need to bump the magic byte. The
good news is
that we don't have to support all versions forever. We have said that we
will support direct upgrades for 2 years. That means that message format
version n could, in theory, be removed 2 years after the it's introduced.

Just a heads up, I would like to mention that even without bumping magic
byte, we will *NOT* loose zero copy as in the client(x+1) in my explanation
above will convert internally a null value to have a tombstone bit set and
a tombstone bit set to have a null value automatically internally and by
the time we move to version (x+2), the clients would have upgraded.
Obviously if we support a request from consumer(x), we will loose zero copy
but that is the same case with magic byte.

But if magic byte bump makes life easier for transition for the above
reasons that you explained, I am OK with it since we are going to meet the
end goal down the road :)

On a side note can we update the doc here on magic byte to say that "*it
should be bumped whenever the message format is changed or the
interpretation of message format (usage of the reserved bits as well) is
changed*".


Hi Michael,

Here is the update plan that we discussed offline yesterday :

Currently the magic-byte which corresponds to the "message.format.version"
is set to 1.

1) On broker it will be set to 1 initially.

2) When a producer client sends a message with magic-byte = 2, since the
broker is on magic-byte = 1, we will down convert it, which means if the
tombstone bit is set, the value will be set to null. A consumer
understanding magic-byte = 1, will still work with this. A consumer working
with magic-byte =2 will also be able to understand this, since it
understands the tombstone.
Now there is still the question of supporting a non-tombstone and null
value from producer client with magic-byte = 2.* (I am not sure if we
should support this. Ismael/Becket can comment here)*

3) When almost all the clients have upgraded, the message.format.version on
the broker can be changed to 2, where in the down conversion in the above
step will not happen. If at this point we get a consumer request from a
older consumer, we might have to down convert where in we loose zero copy,
but these cases should be rare.

Becket can you review this plan and add more details if I have
missed/wronged something, before we put it on KIP.

Thanks,

Mayuresh

On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce 
wrote:

> Thanks guys, for discussing this offline and getting some consensus.
>
> So its clear for myself and others what is proposed now (i think i
> understand, but want to make sure)
>
> Could i ask either directly update the kip to detail the migration
> strategy, or (re-)state your offline discussed and agreed migration
> strategy based on a magic byte is in this thread.
>
>
> The main original driver for the KIP was to support compaction where value
> isn't null, based off the discussions on KIP-82 thread.
>
> We should be able to support non-tombstone + null value by the completion
> of the KIP, as we noted when discussing this kip, having logic based on a
> null value isn't very clean and also separates the concerns.
>
> As discussed already though we can split this into KIP-87a and KIP-87b
>
> Where we look to deliver KIP-87a on a compacted topic (to address the
> immediate issues)
> * tombstone + null value
> * tombstone + non-null value
> * non-tombstone + non-null value
>
> Then we can discuss once KIP-87a is completed options later and how we
> support the second part KIP-87b to deliver:
> * non-tombstone + null value
>
> Cheers
> Mike
>
>
>
> 
> From: Becket Qin 
> Sent: Thursday, November 17, 2016 1:43 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
>
> Renu, Mayuresh and I had an offline discussion, and following is a brief
> summary.
>
> 1. We agreed that not bumping up magic value may result in losing zero copy
> during migration.
> 2. Given that bumping up magic value is almost free 

[jira] [Comment Edited] (KAFKA-4373) Kafka Consumer API jumping offsets

[Conor Hughes (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677107#comment-15677107
 ] 

Conor Hughes edited comment on KAFKA-4373 at 11/18/16 4:42 PM:
---

I have experienced a similar issue using Kafka 0.10.1.0 and Kafka Streams 
0.10.1.0.

I have 2 topologies running:

Topology number 1: Reads messages from Kafka topic, processes them and outputs 
them to another Kafka topic.
Topology number 2: Reads messages from topology number 1's output topic and 
inserts them into a database.

I am tracking the number of messages topology number 2 has read in both inside 
and after the consumer's deserialiser as well as the topology's offsets in 
Kafka.
These counts will incrementally match for a number of minutes then suddenly the 
offsets will jump by up to 40,000 and but the topology's counts remain 
incrementing normally as if this jump never happen so messages are being lost.


was (Author: thatguyhughesy):
I have experienced a similar issue using Kafka 0.10.1.0 and Kafka Stream 
0.10.1.0.

I have 2 topologies running:

Topology number 1: Reads messages from Kafka topic, processes them and outputs 
them to another Kafka topic.
Topology number 2: Reads messages from topology number 1's output topic and 
inserts them into a database.

I am tracking the number of messages topology number 2 has read in both inside 
and after the consumer's deserialiser as well as the topology's offsets in 
Kafka.
These counts will incrementally match for a number of minutes then suddenly the 
offsets will jump by up to 40,000 and but the topology's counts remain 
incrementing normally as if this jump never happen so messages are being lost.

> Kafka Consumer API jumping offsets
> --
>
> Key: KAFKA-4373
> URL: https://issues.apache.org/jira/browse/KAFKA-4373
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Srinivasan Venkatraman
>
> Hi,
> I am using Kafka Version 0.10.0.1 and java consumer API to consume messages 
> from a topic. We are using a single node kafka and zookeeper. It is sometimes 
> observed that the consumer is losing a bulk of message. We are unable to find 
> the exact reason to replicate the issue.
> The scenario is:
> Consumer polls the topic.
> Fetches the messages and gives it to a thread pool to handle the message.
> Waits for the threads to return and then commits the offsets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4373) Kafka Consumer API jumping offsets

[Conor Hughes (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677107#comment-15677107
 ] 

Conor Hughes commented on KAFKA-4373:
-

I have experienced a similar issue using Kafka 0.10.1.0 and Kafka Stream 
0.10.1.0.

I have 2 topologies running:

Topology number 1: Reads messages from Kafka topic, processes them and outputs 
them to another Kafka topic.
Topology number 2: Reads messages from topology number 1's output topic and 
inserts them into a database.

I am tracking the number of messages topology number 2 has read in both inside 
and after the consumer's deserialiser as well as the topology's offsets in 
Kafka.
These counts will incrementally match for a number of minutes then suddenly the 
offsets will jump by up to 40,000 and but the topology's counts remain 
incrementing normally as if this jump never happen so messages are being lost.

> Kafka Consumer API jumping offsets
> --
>
> Key: KAFKA-4373
> URL: https://issues.apache.org/jira/browse/KAFKA-4373
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Srinivasan Venkatraman
>
> Hi,
> I am using Kafka Version 0.10.0.1 and java consumer API to consume messages 
> from a topic. We are using a single node kafka and zookeeper. It is sometimes 
> observed that the consumer is losing a bulk of message. We are unable to find 
> the exact reason to replicate the issue.
> The scenario is:
> Consumer polls the topic.
> Fetches the messages and gives it to a thread pool to handle the message.
> Waits for the threads to return and then commits the offsets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

[radai]

Hi Jun,

3. will (also :-) ) do. do you have ideas for appropriate names/metrics?
I'm thinking along the lines of "MemoryAvailable" (current snapshot value
from pool) and "Throttles" (some moving-avg of how often does throttling
due to no mem kicks in). maybe also "BuffersOutstanding" ?

On Thu, Nov 17, 2016 at 7:01 PM, Jun Rao  wrote:

> Hi, Radai,
>
> 2. Yes, on the server side, the timeout is hardcoded at 300ms. That's not
> too bad. We can just leave it as it is.
>
> 3. Another thing. Do we plan to expose some JMX metrics so that we can
> monitor if there is any memory pressure in the pool?
>
> Thanks,
>
> Jun
>
> On Thu, Nov 17, 2016 at 8:57 AM, radai  wrote:
>
> > Hi Jun,
> >
> > 1. will do.
> >
> > 2. true. for several reasons:
> >2.1. which selector? there's a single pool but 16 selectors (linkedin
> > typical, num.network.threads defaults to 3)
> >2.2. even if i could figure out which selector (all?) the better thing
> > to do would be resume reading not when any memory becomes available
> > (because worst case its not enough for anything) but when some "low
> > watermark" of available memory is hit - so mute when @100% mem, unmute
> when
> > back down to 90%?
> >2.3. on the broker side (which is the current concern for my patch)
> this
> > max wait time is a hardcoded 300 ms (SocketServer.Processor.poll()),
> which
> > i think is acceptable and definitely not arbitrary or configurable.
> >
> >if you still think this needs to be addressed (and you are right that
> in
> > the general case the timeout param could be arbitrary) i can implement
> the
> > watermark approach + pool.waitForLowWatermark(timeout) or something, and
> > make Selector.poll() wait for low watermark at the end of poll() if no
> work
> > has been done (so as not to wait on memory needlessly for requests that
> > done require it, as rajini suggested earlier)
> >
> > On Wed, Nov 16, 2016 at 9:04 AM, Jun Rao  wrote:
> >
> > > Hi, Radai,
> > >
> > > Thanks for the updated proposal. +1 overall. A couple of comments
> below.
> > >
> > > 1. Our current convention is to avoid using getters. Could you change
> > > getSize and getAvailableMemory accordingly? Also, size is bit
> ambiguous,
> > > could we use sth like capacity?
> > >
> > > 2. This is more on the implementation details. I didn't see any code to
> > > wake up the selector when memory is released from the pool. For
> example,
> > > suppose that all socket keys are muted since the pool is full. The
> > > selector.poll() call will wait for the timeout, which could be
> > arbitrarily
> > > long. Now, if some memory is released, it seems that we should wake up
> > the
> > > selector early instead of waiting for the timeout.
> > >
> > > Jun
> > >
> > >
> > > On Mon, Nov 14, 2016 at 11:41 AM, Rajini Sivaram <
> > > rajinisiva...@googlemail.com> wrote:
> > >
> > > > +1
> > > >
> > > > Thank you for the KIP, Radai.
> > > >
> > > > On Mon, Nov 14, 2016 at 6:07 PM, Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > > > +1. We've also been hit by OOMs on the broker because we were not
> > able
> > > > > to properly bound its memory usage.
> > > > >
> > > > > On Mon, Nov 14, 2016 at 5:56 PM, radai  >
> > > > wrote:
> > > > > > @rajini - fixed the hasBytesBuffered() method. also updated
> poll()
> > so
> > > > > that
> > > > > > no latency is added for picking up data stuck in ssl buffers
> > (timeout
> > > > is
> > > > > > set to 0, just like with immediately connected keys and staged
> > > > receives).
> > > > > > thank you for pointing these out.
> > > > > > added ssl (re) testing to the KIP testing plan.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 14, 2016 at 7:24 AM, Rajini Sivaram <
> > > > > > rajinisiva...@googlemail.com> wrote:
> > > > > >
> > > > > >> Open point 1. I would just retain the current long value that
> > > > specifies
> > > > > >> queued.max.bytes as long and not as %heap since it is simple and
> > > easy
> > > > to
> > > > > >> use. And keeps it consistent with other ".bytes" configs.
> > > > > >>
> > > > > >> Point 3. ssl buffers - I am not quite sure the implementation
> > looks
> > > > > >> correct. hasBytesBuffered() is checking position() of buffers ==
> > 0.
> > > > And
> > > > > the
> > > > > >> code checks this only when poll with a timeout returns (adding a
> > > delay
> > > > > when
> > > > > >> there is nothing else to read).
> > > > > >> But since this and open point 2 (optimization) are
> implementation
> > > > > details,
> > > > > >> they can be looked at during PR review.
> > > > > >>
> > > > > >> It will be good to add SSL testing to the test plan as well,
> since
> > > > > there is
> > > > > >> additional code to test for SSL.
> > > > > >>
> > > > > >>
> > > > > >> On Fri, Nov 11, 2016 at 9:03 PM, radai <
> > radai.rosenbl...@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >> > ok, 

[jira] [Created] (KAFKA-4423) Drop support for Java 7

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-4423:
--

 Summary: Drop support for Java 7
 Key: KAFKA-4423
 URL: https://issues.apache.org/jira/browse/KAFKA-4423
 Project: Kafka
  Issue Type: Task
Reporter: Ismael Juma
Assignee: Ismael Juma


Java 7 was released in July 2011, it hasn't received public updates since April 
2015, Java 8 was released in March 2014 and Java 9 is scheduled to be released 
in July 2017.

The last public release of JDK 7 by Oracle contains a large number of known 
security vulnerabilities and Java 8 introduces a number of
compelling features and we will soon have to support Java 9 so it would be good 
to drop support for Java 7 in 2017. The actual timing would depend on when we 
release the next major release of Kafka.

More details on pros and cons are captured in the following discussion thread 
in the mailing list:

http://search-hadoop.com/m/Kafka/uyzND1oIhV61GS5Sf2

Before we can do this, we need to discuss and vote on a concrete proposal in 
the mailing list.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4422) Drop support for Scala 2.10

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4422:
---
Issue Type: Task  (was: Bug)

> Drop support for Scala 2.10
> ---
>
> Key: KAFKA-4422
> URL: https://issues.apache.org/jira/browse/KAFKA-4422
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.11.0.0
>
>
> Now that Scala 2.12 has been released, we should drop support for Scala 2.10  
> in the next major Kafka version so that we keep the number of supported 
> versions at 2. Since we have to compile and run the tests on each supported 
> version, there is a non-trivial cost from a development and testing 
> perspective.
> The clients library is in Java and we recommend people use the Java clients 
> instead of the Scala ones, so dropping support for Scala 2.10 should have a 
> smaller impact than it would have had in the past. Scala 2.10 was released in 
> January 2013 and support ended in March 2015. 
> Once we drop support for Scala 2.10, we can take advantage of APIs and 
> compiler improvements introduced in Scala 2.11 (introduced in April 2014): 
> http://scala-lang.org/news/2.11.0
> Before we do this, we should start a discussion thread followed by a vote in 
> the mailing list.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4422) Drop support for Scala 2.10

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-4422:
--

 Summary: Drop support for Scala 2.10
 Key: KAFKA-4422
 URL: https://issues.apache.org/jira/browse/KAFKA-4422
 Project: Kafka
  Issue Type: Bug
Reporter: Ismael Juma
Assignee: Ismael Juma
 Fix For: 0.11.0.0


Now that Scala 2.12 has been released, we should drop support for Scala 2.10  
in the next major Kafka version so that we keep the number of supported 
versions at 2. Since we have to compile and run the tests on each supported 
version, there is a non-trivial cost from a development and testing perspective.

The clients library is in Java and we recommend people use the Java clients 
instead of the Scala ones, so dropping support for Scala 2.10 should have a 
smaller impact than it would have had in the past. Scala 2.10 was released in 
January 2013 and support ended in March 2015. 

Once we drop support for Scala 2.10, we can take advantage of APIs and compiler 
improvements introduced in Scala 2.11 (introduced in April 2014): 
http://scala-lang.org/news/2.11.0

Before we do this, we should start a discussion thread followed by a vote in 
the mailing list.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2149: HOTFIX: Hotfix streams smoke test

[enothereska]

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/2149

HOTFIX: Hotfix streams smoke test



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka hotfix-streams-smoke-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2149.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2149


commit 87992a8fc548855da28adbced3b1aaad8616b99e
Author: Eno Thereska 
Date:   2016-11-18T14:17:18Z

Increased wait time




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-4421) Update release process so that Scala 2.12 artifacts are published

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-4421:
--

 Summary: Update release process so that Scala 2.12 artifacts are 
published
 Key: KAFKA-4421
 URL: https://issues.apache.org/jira/browse/KAFKA-4421
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma
 Fix For: 0.10.2.0


Since Scala 2.12 requires Java 8 while Kafka still supports Java 7, the *All 
commands don't include Scala 2.12. As such, simply running releaseTarGzAll 
won't generate the Scala 2.12 artifacts and we also need to run `./gradlew 
releaseTagGz -PscalaVersion=2.12`.

The following page needs to be updated to include this and any other change 
required:

https://cwiki.apache.org/confluence/display/KAFKA/Release+Process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4377) Remove deprecated scala.collection.JavaConversions

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-4377.

Resolution: Fixed

> Remove deprecated scala.collection.JavaConversions
> --
>
> Key: KAFKA-4377
> URL: https://issues.apache.org/jira/browse/KAFKA-4377
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Reporter: Bernard Leach
>Assignee: Bernard Leach
> Fix For: 0.10.2.0
>
>
> The 2.12 compiler is generating a large number of deprecation warnings, those 
> that can be resolved in a 2.10 compatible way should be fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4377) Remove deprecated scala.collection.JavaConversions

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4377:
---
Summary: Remove deprecated scala.collection.JavaConversions  (was: Address 
2.12 deprecations)

> Remove deprecated scala.collection.JavaConversions
> --
>
> Key: KAFKA-4377
> URL: https://issues.apache.org/jira/browse/KAFKA-4377
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Reporter: Bernard Leach
> Fix For: 0.10.2.0
>
>
> The 2.12 compiler is generating a large number of deprecation warnings, those 
> that can be resolved in a 2.10 compatible way should be fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4377) Remove deprecated scala.collection.JavaConversions

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4377:
---
Fix Version/s: 0.10.2.0

> Remove deprecated scala.collection.JavaConversions
> --
>
> Key: KAFKA-4377
> URL: https://issues.apache.org/jira/browse/KAFKA-4377
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Reporter: Bernard Leach
>Assignee: Bernard Leach
> Fix For: 0.10.2.0
>
>
> The 2.12 compiler is generating a large number of deprecation warnings, those 
> that can be resolved in a 2.10 compatible way should be fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4377) Remove deprecated scala.collection.JavaConversions

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4377:
---
Assignee: Bernard Leach

> Remove deprecated scala.collection.JavaConversions
> --
>
> Key: KAFKA-4377
> URL: https://issues.apache.org/jira/browse/KAFKA-4377
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Reporter: Bernard Leach
>Assignee: Bernard Leach
> Fix For: 0.10.2.0
>
>
> The 2.12 compiler is generating a large number of deprecation warnings, those 
> that can be resolved in a 2.10 compatible way should be fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4377) Address 2.12 deprecations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15676770#comment-15676770
 ] 

ASF GitHub Bot commented on KAFKA-4377:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2101


> Address 2.12 deprecations
> -
>
> Key: KAFKA-4377
> URL: https://issues.apache.org/jira/browse/KAFKA-4377
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Reporter: Bernard Leach
>
> The 2.12 compiler is generating a large number of deprecation warnings, those 
> that can be resolved in a 2.10 compatible way should be fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4355) StreamThread intermittently dies with "Topic not found during partition assignment" when broker restarted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15676554#comment-15676554
 ] 

ASF GitHub Bot commented on KAFKA-4355:
---

Github user enothereska closed the pull request at:

https://github.com/apache/kafka/pull/2133


> StreamThread intermittently dies with "Topic not found during partition 
> assignment" when broker restarted
> -
>
> Key: KAFKA-4355
> URL: https://issues.apache.org/jira/browse/KAFKA-4355
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0, 0.10.0.0
> Environment: kafka 0.10.0.0
> kafka 0.10.1.0
> uname -a
> Linux lp02485 4.4.0-34-generic #53~14.04.1-Ubuntu SMP Wed Jul 27 16:56:40 UTC 
> 2016 x86_64 x86_64 x86_64 GNU/Linux
> java -version
> java version "1.8.0_92"
> Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
> Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)
>Reporter: Michal Borowiecki
>Assignee: Eno Thereska
>  Labels: architecture
>
> When (a) starting kafka streams app before the broker or
> (b) restarting the broker while the streams app is running:
> the stream thread intermittently dies with "Topic not found during partition 
> assignment" StreamsException.
> This happens about between one in 5 or one in 10 times.
> Stack trace:
> {noformat}
> Exception in thread "StreamThread-2" 
> org.apache.kafka.streams.errors.StreamsException: Topic not found during 
> partition assignment: scheduler
>   at 
> org.apache.kafka.streams.processor.DefaultPartitionGrouper.maxNumPartitions(DefaultPartitionGrouper.java:81)
>   at 
> org.apache.kafka.streams.processor.DefaultPartitionGrouper.partitionGroups(DefaultPartitionGrouper.java:55)
>   at 
> org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.assign(StreamPartitionAssignor.java:370)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:313)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:467)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1000(AbstractCoordinator.java:88)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:419)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:395)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:742)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:722)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:186)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:149)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:116)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:479)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:316)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:256)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:180)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:308)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:277)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:259)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:407)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242)
> {noformat}
> Our app has 2 streams in it, consuming from 2 different topics.
> Sometimes the exception happens on both stream threads. Sometimes only on one 
> of the stream threads.
> The exception is preceded by:
> {noformat}
> [2016-10-28 16:17:55,239] INFO [StreamThread-2] (Re-)joining group 
> pool-scheduler 
> 

[jira] [Commented] (KAFKA-4355) StreamThread intermittently dies with "Topic not found during partition assignment" when broker restarted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15676555#comment-15676555
 ] 

ASF GitHub Bot commented on KAFKA-4355:
---

GitHub user enothereska reopened a pull request:

https://github.com/apache/kafka/pull/2133

KAFKA-4355: Skip topics that have no partitions



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka KAFKA-4355-topic-not-found

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2133.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2133


commit 6acb95f2da291072f20b92b47bd078a47922c2e5
Author: Eno Thereska 
Date:   2016-11-15T10:03:21Z

Skip topics that have no partitions

commit 3a0a5b0f4385aebe0c7adfbdc781cd265ea3729a
Author: Eno Thereska 
Date:   2016-11-18T10:17:50Z

Added a test




> StreamThread intermittently dies with "Topic not found during partition 
> assignment" when broker restarted
> -
>
> Key: KAFKA-4355
> URL: https://issues.apache.org/jira/browse/KAFKA-4355
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0, 0.10.0.0
> Environment: kafka 0.10.0.0
> kafka 0.10.1.0
> uname -a
> Linux lp02485 4.4.0-34-generic #53~14.04.1-Ubuntu SMP Wed Jul 27 16:56:40 UTC 
> 2016 x86_64 x86_64 x86_64 GNU/Linux
> java -version
> java version "1.8.0_92"
> Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
> Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)
>Reporter: Michal Borowiecki
>Assignee: Eno Thereska
>  Labels: architecture
>
> When (a) starting kafka streams app before the broker or
> (b) restarting the broker while the streams app is running:
> the stream thread intermittently dies with "Topic not found during partition 
> assignment" StreamsException.
> This happens about between one in 5 or one in 10 times.
> Stack trace:
> {noformat}
> Exception in thread "StreamThread-2" 
> org.apache.kafka.streams.errors.StreamsException: Topic not found during 
> partition assignment: scheduler
>   at 
> org.apache.kafka.streams.processor.DefaultPartitionGrouper.maxNumPartitions(DefaultPartitionGrouper.java:81)
>   at 
> org.apache.kafka.streams.processor.DefaultPartitionGrouper.partitionGroups(DefaultPartitionGrouper.java:55)
>   at 
> org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.assign(StreamPartitionAssignor.java:370)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:313)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:467)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1000(AbstractCoordinator.java:88)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:419)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:395)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:742)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:722)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:186)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:149)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:116)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:479)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:316)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:256)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:180)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:308)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:277)
>   at 
> 

[GitHub] kafka pull request #2133: KAFKA-4355: Skip topics that have no partitions

[enothereska]

GitHub user enothereska reopened a pull request:

https://github.com/apache/kafka/pull/2133

KAFKA-4355: Skip topics that have no partitions



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka KAFKA-4355-topic-not-found

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2133.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2133


commit 6acb95f2da291072f20b92b47bd078a47922c2e5
Author: Eno Thereska 
Date:   2016-11-15T10:03:21Z

Skip topics that have no partitions

commit 3a0a5b0f4385aebe0c7adfbdc781cd265ea3729a
Author: Eno Thereska 
Date:   2016-11-18T10:17:50Z

Added a test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2133: KAFKA-4355: Skip topics that have no partitions

[enothereska]

Github user enothereska closed the pull request at:

https://github.com/apache/kafka/pull/2133


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4391) On Windows, Kafka server stops with uncaught exception after coming back from sleep

[Yiquan Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15676418#comment-15676418
 ] 

Yiquan Zhou commented on KAFKA-4391:


I did some tests with 0.10.1.0, I still got the same issue with a different 
exception, probably due to the use of Files.move instead of Files.renameTo:
{code}
[2016-11-18 11:24:31,357] FATAL [Replica Manager on Broker 0]: Error writing to 
highwatermark file:  (kafka.server.ReplicaManager)
java.nio.file.FileAlreadyExistsException: 
D:\tmp\kafka-logs\replication-offset-checkpoint.tmp -> 
D:\tmp\kafka-logs\replication-offset-checkpoint
at 
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:81)
at 
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
at 
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1345)
at 
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670)
at kafka.server.OffsetCheckpoint.write(OffsetCheckpoint.scala:74)
at 
kafka.server.ReplicaManager$$anonfun$checkpointHighWatermarks$2.apply(ReplicaManager.scala:927)
at 
kafka.server.ReplicaManager$$anonfun$checkpointHighWatermarks$2.apply(ReplicaManager.scala:924)
at 
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
at 
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at 
kafka.server.ReplicaManager.checkpointHighWatermarks(ReplicaManager.scala:924)
at 
kafka.server.ReplicaManager$$anonfun$1.apply$mcV$sp(ReplicaManager.scala:162)
at 
kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110)
at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:58)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Suppressed: java.nio.file.AccessDeniedException: 
D:\tmp\kafka-logs\replication-offset-checkpoint.tmp -> 
D:\tmp\kafka-logs\replication-offset-checkpoint
at 
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at 
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
at 
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1345)
at 
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667)
... 17 more
{code}

So maybe this is not the root cause of the issue.


> On Windows, Kafka server stops with uncaught exception after coming back from 
> sleep
> ---
>
> Key: KAFKA-4391
> URL: https://issues.apache.org/jira/browse/KAFKA-4391
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
> Environment: Windows 10, jdk1.8.0_111
>Reporter: Yiquan Zhou
>
> Steps to reproduce:
> 1. start the zookeeper
> $ bin\windows\zookeeper-server-start.bat config/zookeeper.properties
> 2. start the Kafka server with the default properties
> $ bin\windows\kafka-server-start.bat config/server.properties
> 3. put Windows into sleep mode for 1-2 hours
> 4. activate Windows again, an exception occurs in Kafka server console and 
> the server is stopped:
> {code:title=kafka console log}
> [2016-11-08 21:45:35,185] INFO Client session timed out, have not heard from 
> server in 10081379ms for sessionid 0x1584514da47, closing socket 
> connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
> [2016-11-08 21:45:40,698] INFO zookeeper state changed (Disconnected) 
> (org.I0Itec.zkclient.ZkClient)
> [2016-11-08 21:45:43,029] INFO Opening socket connection to server 
> 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL 
> (unknown error) (org.apache.zookeeper.ClientCnxn)
> [2016-11-08 21:45:43,044] INFO Socket connection established to 
> 127.0.0.1/127.0.0.1:2181, initiating session 

Re: [DISCUSS] KIP-82 - Add Record Headers

[K Burstev]

For what it is worth also i agree. As a user:

 1) Yes - Headers are worthwhile
 2) Yes - Headers should be a top level option

14.11.2016, 21:15, "Ignacio Solis" :
> 1) Yes - Headers are worthwhile
> 2) Yes - Headers should be a top level option
>
> On Mon, Nov 14, 2016 at 9:16 AM, Michael Pearce 
> wrote:
>
>>  Hi Roger,
>>
>>  The kip details/examples the original proposal for key spacing , not the
>>  new mentioned as per discussion namespace idea.
>>
>>  We will need to update the kip, when we get agreement this is a better
>>  approach (which seems to be the case if I have understood the general
>>  feeling in the conversation)
>>
>>  Re the variable ints, at very early stage we did think about this. I think
>>  the added complexity for the saving isn't worth it. I'd rather go with, if
>>  we want to reduce overheads and size int16 (2bytes) keys as it keeps it
>>  simple.
>>
>>  On the note of no headers, there is as per the kip as we use an attribute
>>  bit to denote if headers are present or not as such provides a zero
>>  overhead currently if headers are not used.
>>
>>  I think as radai mentions would be good first if we can get clarity if do
>>  we now have general consensus that (1) headers are worthwhile and useful,
>>  and (2) we want it as a top level entity.
>>
>>  Just to state the obvious i believe (1) headers are worthwhile and (2)
>>  agree as a top level entity.
>>
>>  Cheers
>>  Mike
>>  
>>  From: Roger Hoover 
>>  Sent: Wednesday, November 9, 2016 9:10:47 PM
>>  To: dev@kafka.apache.org
>>  Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>>
>>  Sorry for going a little in the weeds but thanks for the replies regarding
>>  varint.
>>
>>  Agreed that a prefix and {int, int} can be the same. It doesn't look like
>>  that's what the KIP is saying the "Open" section. The example shows
>>  211
>>  for New Relic and 210002 for App Dynamics implying that the New Relic
>>  organization will have only a single header id to work with. Or is 211
>>  a prefix? The main point of a namespace or prefix is to reduce the
>>  overhead of config mapping or registration depending on how
>>  namespaces/prefixes are managed.
>>
>>  Would love to hear more feedback on the higher-level questions though...
>>
>>  Cheers,
>>
>>  Roger
>>
>>  On Wed, Nov 9, 2016 at 11:38 AM, radai  wrote:
>>
>>  > I think this discussion is getting a bit into the weeds on technical
>>  > implementation details.
>>  > I'd liek to step back a minute and try and establish where we are in the
>>  > larger picture:
>>  >
>>  > (re-wording nacho's last paragraph)
>>  > 1. are we all in agreement that headers are a worthwhile and useful
>>  > addition to have? this was contested early on
>>  > 2. are we all in agreement on headers as top level entity vs headers
>>  > squirreled-away in V?
>>  >
>>  > if there are still concerns around these #2 points (#jay? #jun?)?
>>  >
>>  > (and now back to our normal programming ...)
>>  >
>>  > varints are nice. having said that, its adding complexity (see
>>  > https://github.com/addthis/stream-lib/blob/master/src/
>>  > main/java/com/clearspring/analytics/util/Varint.java
>>  > as 1st google result) and would require anyone writing other clients (C?
>>  > Python? Go? Bash? ;-) ) to get/implement the same, and for relatively
>>  > little gain (int vs string is order of magnitude, this isnt).
>>  >
>>  > int namespacing vs {int, int} namespacing are basically the same thing -
>>  > youre just namespacing an int64 and giving people while 2^32 ranges at a
>>  > time. the part i like about this is letting people have a large swath of
>>  > numbers with one registration so they dont have to come back for every
>>  > single plugin/header they want to "reserve".
>>  >
>>  >
>>  > On Wed, Nov 9, 2016 at 11:01 AM, Roger Hoover 
>>  > wrote:
>>  >
>>  > > Since some of the debate has been about overhead + performance, I'm
>>  > > wondering if we have considered a varint encoding (
>>  > > https://developers.google.com/protocol-buffers/docs/encoding#varints)
>>  > for
>>  > > the header length field (int32 in the proposal) and for header ids? If
>>  > you
>>  > > don't use headers, the overhead would be a single byte and for each
>>  > header
>>  > > id < 128 would also need only a single byte?
>>  > >
>>  > >
>>  > >
>>  > > On Wed, Nov 9, 2016 at 6:43 AM, radai 
>>  > wrote:
>>  > >
>>  > > > @magnus - and very dangerous (youre essentially downloading and
>>  > executing
>>  > > > arbitrary code off the internet on your servers ... bad idea without
>>  a
>>  > > > sandbox, even with)
>>  > > >
>>  > > > as for it being a purely administrative task - i disagree.
>>  > > >
>>  > > > i wish it would, really, because then my earlier point on the
>>  > complexity
>>  > > of
>>  > > > the remapping process would be