date:20200109

[jira] [Resolved] (KAFKA-9385) Connect cluster: connector task repeat like a splitbrain cluster problem

2020-01-09 Thread kaikai.hou (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kaikai.hou resolved KAFKA-9385.
---
Resolution: Duplicate

> Connect cluster: connector task repeat like a splitbrain cluster problem 
> -
>
> Key: KAFKA-9385
> URL: https://issues.apache.org/jira/browse/KAFKA-9385
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: kaikai.hou
>Priority: Major
> Attachments: 12_31_d8c7j_1.jpg
>
>
> I am using Debezium. And find a task repeat 
> problem.[Jump|[https://issues.redhat.com/browse/DBZ-1573?jql=key%20in%20watchedIssues()]]
>  
> 1. I push the Debezium image to our private image repository.
> 2. Deploy the connect cluster with the following *Deployment Config*：
> {code:java}
> //代码占位符
> apiVersion: apps.openshift.io/v1
> kind: DeploymentConfig
> metadata:
>   annotations:
> openshift.io/generated-by: OpenShiftWebConsole
>   creationTimestamp: '2019-10-14T07:45:41Z'
>   generation: 29
>   labels:
> app: debezium-test-cloud
>   name: debezium-test-cloud
>   namespace: test
>   resourceVersion: '168496156'
>   selfLink: >-
> 
> /apis/apps.openshift.io/v1/namespaces/test/deploymentconfigs/debezium-test-cloud
>   uid: 9f4f8f4d-ee56-11e9-a5a1-00163e0e008f
> spec:
>   replicas: 2
>   selector:
> app: debezium-test-cloud
> deploymentconfig: debezium-test-cloud
>   strategy:
> activeDeadlineSeconds: 21600
> resources: {}
> rollingParams:
>   intervalSeconds: 1
>   maxSurge: 25%
>   maxUnavailable: 25%
>   timeoutSeconds: 600
>   updatePeriodSeconds: 1
> type: Rolling
>   template:
> metadata:
>   annotations:
> openshift.io/generated-by: OpenShiftWebConsole
>   creationTimestamp: null
>   labels:
> app: debezium-test-cloud
> deploymentconfig: debezium-test-cloud
> spec:
>   containers:
> - env:
> - name: BOOTSTRAP_SERVERS
>   value: '192.168.100.228:9092'
> - name: GROUP_ID
>   value: test-cloud
> - name: CONFIG_STORAGE_TOPIC
>   value: base.test-cloud.config
> - name: OFFSET_STORAGE_TOPIC
>   value: base.test-cloud.offset
> - name: STATUS_STORAGE_TOPIC
>   value: base.test-cloud.status
> - name: CONNECT_KEY_CONVERTER_SCHEMAS_ENABLE
>   value: 'true'
> - name: CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE
>   value: 'true'
> - name: CONNECT_PRODUCER_MAX_REQUEST_SIZE
>   value: '20971520'
> - name: CONNECT_DATABASE_HISTORY_KAFKA_RECOVERY_POLL_INTERVAL_MS
>   value: '1000'
> - name: HEAP_OPTS
>   value: '-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0'
>   image: 
> 'registry.cn-hangzhou.aliyuncs.com/eshine/debeziumconnect:1.0.0.Beta2'
>   imagePullPolicy: IfNotPresent
>   name: debezium-test-cloud
>   ports:
> - containerPort: 8083
>   protocol: TCP
> - containerPort: 8778
>   protocol: TCP
> - containerPort: 9092
>   protocol: TCP
> - containerPort: 9779
>   protocol: TCP
>   resources:
> limits:
>   cpu: 400m
>   memory: 1Gi
> requests:
>   cpu: 200m
>   memory: 1Gi
>   terminationMessagePath: /dev/termination-log
>   terminationMessagePolicy: File
>   volumeMounts:
> - mountPath: /kafka/config
>   name: debezium-test-cloud-1
> - mountPath: /kafka/data
>   name: debezium-test-cloud-2
> - mountPath: /kafka/logs
>   name: debezium-test-cloud-3
>   dnsPolicy: ClusterFirst
>   restartPolicy: Always
>   schedulerName: default-scheduler
>   securityContext: {}
>   terminationGracePeriodSeconds: 30
>   volumes:
> - emptyDir: {}
>   name: debezium-test-cloud-1
> - emptyDir: {}
>   name: debezium-test-cloud-2
> - emptyDir: {}
>   name: debezium-test-cloud-3
>   test: false
>   triggers:
> - type: ConfigChange
> status:
>   availableReplicas: 2
>   conditions:
> - lastTransitionTime: '2019-11-25T06:44:30Z'
>   lastUpdateTime: '2019-11-25T06:44:44Z'
>   message: replication controller "debezium-test-cloud-15" successfully 
> rolled out
>   reason: NewReplicationControllerAvailable
>   status: 'True'
>   type: Progressing
> - lastTransitionTime: '2019-12-31T10:06:23Z'
>   lastUpdateTime: '2019-12-31T10:06:23Z'
>   message: Deployment config has minimum availability.
>   status: 'True'
>

Build failed in Jenkins: kafka-2.4-jdk8 #125

2020-01-09 Thread Apache Jenkins Server

See 


Changes:

[jason] KAFKA-9144; Track timestamp from txn markers to prevent early producer


--
[...truncated 5.50 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.st

Build failed in Jenkins: kafka-trunk-jdk11 #1068

2020-01-09 Thread Apache Jenkins Server

See 

Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H23 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Could not init 

at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:916)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:708)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:212)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to 
H23
at 
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
at 
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:957)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
at sun.reflect.GeneratedMethodAccessor975.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
at com.sun.proxy.$Proxy144.execute(Unknown Source)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1152)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
at hudson.scm.SCM.checkout(SCM.java:504)
at 
hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at 
jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1815)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at 
hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Error performing git command
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2181)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2140)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2136)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1741)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:914)
... 11 more
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at hudson.Proc$LocalProc.(Proc.java:270)
at hudson.Proc$LocalProc.(Proc.java:219)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:937)
at hudson.Launcher$ProcStarter.start(Launcher.java:455)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2168)
... 15 more
ERROR: Error cloning remote repo 'origin'
Retrying after 10 seconds
No credentials specified
 > git rev-parse --is-inside-work-tree # timeout=10
ERROR: Workspace has a .git repository, but it appears to be corrupt.
hudson.plugins.git.GitException: Error performing git command
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCo

Build failed in Jenkins: kafka-trunk-jdk11 #1067

2020-01-09 Thread Apache Jenkins Server

See 

Changes:


--
Started by an SCM change
Started by an SCM change
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H23 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Could not init 

at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:916)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:708)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:212)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to 
H23
at 
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
at 
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:957)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
at sun.reflect.GeneratedMethodAccessor975.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
at com.sun.proxy.$Proxy144.execute(Unknown Source)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1152)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
at hudson.scm.SCM.checkout(SCM.java:504)
at 
hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at 
jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1815)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at 
hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Error performing git command
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2181)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2140)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2136)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1741)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:914)
... 11 more
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at hudson.Proc$LocalProc.(Proc.java:270)
at hudson.Proc$LocalProc.(Proc.java:219)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:937)
at hudson.Launcher$ProcStarter.start(Launcher.java:455)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2168)
... 15 more
ERROR: Error cloning remote repo 'origin'
Retrying after 10 seconds
No credentials specified
 > git rev-parse --is-inside-work-tree # timeout=10
ERROR: Workspace has a .git repository, but it appears to be corrupt.
hudson.plugins.git.GitException: Error performing git command
at 
org.

Build failed in Jenkins: kafka-trunk-jdk11 #1066

2020-01-09 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-9144; Track timestamp from txn markers to prevent early producer

[github] KAFKA-9294: Add tests for Named parameter (#7874)

[github] KAFKA-8421: Still return data during rebalance (#7312)


--
[...truncated 5.71 MB...]

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTime

Jenkins build is back to normal : kafka-trunk-jdk8 #4146

2020-01-09 Thread Apache Jenkins Server

See

[jira] [Resolved] (KAFKA-8617) Replace EndTxn request/response with automated protocol

2020-01-09 Thread Jason Gustafson (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8617.

Fix Version/s: 2.5.0
   Resolution: Fixed

> Replace EndTxn request/response with automated protocol
> ---
>
> Key: KAFKA-8617
> URL: https://issues.apache.org/jira/browse/KAFKA-8617
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Boyang Chen
>Assignee: Boyang Chen
>Priority: Major
> Fix For: 2.5.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-555: Deprecate Direct Zookeeper access in Kafka Administrative Tools

2020-01-09 Thread Colin McCabe

That's a good question.  The current plan is for the 3.x releases to still 
require ZooKeeper.  What we will drop in 3.x is direct ZK access in 
command-line administrative tools (unless those tools are specifically about 
administering ZK itself, like launching it, stopping it, editing its internal 
settings, etc.)

best,
Colin

On Thu, Jan 9, 2020, at 16:45, Jose Garcia Sancio wrote:
> Thanks Colin,
> 
> For the tools that only support zookeeper (zookeeper-security-migration.sh
> and zookeeper-shell.sh) should we be deprecating the entire tool for
> removal in a future 3.0 release?
> 
> On Thu, Jan 9, 2020 at 4:24 PM Colin McCabe  wrote:
> 
> > Hi all,
> >
> > I wrote KIP about deprecating the --zookeeper flag in the administrative
> > tools.  Take a look if you get a chance:
> >
> > https://cwiki.apache.org/confluence/x/sotSC
> >
> > best,
> > Colin
> >
> 
> 
> -- 
> -Jose
>

Re: [DISCUSS] KIP-555: Deprecate Direct Zookeeper access in Kafka Administrative Tools

2020-01-09 Thread Colin McCabe

On Thu, Jan 9, 2020, at 16:46, Gwen Shapira wrote:
> I think you linked to the wrong KIP? Here's a link that worked for me:
> https://s.apache.org/ve15g

Oops!  Thanks for the correction.  That is indeed the right link.

Colin

> 
> On Thu, Jan 9, 2020 at 4:24 PM Colin McCabe  wrote:
> >
> > Hi all,
> >
> > I wrote KIP about deprecating the --zookeeper flag in the administrative 
> > tools.  Take a look if you get a chance:
> >
> > https://cwiki.apache.org/confluence/x/sotSC
> >
> > best,
> > Colin
>

Re: [DISCUSS] KIP-551: Expose disk read and write metrics

2020-01-09 Thread Colin McCabe

On Thu, Jan 9, 2020, at 16:39, Jose Garcia Sancio wrote:
> Thanks Colin,
> 
> LGTM in general. The Linux documentation (
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/filesystems/proc.txt?id=HEAD#n1644)
> defines these metrics as
> 
> read_bytes
> > --
> >
> > I/O counter: bytes read
> > Attempt to count the number of bytes which this process really did cause to
> > be fetched from the storage layer. Done at the submit_bio() level, so it is
> > accurate for block-backed filesystems.  > CIFS at a later time>
> >
> >
> > write_bytes
> > ---
> >
> > I/O counter: bytes written
> > Attempt to count the number of bytes which this process caused to be sent
> > to
> > the storage layer. This is done at page-dirtying time.
> >
> 
> It looks like there is also another metric (cancelled_write_bytes) that
> affects the value reported in written_bytes. Do we want to take that into
> account when reporting the JMX metric
> kafka.server:type=KafkaServer,name=DiskWriteBytes?
> 
> cancelled_write_bytes
> > -
> >
> > The big inaccuracy here is truncate. If a process writes 1MB to a file and
> > then deletes the file, it will in fact perform no writeout. But it will
> > have
> > been accounted as having caused 1MB of write.
> > In other words: The number of bytes which this process caused to not
> > happen,
> > by truncating pagecache. A task can cause "negative" IO too. If this task
> > truncates some dirty pagecache, some IO which another task has been
> > accounted
> > for (in its write_bytes) will not be happening. We _could_ just subtract
> > that
> > from the truncating task's write_bytes, but there is information loss in
> > doing
> > that.

Hi Jose,

That's a good point, which I had overlooked!  I think we should just subtract 
out the cancelled_write_bytes, since it doesn't reflect actual I/O that was 
done.

I don't think the "cancelled" number typically gets that big, but there isn't 
really a reason to count bytes which we intended to write out but then never 
did.  I added a discussion of this to the KIP.

best,
Colin

> 
> 
> 
> On Mon, Jan 6, 2020 at 5:28 PM Colin McCabe  wrote:
> 
> > On Tue, Dec 10, 2019, at 11:10, Magnus Edenhill wrote:
> > > Hi Colin,
> > >
> >
> > Hi Magnus,
> >
> > Thanks for taking a look.
> >
> > > aren't those counters (ever increasing), rather than gauges
> > (fluctuating)?
> >
> > Since this is in the Kafka broker, we're using Yammer.  This might be
> > confusing, but Yammer's concept of a "counter" is not actually monotonic.
> > It can decrease as well as increase.
> >
> > In general Yammer counters require you to call inc(amount) or dec(amount)
> > on them.  This doesn't match up with what we need to do here, which is to
> > (essentially) make a callback into the kernel by reading from /proc.
> >
> > The counter/gauge dichotomy doesn't affect the JMX, (I think?), so it's
> > really kind of an implementation detail.
> >
> > >
> > > You also mention CPU usage as a side note, you could use getrusage(2)'s
> > > ru_utime (user) and ru_stime (sys)
> > > to allow the broker to monitor its own CPU usage.
> > >
> >
> > Interesting idea.  It might be better to save that for a future KIP,
> > though, to avoid scope creep.
> >
> > best,
> > Colin
> >
> > > /Magnus
> > >
> > > Den tis 10 dec. 2019 kl 19:33 skrev Colin McCabe :
> > >
> > > > Hi all,
> > > >
> > > > I wrote KIP about adding support for exposing disk read and write
> > > > metrics.  Check it out here:
> > > >
> > > > https://cwiki.apache.org/confluence/x/sotSC
> > > >
> > > > best,
> > > > Colin
> > > >
> > >
> >
> 
> 
> -- 
> -Jose
>

Re: [DISCUSS] KIP-551: Expose disk read and write metrics

2020-01-09 Thread Colin McCabe

On Thu, Jan 9, 2020, at 16:34, Lucas Bradstreet wrote:
> Hi Colin,
> 
> This is a great idea, as it is very useful to have these metrics in
> addition to the usual Kafka metrics given the impact of hitting disk
> outside of page cache. Describing it as a gauge did initially strike me as
> oldd, but given the way this is works it makes sense to me.
> 
> /proc/[pid]/io appears to only be supported as of kernel 2.6.20. Given that
> was released back in 2007, maybe it's safe enough to assume it exists, but
> I thought I would mention that anyway.

Hi Lucas,

Thanks for taking a look.

Systems without /proc/[pid]/io will silently fall back to not having this 
metric.  I think there should be very few Linux systems that don't have it, 
though, as you noted.

> 
> Without bikeshedding the metric names, would including a "Total" in the
> name be better e.g. kafka.server:type=KafkaServer,name=DiskReadBytesTotal?
> 

That's a good idea.  I changed the proposed names to TotalDiskReadBytes and 
TotalDiskWriteBytes.

regards,
Colin

> Cheers,
> 
> Lucas
> 
> 
> On Mon, Jan 6, 2020 at 5:28 PM Colin McCabe  wrote:
> 
> > On Tue, Dec 10, 2019, at 11:10, Magnus Edenhill wrote:
> > > Hi Colin,
> > >
> >
> > Hi Magnus,
> >
> > Thanks for taking a look.
> >
> > > aren't those counters (ever increasing), rather than gauges
> > (fluctuating)?
> >
> > Since this is in the Kafka broker, we're using Yammer.  This might be
> > confusing, but Yammer's concept of a "counter" is not actually monotonic.
> > It can decrease as well as increase.
> >
> > In general Yammer counters require you to call inc(amount) or dec(amount)
> > on them.  This doesn't match up with what we need to do here, which is to
> > (essentially) make a callback into the kernel by reading from /proc.
> >
> > The counter/gauge dichotomy doesn't affect the JMX, (I think?), so it's
> > really kind of an implementation detail.
> >
> > >
> > > You also mention CPU usage as a side note, you could use getrusage(2)'s
> > > ru_utime (user) and ru_stime (sys)
> > > to allow the broker to monitor its own CPU usage.
> > >
> >
> > Interesting idea.  It might be better to save that for a future KIP,
> > though, to avoid scope creep.
> >
> > best,
> > Colin
> >
> > > /Magnus
> > >
> > > Den tis 10 dec. 2019 kl 19:33 skrev Colin McCabe :
> > >
> > > > Hi all,
> > > >
> > > > I wrote KIP about adding support for exposing disk read and write
> > > > metrics.  Check it out here:
> > > >
> > > > https://cwiki.apache.org/confluence/x/sotSC
> > > >
> > > > best,
> > > > Colin
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-555: Deprecate Direct Zookeeper access in Kafka Administrative Tools

2020-01-09 Thread Gwen Shapira

I think you linked to the wrong KIP? Here's a link that worked for me:
https://s.apache.org/ve15g

On Thu, Jan 9, 2020 at 4:24 PM Colin McCabe  wrote:
>
> Hi all,
>
> I wrote KIP about deprecating the --zookeeper flag in the administrative 
> tools.  Take a look if you get a chance:
>
> https://cwiki.apache.org/confluence/x/sotSC
>
> best,
> Colin

Re: [DISCUSS] KIP-555: Deprecate Direct Zookeeper access in Kafka Administrative Tools

2020-01-09 Thread Jose Garcia Sancio

Thanks Colin,

For the tools that only support zookeeper (zookeeper-security-migration.sh
and zookeeper-shell.sh) should we be deprecating the entire tool for
removal in a future 3.0 release?

On Thu, Jan 9, 2020 at 4:24 PM Colin McCabe  wrote:

> Hi all,
>
> I wrote KIP about deprecating the --zookeeper flag in the administrative
> tools.  Take a look if you get a chance:
>
> https://cwiki.apache.org/confluence/x/sotSC
>
> best,
> Colin
>

-- 
-Jose

Re: [DISCUSS] KIP-551: Expose disk read and write metrics

2020-01-09 Thread Jose Garcia Sancio

Thanks Colin,

LGTM in general. The Linux documentation (
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/filesystems/proc.txt?id=HEAD#n1644)
defines these metrics as

read_bytes
> --
>
> I/O counter: bytes read
> Attempt to count the number of bytes which this process really did cause to
> be fetched from the storage layer. Done at the submit_bio() level, so it is
> accurate for block-backed filesystems.  CIFS at a later time>
>
>
> write_bytes
> ---
>
> I/O counter: bytes written
> Attempt to count the number of bytes which this process caused to be sent
> to
> the storage layer. This is done at page-dirtying time.
>

It looks like there is also another metric (cancelled_write_bytes) that
affects the value reported in written_bytes. Do we want to take that into
account when reporting the JMX metric
kafka.server:type=KafkaServer,name=DiskWriteBytes?

cancelled_write_bytes
> -
>
> The big inaccuracy here is truncate. If a process writes 1MB to a file and
> then deletes the file, it will in fact perform no writeout. But it will
> have
> been accounted as having caused 1MB of write.
> In other words: The number of bytes which this process caused to not
> happen,
> by truncating pagecache. A task can cause "negative" IO too. If this task
> truncates some dirty pagecache, some IO which another task has been
> accounted
> for (in its write_bytes) will not be happening. We _could_ just subtract
> that
> from the truncating task's write_bytes, but there is information loss in
> doing
> that.



On Mon, Jan 6, 2020 at 5:28 PM Colin McCabe  wrote:

> On Tue, Dec 10, 2019, at 11:10, Magnus Edenhill wrote:
> > Hi Colin,
> >
>
> Hi Magnus,
>
> Thanks for taking a look.
>
> > aren't those counters (ever increasing), rather than gauges
> (fluctuating)?
>
> Since this is in the Kafka broker, we're using Yammer.  This might be
> confusing, but Yammer's concept of a "counter" is not actually monotonic.
> It can decrease as well as increase.
>
> In general Yammer counters require you to call inc(amount) or dec(amount)
> on them.  This doesn't match up with what we need to do here, which is to
> (essentially) make a callback into the kernel by reading from /proc.
>
> The counter/gauge dichotomy doesn't affect the JMX, (I think?), so it's
> really kind of an implementation detail.
>
> >
> > You also mention CPU usage as a side note, you could use getrusage(2)'s
> > ru_utime (user) and ru_stime (sys)
> > to allow the broker to monitor its own CPU usage.
> >
>
> Interesting idea.  It might be better to save that for a future KIP,
> though, to avoid scope creep.
>
> best,
> Colin
>
> > /Magnus
> >
> > Den tis 10 dec. 2019 kl 19:33 skrev Colin McCabe :
> >
> > > Hi all,
> > >
> > > I wrote KIP about adding support for exposing disk read and write
> > > metrics.  Check it out here:
> > >
> > > https://cwiki.apache.org/confluence/x/sotSC
> > >
> > > best,
> > > Colin
> > >
> >
>


-- 
-Jose

Re: [DISCUSS] KIP-551: Expose disk read and write metrics

2020-01-09 Thread Lucas Bradstreet

Hi Colin,

This is a great idea, as it is very useful to have these metrics in
addition to the usual Kafka metrics given the impact of hitting disk
outside of page cache. Describing it as a gauge did initially strike me as
oldd, but given the way this is works it makes sense to me.

/proc/[pid]/io appears to only be supported as of kernel 2.6.20. Given that
was released back in 2007, maybe it's safe enough to assume it exists, but
I thought I would mention that anyway.

Without bikeshedding the metric names, would including a "Total" in the
name be better e.g. kafka.server:type=KafkaServer,name=DiskReadBytesTotal?

Cheers,

Lucas

On Mon, Jan 6, 2020 at 5:28 PM Colin McCabe  wrote:

> On Tue, Dec 10, 2019, at 11:10, Magnus Edenhill wrote:
> > Hi Colin,
> >
>
> Hi Magnus,
>
> Thanks for taking a look.
>
> > aren't those counters (ever increasing), rather than gauges
> (fluctuating)?
>
> Since this is in the Kafka broker, we're using Yammer.  This might be
> confusing, but Yammer's concept of a "counter" is not actually monotonic.
> It can decrease as well as increase.
>
> In general Yammer counters require you to call inc(amount) or dec(amount)
> on them.  This doesn't match up with what we need to do here, which is to
> (essentially) make a callback into the kernel by reading from /proc.
>
> The counter/gauge dichotomy doesn't affect the JMX, (I think?), so it's
> really kind of an implementation detail.
>
> >
> > You also mention CPU usage as a side note, you could use getrusage(2)'s
> > ru_utime (user) and ru_stime (sys)
> > to allow the broker to monitor its own CPU usage.
> >
>
> Interesting idea.  It might be better to save that for a future KIP,
> though, to avoid scope creep.
>
> best,
> Colin
>
> > /Magnus
> >
> > Den tis 10 dec. 2019 kl 19:33 skrev Colin McCabe :
> >
> > > Hi all,
> > >
> > > I wrote KIP about adding support for exposing disk read and write
> > > metrics.  Check it out here:
> > >
> > > https://cwiki.apache.org/confluence/x/sotSC
> > >
> > > best,
> > > Colin
> > >
> >
>

[jira] [Created] (KAFKA-9397) Deprecate Direct Zookeeper access in Kafka Administrative Tools

2020-01-09 Thread Colin McCabe (Jira)

Colin McCabe created KAFKA-9397:
---

 Summary: Deprecate Direct Zookeeper access in Kafka Administrative 
Tools
 Key: KAFKA-9397
 URL: https://issues.apache.org/jira/browse/KAFKA-9397
 Project: Kafka
  Issue Type: Improvement
  Components: tools
Affects Versions: 2.5.0
Reporter: Colin McCabe
Assignee: Colin McCabe
 Fix For: 2.5.0


KIP-555: Deprecate Direct Zookeeper access in Kafka Administrative Tools



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] KIP-555: Deprecate Direct Zookeeper access in Kafka Administrative Tools

2020-01-09 Thread Colin McCabe

Hi all,

I wrote KIP about deprecating the --zookeeper flag in the administrative tools. 
 Take a look if you get a chance:

https://cwiki.apache.org/confluence/x/sotSC

best,
Colin

[jira] [Resolved] (KAFKA-8179) Incremental Rebalance Protocol for Kafka Consumer

2020-01-09 Thread Guozhang Wang (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-8179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-8179.
--
Fix Version/s: 2.4.0
   2.5.0
   Resolution: Fixed

> Incremental Rebalance Protocol for Kafka Consumer
> -
>
> Key: KAFKA-8179
> URL: https://issues.apache.org/jira/browse/KAFKA-8179
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>Priority: Major
> Fix For: 2.5.0, 2.4.0
>
>
> Recently Kafka community is promoting cooperative rebalancing to mitigate the 
> pain points in the stop-the-world rebalancing protocol. This ticket is 
> created to initiate that idea at the Kafka consumer client, which will be 
> beneficial for heavy-stateful consumers such as Kafka Streams applications.
> In short, the scope of this ticket includes reducing unnecessary rebalance 
> latency due to heavy partition migration: i.e. partitions being revoked and 
> re-assigned. This would make the built-in consumer assignors (range, 
> round-robin etc) to be aware of previously assigned partitions and be sticky 
> in best-effort.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-8421) Allow consumer.poll() to return data in the middle of rebalance

2020-01-09 Thread Guozhang Wang (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-8421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-8421.
--
Fix Version/s: 2.5.0
 Assignee: Guozhang Wang
   Resolution: Fixed

> Allow consumer.poll() to return data in the middle of rebalance
> ---
>
> Key: KAFKA-8421
> URL: https://issues.apache.org/jira/browse/KAFKA-8421
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer, streams
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>Priority: Major
> Fix For: 2.5.0
>
>
> With KIP-429 in place, today when a consumer is about to send join-group 
> request its owned partitions may not be empty, meaning that some of its 
> fetched data can still be returned. Nevertheless, today the logic is strict:
> {code}
> if (!updateAssignmentMetadataIfNeeded(timer)) {
> return ConsumerRecords.empty();
> }
> {code}
> I.e. if the consumer enters a rebalance it always returns no data. 
> As an optimization, we can consider letting consumers to still return 
> messages that still belong to its owned partitions even when it is within a 
> rebalance, because we know it is safe that no one else would claim those 
> partitions in this rebalance yet, and we can still commit offsets if, after 
> this rebalance, the partitions need to be revoked then.
> One thing we need to take care though is the rebalance timeout, i.e. when 
> consumer's processing those records they may not call the next poll() in time 
> (think: Kafka Streams num.iterations mechanism), which may leads to consumer 
> dropping out of the group during rebalance.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-8242) Exception in ReplicaFetcher blocks replication of all other partitions

2020-01-09 Thread Jason Gustafson (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-8242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8242.

Resolution: Fixed

Marking this as resolved. We believe we having fixed the cause of the 
coordinator fenced error and we have also improved the replica fetcher behavior 
in KIP-461, as Dhruvil notes.

> Exception in ReplicaFetcher blocks replication of all other partitions
> --
>
> Key: KAFKA-8242
> URL: https://issues.apache.org/jira/browse/KAFKA-8242
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 1.1.1
>Reporter: Nevins Bartolomeo
>Priority: Major
>
> We're seeing the following exception in our replication threads. 
> {code:java}
> [2019-04-16 14:14:39,724] ERROR [ReplicaFetcher replicaId=15, leaderId=8, 
> fetcherId=0] Error due to (kafka.server.ReplicaFetcherThread)
> kafka.common.KafkaException: Error processing data for partition 
> testtopic-123 offset 9880379
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:204)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:169)
> at scala.Option.foreach(Option.scala:257)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:169)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:166)
> at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply$mcV$sp(AbstractFetcherThread.scala:166)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:166)
> at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:166)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:250)
> at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:164)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:111)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
> Caused by: 
> org.apache.kafka.common.errors.TransactionCoordinatorFencedException: Invalid 
> coordinator epoch: 27 (zombie), 31 (current)
> {code}
> While this is an issue itself the larger issue is that this exception kills 
> the replication threads so no other partitions get replicated to this broker. 
> That a single corrupt partition can affect the availability of multiple 
> topics is a great concern to us.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-9144) Early expiration of producer state can cause coordinator epoch to regress

2020-01-09 Thread Jason Gustafson (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-9144.

Fix Version/s: 2.4.1
   Resolution: Fixed

> Early expiration of producer state can cause coordinator epoch to regress
> -
>
> Key: KAFKA-9144
> URL: https://issues.apache.org/jira/browse/KAFKA-9144
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 2.4.1
>
>
> Transaction markers are written by the transaction coordinator. In order to 
> fence zombie coordinators, we use the leader epoch associated with the 
> coordinator partition. Partition leaders verify the epoch in the 
> WriteTxnMarker request and ensure that it can only increase. However, when 
> producer state expires, we stop tracking the epoch and it is possible for 
> monotonicity to be violated. Generally we expect expiration to be on the 
> order of days, so it should be unlikely for this to be a problem.
> At least that is the theory. We observed a case where a coordinator epoch 
> decreased between nearly consecutive writes within a couple minutes of each 
> other. Upon investigation, we found that producer state had been incorrectly 
> expired. We believe the sequence of events is the following:
>  # Producer writes transactional data and fails before committing
>  # Coordinator times out the transaction and writes ABORT markers
>  # Upon seeing the ABORT and the bumped epoch, the partition leader deletes 
> state from the last epoch, which effectively resets the last timestamp for 
> the producer to -1.
>  # The coordinator becomes a zombie before getting a successful response and 
> continues trying to send
>  # The new coordinator notices the incomplete transaction and also sends 
> markers
>  # The partition leader accepts the write from the new coordinator
>  # The producer state is expired because the last timestamp was -1
>  # The partition leader accepts the write from the old coordinator
> Basically it takes an alignment of planets to hit this bug, but it is 
> possible. If you hit it, then the broker may be unable to start because we 
> validate epoch monotonicity during log recovery. The problem is in 3 when the 
> timestamp gets reset. We should use the timestamp from the marker instead.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk8 #4145

2020-01-09 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-9287; Fix unneeded delay before failing pending transaction 
commit


--
[...truncated 2.79 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain NO-S

Jenkins build is back to normal : kafka-trunk-jdk11 #1064

2020-01-09 Thread Apache Jenkins Server

See

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

2020-01-09 Thread Matthias J. Sax

Good question about `StreamsRebalancingException` -- when this KIP was
started, KIP-535 was not on the horizon yet.

What I am wondering is, if we should allow people to opt-in into
querying during a rebalance, or to be more precise during a restore (if
a state store is not migrated, it will be up-to-date during a rebalance
and can be queried returning correct, ie, non-stall, data)?

Otherwise, if people want to get only correct results, ie, they never
want to query stall state, they have no way to implement it, because
they are always subject to a race condition.

For this case, we could have a `StateStoreIsRecoveringException` (or
similar) that is only throw during a restore phases (and people can
opt-in / opt-out if this exception should be throws or not, ie, if they
want to query stall state during recovery or not).

It's unclear to me though atm, how a user would opt-in/opt-out and what
the default should be (maybe better to throw the exception by default to
have strong consistency guarantees by default?)


-Matthias


On 1/9/20 11:35 AM, Vinoth Chandar wrote:
> +1 on merging `StreamsNotRunningException` and 
> `StateStoreNotAvailableException`, both exceptions are fatal anyway. IMO its 
> best to have these exceptions be about the state store (and not streams 
> state), to easier understanding. 
> 
> Additionally, KIP-535 allows for querying of state stores in rebalancing 
> state. So do we need the StreamsRebalancingException? 
> 
> 
> On 2020/01/09 03:38:11, "Matthias J. Sax"  wrote: 
>> Sorry that I dropped the ball on this...
>>
>> Thanks for updating the KIP. Overall LGTM now. Feel free to start a VOTE
>> thread.
>>
>> What is still unclear to me is, what we gain by having both
>> `StreamsNotRunningException` and `StateStoreNotAvailableException`. Both
>> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
>> state store is closed on not -- I can't query it anyway? Maybe I miss
>> something thought?
>>
>>
>> -Matthias
>>
>>
>> On 11/3/19 6:07 PM, Vito Jeng wrote:
>>> Sorry for the late reply, thanks for the review.
>>>
>>>
 About `StateStoreMigratedException`:

 Why is it only thrown if the state is REBALANCING? A store might be
 migrated during a rebalance, and Kafka Streams might resume back to
 RUNNING state and afterward somebody tries to use an old store handle.
 Also, if state is REBALANCING, should we throw
 `StreamThreadRebalancingException`? Hence, I think
 `StateStoreMigratedException` does only make sense during `RUNNING` state.

>>>
>>> Thank you point this, already updated.
>>>
>>>
>>> Why do we need to distinguish between `KafkaStreamsNotRunningException`
 and `StateStoreNotAvailableException`?

>>>
>>> `KafkaStreamsNotRunningException` may be caused by various reasons, I think
>>> it would be helpful that the
>>> user can distinguish whether it is caused by the state store closed.
>>> (Maybe I am wrong...)
>>>
>>>
>>> Last, why do we distinguish between `KafkaStreams` instance and
 `StreamsThread`? To me, it seems we should always refer to the instance,
 because that is the level of granularity in which we enable/disable IQ atm.

>>>
>>> Totally agree. Do you mean the naming of state store exceptions?
>>> I don't have special reason to distinguish these two.
>>> Your suggestion look more reasonable for the exception naming.
>>>
>>>
>>> Last, for `StateStoreMigratedException`, I would add that a user need to
 rediscover the store and cannot blindly retry as the store handle is
 invalid and a new store handle must be retrieved. That is a difference
 to `StreamThreadRebalancingException` that allows for "blind" retries
 that either resolve (if the store is still on the same instance after
 rebalancing finishes, or changes to `StateStoreMigratedException` if the
 store was migrated away during rebalancing).

>>>
>>> Nice, it's great! Thank you.
>>>
>>>
>>> The KIP already updated, please take a look. :)
>>>
>>>
>>>
>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax 
>>> wrote:
>>>
 Any update on this KIP?

 On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> Sorry for the late reply. The 2.4 deadline kept us quite busy.
>
> About `StateStoreMigratedException`:
>
> Why is it only thrown if the state is REBALANCING? A store might be
> migrated during a rebalance, and Kafka Streams might resume back to
> RUNNING state and afterward somebody tries to use an old store handle.
> Also, if state is REBALANCING, should we throw
> `StreamThreadRebalancingException`? Hence, I think
> `StateStoreMigratedException` does only make sense during `RUNNING`
 state.
>
>
> Why do we need to distinguish between `KafkaStreamsNotRunningException`
> and `StateStoreNotAvailableException`?
>
>
> Last, why do we distinguish between `KafkaStreams` instance and
>

Build failed in Jenkins: kafka-trunk-jdk8 #4144

2020-01-09 Thread Apache Jenkins Server

See 


Changes:

[matthias] KAFKA-9383: Expose consumer group metadata (#7906)


--
[...truncated 2.78 MB...]
org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:compileTestJava
> Task :streams:upgrade-system-tests-0101:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:testClasses
> Task :streams:upgrade-system-tests-0101:checkstyleTest
> Task :streams:upgrade-system-tests-0101:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:test
> Task :streams:upgrade-system-tests-0102:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0102:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0102:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0102:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0102:compileTestJava
> Task :streams:upgrade-system-tests-0102:processTestResources NO-SOURCE
> Task :streams:upgra

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

2020-01-09 Thread Vinoth Chandar

+1 on merging `StreamsNotRunningException` and 
`StateStoreNotAvailableException`, both exceptions are fatal anyway. IMO its 
best to have these exceptions be about the state store (and not streams state), 
to easier understanding. 

Additionally, KIP-535 allows for querying of state stores in rebalancing state. 
So do we need the StreamsRebalancingException? 


On 2020/01/09 03:38:11, "Matthias J. Sax"  wrote: 
> Sorry that I dropped the ball on this...
> 
> Thanks for updating the KIP. Overall LGTM now. Feel free to start a VOTE
> thread.
> 
> What is still unclear to me is, what we gain by having both
> `StreamsNotRunningException` and `StateStoreNotAvailableException`. Both
> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
> state store is closed on not -- I can't query it anyway? Maybe I miss
> something thought?
> 
> 
> -Matthias
> 
> 
> On 11/3/19 6:07 PM, Vito Jeng wrote:
> > Sorry for the late reply, thanks for the review.
> > 
> > 
> >> About `StateStoreMigratedException`:
> >>
> >> Why is it only thrown if the state is REBALANCING? A store might be
> >> migrated during a rebalance, and Kafka Streams might resume back to
> >> RUNNING state and afterward somebody tries to use an old store handle.
> >> Also, if state is REBALANCING, should we throw
> >> `StreamThreadRebalancingException`? Hence, I think
> >> `StateStoreMigratedException` does only make sense during `RUNNING` state.
> >>
> > 
> > Thank you point this, already updated.
> > 
> > 
> > Why do we need to distinguish between `KafkaStreamsNotRunningException`
> >> and `StateStoreNotAvailableException`?
> >>
> > 
> > `KafkaStreamsNotRunningException` may be caused by various reasons, I think
> > it would be helpful that the
> > user can distinguish whether it is caused by the state store closed.
> > (Maybe I am wrong...)
> > 
> > 
> > Last, why do we distinguish between `KafkaStreams` instance and
> >> `StreamsThread`? To me, it seems we should always refer to the instance,
> >> because that is the level of granularity in which we enable/disable IQ atm.
> >>
> > 
> > Totally agree. Do you mean the naming of state store exceptions?
> > I don't have special reason to distinguish these two.
> > Your suggestion look more reasonable for the exception naming.
> > 
> > 
> > Last, for `StateStoreMigratedException`, I would add that a user need to
> >> rediscover the store and cannot blindly retry as the store handle is
> >> invalid and a new store handle must be retrieved. That is a difference
> >> to `StreamThreadRebalancingException` that allows for "blind" retries
> >> that either resolve (if the store is still on the same instance after
> >> rebalancing finishes, or changes to `StateStoreMigratedException` if the
> >> store was migrated away during rebalancing).
> >>
> > 
> > Nice, it's great! Thank you.
> > 
> > 
> > The KIP already updated, please take a look. :)
> > 
> > 
> > 
> > On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax 
> > wrote:
> > 
> >> Any update on this KIP?
> >>
> >> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> >>> Sorry for the late reply. The 2.4 deadline kept us quite busy.
> >>>
> >>> About `StateStoreMigratedException`:
> >>>
> >>> Why is it only thrown if the state is REBALANCING? A store might be
> >>> migrated during a rebalance, and Kafka Streams might resume back to
> >>> RUNNING state and afterward somebody tries to use an old store handle.
> >>> Also, if state is REBALANCING, should we throw
> >>> `StreamThreadRebalancingException`? Hence, I think
> >>> `StateStoreMigratedException` does only make sense during `RUNNING`
> >> state.
> >>>
> >>>
> >>> Why do we need to distinguish between `KafkaStreamsNotRunningException`
> >>> and `StateStoreNotAvailableException`?
> >>>
> >>>
> >>> Last, why do we distinguish between `KafkaStreams` instance and
> >>> `StreamsThread`? To me, it seems we should always refer to the instance,
> >>> because that is the level of granularity in which we enable/disable IQ
> >> atm.
> >>>
> >>>
> >>> Last, for `StateStoreMigratedException`, I would add that a user need to
> >>> rediscover the store and cannot blindly retry as the store handle is
> >>> invalid and a new store handle must be retrieved. That is a difference
> >>> to `StreamThreadRebalancingException` that allows for "blind" retries
> >>> that either resolve (if the store is still on the same instance after
> >>> rebalancing finishes, or changes to `StateStoreMigratedException` if the
> >>> store was migrated away during rebalancing).
> >>>
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 8/9/19 10:20 AM, Vito Jeng wrote:
>  My bad. The short link `https://shorturl.at/CDNT9`
> >> 
>   seems incorrect.
> 
>  Please use the following instead: https://shorturl.at/bkKQU
> 
> 
>  ---
>  Vito
> 
> 
>  On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng  wrote:
>

[jira] [Resolved] (KAFKA-9287) Transaction completion may block unnecessarily after abortable error

2020-01-09 Thread Jason Gustafson (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-9287.

Fix Version/s: 2.5.0
   Resolution: Fixed

> Transaction completion may block unnecessarily after abortable error
> 
>
> Key: KAFKA-9287
> URL: https://issues.apache.org/jira/browse/KAFKA-9287
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 2.5.0
>
>
> This was discovered while investigating the delay in 
> `AuthorizerIntegrationTest.testTransactionalProducerTopicAuthorizationExceptionInCommit`
>  which typically takes 35 seconds rather than 5 seconds like most of these 
> other tests in this class. There is an edge case on transaction completion if 
> an AddPartitionsToTxn request fails with an abortable error in which the 
> producer may be left blocking in `NetworkClient.poll` without failing the 
> pending commit and without any pending requests. Ultimately the test case was 
> blocking the full 30s request timeout before failing the transaction.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9395) Improve Kafka scheduler's periodic maybeShrinkIsr()

2020-01-09 Thread Brian Byrne (Jira)

Brian Byrne created KAFKA-9395:
--

 Summary: Improve Kafka scheduler's periodic maybeShrinkIsr()
 Key: KAFKA-9395
 URL: https://issues.apache.org/jira/browse/KAFKA-9395
 Project: Kafka
  Issue Type: Improvement
Reporter: Brian Byrne
Assignee: Brian Byrne


The ReplicaManager schedules a periodic call to maybeShrinkIsr() with the 
KafkaScheduler for a period of replica.lag.time.max.ms / 2. While 
replica.lag.time.max.ms defaults to 30s, my setup was 45s, which means 
maybeShrinkIsr() was being called every 22.5 seconds. Normally this is not a 
problem.

Fetch/produce requests hold a partition's leaderIsrUpdateLock in reader mode 
while they are running. When a partition is requested to check whether it 
should shrink its ISR, it acquires a write lock. So there's potential for 
contention here, and if the fetch/produce requests are long running, they may 
block maybeShrinkIsr() for hundreds of ms.

This becomes a problem due to the way the scheduler runnable is set up: it 
calls maybeShrinkIsr() for partition per single scheduler invocation. If 
there's a lot of partitions, this could take many seconds, even minutes. 
However, the runnable is scheduled via 
ScheduledThreadPoolExecutor#scheduleAtFixedRate, which means if it exceeds its 
period, it's immediately scheduled to run again. So it backs up enough that the 
scheduler is always executing this function.

This may cause partitions to periodically check their ISR a lot less frequently 
than intended. This also contributes a huge source of contention for cases 
where the produce/fetch requests are long-running.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9394) Add partial success for Consumer offset fetch

2020-01-09 Thread Boyang Chen (Jira)

Boyang Chen created KAFKA-9394:
--

 Summary: Add partial success for Consumer offset fetch
 Key: KAFKA-9394
 URL: https://issues.apache.org/jira/browse/KAFKA-9394
 Project: Kafka
  Issue Type: Improvement
Reporter: Boyang Chen


After https://issues.apache.org/jira/browse/KAFKA-9346, it is possible to 
return a type  of {{Map> to 
allow partial success of offset fetch, while letting other pending offsets 
resolved.}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-9383) Expose Consumer Group Metadata for Transactional Producer

2020-01-09 Thread Matthias J. Sax (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-9383.

Fix Version/s: 2.5.0
   Resolution: Fixed

> Expose Consumer Group Metadata for Transactional Producer
> -
>
> Key: KAFKA-9383
> URL: https://issues.apache.org/jira/browse/KAFKA-9383
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Boyang Chen
>Assignee: Boyang Chen
>Priority: Major
> Fix For: 2.5.0
>
>
> As stated in the KIP, we need a mechanism to get latest consumer group 
> metadata for proper commit fencing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

A Grammar for the Kafka Streams DSL

2020-01-09 Thread John Roesler

Hello all,

After many smaller-scope discussions about how to format the objects and 
methods in the Streams DSL, it seems like the time is long past due to 
establish a definition of the language.

I've drafted a grammar that I think will greatly improve the ergonomics of the 
DSL, as well as make like a lot simpler for maintainers: 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+DSL+Grammar . 

This isn't a KIP right now, more like a basis for thinking formally about 
Streams KIPs that touch on the DSL. Nevertheless, I wanted to send this message 
to start a meta-discussion about the grammar I proposed. IMO, it makes more 
sense to have KIPs later on for actually changing the API to comply with the 
grammar, and potentially revising the grammar at that time if it yields absurd 
results.

Looking forward to your comments!

Thanks,
-John

Build failed in Jenkins: kafka-trunk-jdk8 #4143

2020-01-09 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-9379; Fix flaky test

[github] KAFKA-9386; Apply delete ACL filters to resources from filter even if


--
[...truncated 2.78 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldStoreAndReturnStateStores STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldStoreAndReturnStateStores PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureOutputRecordsUsingTo STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureOutputRecordsUsingTo PASSED

org.apache.kafka.streams.MockProcessorContextTest > shouldCaptureOutputRecords 
STARTED

org.apache.kafka.streams.MockProcessorContextTest > shouldCaptureOutputRecords 
PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
fullConstructorShouldSetAllExpectedAttributes STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
fullConstructorShouldSetAllExpectedAttributes PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureCommitsAndAllowReset STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureCommitsAndAllowReset PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildName STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildName PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildIndex STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildIndex PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureApplicationAndRecordMetadata STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureApplicationAndRecordMetadata PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureRecordsOutputToChildByName STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureRecordsOutputToChildByName PASSED

org.apache.kafka.streams.MockProcessorContextTest > shouldCapturePunctuator 
STARTED

org.apache.kafka.streams.MockProcessorContextTest > shouldCapturePunctuator 
PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-s

[jira] [Created] (KAFKA-9393) DeleteRecords triggers extreme lock contention for large partition directories

2020-01-09 Thread Lucas Bradstreet (Jira)

Lucas Bradstreet created KAFKA-9393:
---

 Summary: DeleteRecords triggers extreme lock contention for large 
partition directories
 Key: KAFKA-9393
 URL: https://issues.apache.org/jira/browse/KAFKA-9393
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 2.4.0, 2.3.0, 2.2.0
Reporter: Lucas Bradstreet


DeleteRecords, frequently used by KStreams triggers a 
Log.maybeIncrementLogStartOffset call, calling 
kafka.log.ProducerStateManager.listSnapshotFiles which calls 
java.io.File.listFiles on the partition dir. The time taken to list this 
directory can be extreme for partitions with many small segments (e.g 2) 
taking multiple seconds to finish. This causes lock contention for the log, and 
if produce requests are also occurring for the same log can cause a majority of 
request handler threads to become blocked waiting for the DeleteRecords call to 
finish.

I believe this is a problem going back to the initial implementation of the 
transactional producer, but I need to confirm how far back it goes.

One possible solution is to maintain a producer state snapshot aligned to the 
log segment, and simply delete it whenever we delete a segment. This would 
ensure that we never have to perform a directory scan.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk11 #1063

2020-01-09 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-9379; Fix flaky test


--
[...truncated 2.80 MB...]

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
sh

[jira] [Resolved] (KAFKA-8786) Deprecated Gradle features making it incompatible with Gradle 6.0.

2020-01-09 Thread Ismael Juma (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-8786.

Fix Version/s: 2.5.0
   Resolution: Fixed

> Deprecated Gradle features making it incompatible with Gradle 6.0.
> --
>
> Key: KAFKA-8786
> URL: https://issues.apache.org/jira/browse/KAFKA-8786
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 2.2.1
>Reporter: Aljoscha Pörtner
>Priority: Trivial
>  Labels: easyfix, gradle
> Fix For: 2.5.0
>
> Attachments: Unbenannt.PNG
>
>   Original Estimate: 5m
>  Remaining Estimate: 5m
>
> The lines 549-552 of the build.gradle makes it incompatible with Gradle 6.0. 
> It can be fixed by changing
> {code:java}
> additionalSourceDirs = files(javaProjects.sourceSets.main.allSource.srcDirs)
> sourceDirectories = files(javaProjects.sourceSets.main.allSource.srcDirs)
> classDirectories = files(javaProjects.sourceSets.main.output)
> executionData = files(javaProjects.jacocoTestReport.executionData)
> {code}
> to
> {code:java}
>   additionalSourceDirs.from = javaProjects.sourceSets.main.allSource.srcDirs
>   sourceDirectories.from = javaProjects.sourceSets.main.allSource.srcDirs
>   classDirectories.from = javaProjects.sourceSets.main.output
>   executionData.from = javaProjects.jacocoTestReport.executionData
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-9386) Flaky test AclAuthorizerTest.testHighConcurrencyDeletionOfResourceAcls

2020-01-09 Thread Rajini Sivaram (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-9386.
---
  Reviewer: Manikumar
Resolution: Fixed

> Flaky test AclAuthorizerTest.testHighConcurrencyDeletionOfResourceAcls
> --
>
> Key: KAFKA-9386
> URL: https://issues.apache.org/jira/browse/KAFKA-9386
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.4.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Major
> Fix For: 2.5.0
>
>
> Failure:
> org.scalatest.exceptions.TestFailedException: expected acls:
>  
> but got:
>  (principal=User:alice, host=*, operation=ALL, permissionType=ALLOW)
> at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530)
>  at 
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529)
>  at 
> org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389)
>  at org.scalatest.Assertions.fail(Assertions.scala:1091)
>  at org.scalatest.Assertions.fail$(Assertions.scala:1087)
>  at org.scalatest.Assertions$.fail(Assertions.scala:1389)
>  at kafka.utils.TestUtils$.waitAndVerifyAcls(TestUtils.scala:843)
>  at 
> kafka.security.authorizer.AclAuthorizerTest.testHighConcurrencyDeletionOfResourceAcls(AclAuthorizerTest.scala:552)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-9379) Flaky Test TopicCommandWithAdminClientTest.testCreateAlterTopicWithRackAware

2020-01-09 Thread Rajini Sivaram (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-9379.
---
Fix Version/s: 2.5.0
 Reviewer: Ismael Juma
   Resolution: Fixed

> Flaky Test TopicCommandWithAdminClientTest.testCreateAlterTopicWithRackAware
> 
>
> Key: KAFKA-9379
> URL: https://issues.apache.org/jira/browse/KAFKA-9379
> Project: Kafka
>  Issue Type: Bug
>  Components: admin, core, unit tests
>Reporter: Matthias J. Sax
>Assignee: Rajini Sivaram
>Priority: Critical
>  Labels: flaky-test
> Fix For: 2.5.0
>
>
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/116/testReport/junit/kafka.admin/TopicCommandWithAdminClientTest/testCreateAlterTopicWithRackAware/]
> {quote}java.lang.IllegalArgumentException: Topic 
> 'testCreateAlterTopicWithRackAware-1Ski7jYwdP' does not exist as expected at 
> kafka.admin.TopicCommand$.kafka$admin$TopicCommand$$ensureTopicExists(TopicCommand.scala:503)
>  at 
> kafka.admin.TopicCommand$AdminClientTopicService.alterTopic(TopicCommand.scala:257)
>  at 
> kafka.admin.TopicCommandWithAdminClientTest.testCreateAlterTopicWithRackAware(TopicCommandWithAdminClientTest.scala:476){quote}
> STDOUT
> {quote}[2020-01-07 17:50:17,384] ERROR [ReplicaFetcher replicaId=1, 
> leaderId=0, fetcherId=0] Error for partition 
> testAlterPartitionCount-MixWA3fYA3-1 at offset 0 
> (kafka.server.ReplicaFetcherThread:76) 
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
> does not host this topic-partition. [2020-01-07 17:50:17,631] ERROR 
> [ReplicaFetcher replicaId=4, leaderId=1, fetcherId=0] Error for partition 
> testAlterPartitionCount-MixWA3fYA3-2 at offset 0 
> (kafka.server.ReplicaFetcherThread:76) 
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
> does not host this topic-partition. [2020-01-07 17:50:24,498] ERROR 
> [ReplicaFetcher replicaId=0, leaderId=3, fetcherId=0] Error for partition 
> testDescribeAtMinIsrPartitions-7B6hJOHCF7-0 at offset 0 
> (kafka.server.ReplicaFetcherThread:76) 
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
> does not host this topic-partition. [2020-01-07 17:50:24,506] ERROR 
> [ReplicaFetcher replicaId=1, leaderId=3, fetcherId=0] Error for partition 
> testDescribeAtMinIsrPartitions-7B6hJOHCF7-0 at offset 0 
> (kafka.server.ReplicaFetcherThread:76) 
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
> does not host this topic-partition. [2020-01-07 17:50:24,506] ERROR 
> [ReplicaFetcher replicaId=5, leaderId=3, fetcherId=0] Error for partition 
> testDescribeAtMinIsrPartitions-7B6hJOHCF7-0 at offset 0 
> (kafka.server.ReplicaFetcherThread:76) 
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
> does not host this topic-partition. [2020-01-07 17:50:24,506] ERROR 
> [ReplicaFetcher replicaId=4, leaderId=3, fetcherId=0] Error for partition 
> testDescribeAtMinIsrPartitions-7B6hJOHCF7-0 at offset 0 
> (kafka.server.ReplicaFetcherThread:76) 
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
> does not host this topic-partition. [2020-01-07 17:50:24,507] ERROR 
> [ReplicaFetcher replicaId=2, leaderId=3, fetcherId=0] Error for partition 
> testDescribeAtMinIsrPartitions-7B6hJOHCF7-0 at offset 0 
> (kafka.server.ReplicaFetcherThread:76) 
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
> does not host this topic-partition. [2020-01-07 17:51:29,090] ERROR 
> [ReplicaFetcher replicaId=1, leaderId=4, fetcherId=0] Error for partition 
> kafka.testTopic1-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) 
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
> does not host this topic-partition. [2020-01-07 17:51:29,204] ERROR 
> [ReplicaFetcher replicaId=1, leaderId=5, fetcherId=0] Error for partition 
> __consumer_offsets-0 at offset 0 (kafka.server.ReplicaFetcherThread:76) 
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
> does not host this topic-partition. [2020-01-07 17:51:29,255] ERROR 
> [ReplicaFetcher replicaId=4, leaderId=0, fetcherId=0] Error for partition 
> __consumer_offsets-1 at offset 0 (kafka.server.ReplicaFetcherThread:76) 
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
> does not host this topic-partition. [2020-01-07 17:52:24,546] ERROR 
> [ReplicaFetcher replicaId=0, leaderId=4, fetcherId=0] Error for partition 
> testCreateAlterTopicWithRackAware-1Ski7jYwdP-2 at offset 0 
> (kafka.server.ReplicaFetcherThread:76) 
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server 
> does not host this topic-partition. [2020-01-

Jenkins build is back to normal : kafka-trunk-jdk8 #4142

2020-01-09 Thread Apache Jenkins Server

See

[jira] [Created] (KAFKA-9392) Document and add test for deleteAcls that match single/multiple resources

2020-01-09 Thread Rajini Sivaram (Jira)

Rajini Sivaram created KAFKA-9392:
-

 Summary: Document and add test for deleteAcls that match 
single/multiple resources
 Key: KAFKA-9392
 URL: https://issues.apache.org/jira/browse/KAFKA-9392
 Project: Kafka
  Issue Type: Task
  Components: security
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 2.5.0


>From PR review of 
>[https://github.com/apache/kafka/pull/7911:|https://github.com/apache/kafka/pull/7911,]

If you do {{Admin.createAcls()}} followed by {{Admin.deleteAcls()}}, if you 
specify the ACL in both cases, you are guaranteed to delete the ACL regardless 
of which broker handles the request. If you use a matching filter that doesn't 
specify the resource pattern for {{deleteAcls}}, then we don't provide that 
guarantee.

We should document this and add a deterministic test for the guaranteed 
behaviour to ensure we don't regress.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-373: Allow users to create delegation tokens for other users

2020-01-09 Thread Viktor Somogyi-Vass

Bumping this in the hope of a vote or additional feedback.

Viktor

On Tue, Dec 3, 2019 at 1:07 PM Viktor Somogyi-Vass 
wrote:

> Hi Folks,
>
> I'd like to bump this once more in the hope of a binding vote or any
> additional feedback.
>
> Thanks,
> Viktor
>
> On Fri, Oct 25, 2019 at 2:24 PM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com> wrote:
>
>> Hi All,
>>
>> Would like to bump this in the hope of one binding vote (or any
>> additional feedback).
>>
>> Thanks,
>> Viktor
>>
>> On Wed, Sep 18, 2019 at 5:25 PM Viktor Somogyi-Vass <
>> viktorsomo...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> Harsha, Ryanne: thanks for the vote!
>>>
>>> I'd like to bump this again as today is the KIP freeze date and there is
>>> still one binding vote needed which I'm hoping to get in order to have this
>>> included in 2.4.
>>>
>>> Thanks,
>>> Viktor
>>>
>>> On Tue, Sep 17, 2019 at 1:18 AM Ryanne Dolan 
>>> wrote:
>>>
 +1 non-binding

 Ryanne

 On Mon, Sep 16, 2019, 5:11 PM Harsha Ch  wrote:

 > +1 (binding). Thanks for the KIP Viktor
 >
 > Thanks,
 >
 > Harsha
 >
 > On Mon, Sep 16, 2019 at 3:02 AM, Viktor Somogyi-Vass <
 > viktorsomo...@gmail.com > wrote:
 >
 > >
 > >
 > >
 > > Hi All,
 > >
 > >
 > >
 > > I'd like to bump this again in order to get some more binding votes
 > and/or
 > > feedback in the hope we can push this in for 2.4.
 > >
 > >
 > >
 > > Thank you Manikumar, Gabor and Ryanne so far for the votes! (the
 last two
 > > were on the discussion thread after starting the vote but I think it
 > still
 > > counts :) )
 > >
 > >
 > >
 > > Thanks,
 > > Viktor
 > >
 > >
 > >
 > > On Wed, Aug 21, 2019 at 1:44 PM Manikumar < manikumar. reddy@
 gmail.
 > com (
 > > manikumar.re...@gmail.com ) > wrote:
 > >
 > >
 > >>
 > >>
 > >> Hi,
 > >>
 > >>
 > >>
 > >> +1 (binding).
 > >>
 > >>
 > >>
 > >> Thanks for the updated KIP. LGTM.
 > >>
 > >>
 > >>
 > >> Thanks,
 > >> Manikumar
 > >>
 > >>
 > >>
 > >> On Tue, Aug 6, 2019 at 3:14 PM Viktor Somogyi-Vass < viktorsomogyi@
 > gmail.
 > >> com ( viktorsomo...@gmail.com ) >
 > >> wrote:
 > >>
 > >>
 > >>>
 > >>>
 > >>> Hi All,
 > >>>
 > >>>
 > >>>
 > >>> Bumping this, I'd be happy to get some additional feedback and/or
 > votes.
 > >>>
 > >>>
 > >>>
 > >>> Thanks,
 > >>> Viktor
 > >>>
 > >>>
 > >>>
 > >>> On Wed, Jul 31, 2019 at 11:04 AM Viktor Somogyi-Vass <
 viktorsomogyi@
 > gmail.
 > >>> com ( viktorsomo...@gmail.com ) > wrote:
 > >>>
 > >>>
 > 
 > 
 >  Hi All,
 > 
 > 
 > 
 >  I'd like to start a vote on this KIP.
 > 
 > 
 > >>>
 > >>>
 > >>
 > >>
 > >>
 > >> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/
 > KIP-373%3A+Allow+users+to+create+delegation+tokens+for+other+users
 > >> (
 > >>
 >
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-373%3A+Allow+users+to+create+delegation+tokens+for+other+users
 > >> )
 > >>
 > >>
 > >>>
 > 
 > 
 >  To summarize it: the proposed feature would allow users (usually
 >  superusers) to create delegation tokens for other users. This is
 > 
 > 
 > >>>
 > >>>
 > >>>
 > >>> especially
 > >>>
 > >>>
 > 
 > 
 >  helpful in Spark where the delegation token created this way can
 be
 >  distributed to workers.
 > 
 > 
 > 
 >  I'd be happy to receive any votes or additional feedback.
 > 
 > 
 > 
 >  Viktor
 > 
 > 
 > >>>
 > >>>
 > >>
 > >>
 > >
 > >
 > >

>>>

[jira] [Resolved] (KAFKA-9187) kafka.api.SaslGssapiSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl

2020-01-09 Thread Rajini Sivaram (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-9187.
---
Fix Version/s: 2.4.0
   Resolution: Duplicate

> kafka.api.SaslGssapiSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
> --
>
> Key: KAFKA-9187
> URL: https://issues.apache.org/jira/browse/KAFKA-9187
> Project: Kafka
>  Issue Type: Test
>  Components: core
>Reporter: Bill Bejeck
>Priority: Major
>  Labels: flaky-test
> Fix For: 2.4.0
>
>
> Failed in [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/26593/]
>  
> {noformat}
> Error Messageorg.scalatest.exceptions.TestFailedException: Consumed 0 records 
> before timeout instead of the expected 1 
> recordsStacktraceorg.scalatest.exceptions.TestFailedException: Consumed 0 
> records before timeout instead of the expected 1 records
>   at 
> org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:530)
>   at 
> org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389)
>   at org.scalatest.Assertions$class.fail(Assertions.scala:1091)
>   at org.scalatest.Assertions$.fail(Assertions.scala:1389)
>   at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:842)
>   at kafka.utils.TestUtils$.pollRecordsUntilTrue(TestUtils.scala:793)
>   at 
> kafka.utils.TestUtils$.pollUntilAtLeastNumRecords(TestUtils.scala:1334)
>   at kafka.utils.TestUtils$.consumeRecords(TestUtils.scala:1343)
>   at 
> kafka.api.EndToEndAuthorizationTest.consumeRecords(EndToEndAuthorizationTest.scala:530)
>   at 
> kafka.api.EndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl(EndToEndAuthorizationTest.scala:369)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>   at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
>   at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
>   at 
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(Reflect

[jira] [Resolved] (KAFKA-9264) Reocurrence: Flaky Test DelegationTokenEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl

2020-01-09 Thread Rajini Sivaram (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-9264.
---
Fix Version/s: 2.4.0
   Resolution: Duplicate

> Reocurrence: Flaky Test 
> DelegationTokenEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
> --
>
> Key: KAFKA-9264
> URL: https://issues.apache.org/jira/browse/KAFKA-9264
> Project: Kafka
>  Issue Type: Bug
>  Components: core, security, unit tests
>Reporter: John Roesler
>Priority: Major
>  Labels: flaky-test
> Fix For: 2.4.0
>
>
> This test has failed again for me, so apparently it's not fixed:
> https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/9691/testReport/junit/kafka.api/DelegationTokenEndToEndAuthorizationTest/testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl/
> {noformat}
> Error Message
> org.scalatest.exceptions.TestFailedException: Consumed 0 records before 
> timeout instead of the expected 1 records
> Stacktrace
> org.scalatest.exceptions.TestFailedException: Consumed 0 records before 
> timeout instead of the expected 1 records
>   at 
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530)
>   at 
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529)
>   at 
> org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389)
>   at org.scalatest.Assertions.fail(Assertions.scala:1091)
>   at org.scalatest.Assertions.fail$(Assertions.scala:1087)
>   at org.scalatest.Assertions$.fail(Assertions.scala:1389)
>   at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:842)
>   at kafka.utils.TestUtils$.pollRecordsUntilTrue(TestUtils.scala:796)
>   at 
> kafka.utils.TestUtils$.pollUntilAtLeastNumRecords(TestUtils.scala:1335)
>   at kafka.utils.TestUtils$.consumeRecords(TestUtils.scala:1343)
>   at 
> kafka.api.EndToEndAuthorizationTest.consumeRecords(EndToEndAuthorizationTest.scala:530)
>   at 
> kafka.api.EndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl(EndToEndAuthorizationTest.scala:369)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>   at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
>   at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
>   at 
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProces

Build failed in Jenkins: kafka-trunk-jdk8 #4141

2020-01-09 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-6049: extend Kafka Streams Scala API for cogroup (KIP-150) 
(#7847)

[jason] KAFKA-8988; Replace CreatePartitions Request/Response with automated


--
[...truncated 4.51 MB...]
kafka.tools.DumpLogSegmentsTest > testDumpTimeIndexErrors STARTED

kafka.tools.DumpLogSegmentsTest > testDumpTimeIndexErrors PASSED

kafka.tools.CustomDeserializerTest > checkDeserializerTopicIsNotNull STARTED

kafka.tools.CustomDeserializerTest > checkDeserializerTopicIsNotNull PASSED

kafka.tools.CustomDeserializerTest > checkFormatterCallDeserializerWithHeaders 
STARTED

kafka.tools.CustomDeserializerTest > checkFormatterCallDeserializerWithHeaders 
PASSED

unit.kafka.controller.ControllerContextTest > 
testPartitionFullReplicaAssignmentReturnsEmptyAssignmentIfTopicOrPartitionDoesNotExist
 STARTED

unit.kafka.controller.ControllerContextTest > 
testPartitionFullReplicaAssignmentReturnsEmptyAssignmentIfTopicOrPartitionDoesNotExist
 PASSED

unit.kafka.controller.ControllerContextTest > 
testPartitionReplicaAssignmentForTopicReturnsEmptyMapIfTopicDoesNotExist STARTED

unit.kafka.controller.ControllerContextTest > 
testPartitionReplicaAssignmentForTopicReturnsEmptyMapIfTopicDoesNotExist PASSED

unit.kafka.controller.ControllerContextTest > 
testPartitionReplicaAssignmentForTopicReturnsExpectedReplicaAssignments STARTED

unit.kafka.controller.ControllerContextTest > 
testPartitionReplicaAssignmentForTopicReturnsExpectedReplicaAssignments PASSED

unit.kafka.controller.ControllerContextTest > testReassignTo STARTED

unit.kafka.controller.ControllerContextTest > testReassignTo PASSED

unit.kafka.controller.ControllerContextTest > testPartitionReplicaAssignment 
STARTED

unit.kafka.controller.ControllerContextTest > testPartitionReplicaAssignment 
PASSED

unit.kafka.controller.ControllerContextTest > 
testUpdatePartitionFullReplicaAssignmentUpdatesReplicaAssignment STARTED

unit.kafka.controller.ControllerContextTest > 
testUpdatePartitionFullReplicaAssignmentUpdatesReplicaAssignment PASSED

unit.kafka.controller.ControllerContextTest > 
testPartitionReplicaAssignmentReturnsEmptySeqIfTopicOrPartitionDoesNotExist 
STARTED

unit.kafka.controller.ControllerContextTest > 
testPartitionReplicaAssignmentReturnsEmptySeqIfTopicOrPartitionDoesNotExist 
PASSED

unit.kafka.controller.ControllerContextTest > 
testUpdatePartitionReplicaAssignmentUpdatesReplicaAssignmentOnly STARTED

unit.kafka.controller.ControllerContextTest > 
testUpdatePartitionReplicaAssignmentUpdatesReplicaAssignmentOnly PASSED

unit.kafka.controller.ControllerContextTest > 
testUpdatePartitionReplicaAssignmentUpdatesReplicaAssignmentOnlyAndDoesNotOverwrite
 STARTED

unit.kafka.controller.ControllerContextTest > 
testUpdatePartitionReplicaAssignmentUpdatesReplicaAssignmentOnlyAndDoesNotOverwrite
 PASSED
ERROR: Could not install GRADLE_4_10_3_HOME
java.lang.NullPointerException
at 
hudson.plugins.toolenv.ToolEnvBuildWrapper$1.buildEnvVars(ToolEnvBuildWrapper.java:46)
at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:873)
at hudson.plugins.git.GitSCM.getParamExpandedRepos(GitSCM.java:484)
at 
hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:693)
at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:658)
at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:400)
at hudson.scm.SCM.poll(SCM.java:417)
at hudson.model.AbstractProject._poll(AbstractProject.java:1390)
at hudson.model.AbstractProject.poll(AbstractProject.java:1293)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:603)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:649)
at 
hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
ERROR: Could not install GRADLE_4_10_3_HOME
java.lang.NullPointerException
at 
hudson.plugins.toolenv.ToolEnvBuildWrapper$1.buildEnvVars(ToolEnvBuildWrapper.java:46)
at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:873)
at hudson.plugins.git.GitSCM.getParamExpandedRepos(GitSCM.java:484)
at 
hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:693)
at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:658)
at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:400)
at hudson.scm.SCM.poll(SCM.java:417)
at hudson.model.AbstractProject._poll(AbstractProject.java:1390)
a

Build failed in Jenkins: kafka-2.3-jdk8 #161

2020-01-09 Thread Apache Jenkins Server

See 


Changes:

[matthias] KAFKA-9068: Fix javadoc of Stores.{persistent,inMemory}SessionStore


--
[...truncated 2.98 MB...]
kafka.utils.JsonTest > testLegacyEncodeAsString STARTED

kafka.utils.JsonTest > testLegacyEncodeAsString PASSED

kafka.utils.JsonTest > testEncodeAsBytes STARTED

kafka.utils.JsonTest > testEncodeAsBytes PASSED

kafka.utils.JsonTest > testEncodeAsString STARTED

kafka.utils.JsonTest > testEncodeAsString PASSED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr STARTED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr PASSED

kafka.utils.ZkUtilsTest > testGetSequenceIdMethod STARTED

kafka.utils.ZkUtilsTest > testGetSequenceIdMethod PASSED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testGetAllPartitionsTopicWithoutPartitions STARTED

kafka.utils.ZkUtilsTest > testGetAllPartitionsTopicWithoutPartitions PASSED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testPersistentSequentialPath STARTED

kafka.utils.ZkUtilsTest > testPersistentSequentialPath PASSED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing STARTED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing PASSED

kafka.utils.ZkUtilsTest > testGetLeaderIsrAndEpochForPartition STARTED

kafka.utils.ZkUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.utils.PasswordEncoderTest > testEncoderConfigChange STARTED

kafka.utils.PasswordEncoderTest > testEncoderConfigChange PASSED

kafka.utils.PasswordEncoderTest > testEncodeDecodeAlgorithms STARTED

kafka.utils.PasswordEncoderTest > testEncodeDecodeAlgorithms PASSED

kafka.utils.PasswordEncoderTest > testEncodeDecode STARTED

kafka.utils.PasswordEncoderTest > testEncodeDecode PASSED

kafka.utils.timer.TimerTaskListTest > testAll STARTED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask STARTED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask PASSED

kafka.utils.timer.TimerTest > testTaskExpiration STARTED

kafka.utils.timer.TimerTest > testTaskExpiration PASSED

kafka.utils.ShutdownableThreadTest > testShutdownWhenCalledAfterThreadStart 
STARTED

kafka.utils.ShutdownableThreadTest > testShutdownWhenCalledAfterThreadStart 
PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testUnscheduleProducerTask STARTED

kafka.utils.SchedulerTest > testUnscheduleProducerTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testRestart STARTED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler STARTED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testPeriodicTask STARTED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.json.JsonValueTest > testJsonObjectIterator STARTED

kafka.utils.json.JsonValueTest > testJsonObjectIterator PASSED

kafka.utils.json.JsonValueTest > testDecodeLong STARTED

kafka.utils.json.JsonValueTest > testDecodeLong PASSED

kafka.utils.json.JsonValueTest > testAsJsonObject STARTED

kafka.utils.json.JsonValueTest > testAsJsonObject PASSED

kafka.utils.json.JsonValueTest > testDecodeDouble STARTED

kafka.utils.json.JsonValueTest > testDecodeDouble PASSED

kafka.utils.json.JsonValueTest > testDecodeOption STARTED

kafka.utils.json.JsonValueTest > testDecodeOption PASSED

kafka.utils.json.JsonValueTest > testDecodeString STARTED

kafka.utils.json.JsonValueTest > testDecodeString PASSED

kafka.utils.json.JsonValueTest > testJsonValueToString STARTED

kafka.utils.json.JsonValueTest > testJsonValueToString PASSED

kafka.utils.json.JsonValueTest > testAsJsonObjectOption STARTED

kafka.utils.json.JsonValueTest > testAsJsonObjectOption PASSED

kafka.utils.json.JsonValueTest > testAsJsonArrayOption STARTED

kafka.utils.json.JsonValueTest > testAsJsonArrayOption PASSED

kafka.utils.json.JsonValueTest > testAsJsonArray STARTED

kafka.utils.json.JsonValueTest > testAsJsonArray PASSED

kafka.utils.json.JsonValueTest > testJsonValueHashCode STARTED

kafka.utils.json.JsonValueTest > testJsonValueHashCode PASSED

kafka.utils.json.JsonValueTest > testDecodeInt STARTED

kafka.utils.json.JsonValueTest > testDecodeInt PASSED

kafka.utils.json.JsonValueTest > testDecodeMap STARTED

kafka.utils.json.JsonValueTest > testDecodeMap PASSED

kafka.utils.json.JsonValueTest >

[jira] [Created] (KAFKA-9391) The output of kafka-producer-perf-test.sh is ambiguous

2020-01-09 Thread jiamei xie (Jira)

jiamei xie created KAFKA-9391:
-

 Summary: The output of kafka-producer-perf-test.sh is ambiguous
 Key: KAFKA-9391
 URL: https://issues.apache.org/jira/browse/KAFKA-9391
 Project: Kafka
  Issue Type: Bug
  Components: tools
Reporter: jiamei xie


When running the following command to test producer performance, is records/sec 
in the last line  the average records/sec? If so, maybe avg should be added 
before records/sec to make it more clear.

/home/linux/xjm/kafka/bin/kafka-producer-perf-test.sh --num-records 500 
--topic test6 --producer-props 
bootstrap.servers=wls-x86-hp02:9092,wls-x86-hp04:9092 acks=1 batch.size=8192 
--throughput 500 --record-size 100

2861063 records sent, 572212.6 records/sec (54.57 MB/sec), 380.0 ms avg 
latency, 596.0 ms max latency.
500 records sent, 603354.651864 records/sec (57.54 MB/sec), 379.81 ms avg 
latency, 640.00 ms max latency, 396 ms 50th, 585 ms 95th, 625 ms 99th, 634 ms 
99.9th.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-2.4-jdk8 #124

2020-01-09 Thread Apache Jenkins Server

See 


Changes:

[matthias] KAFKA-9068: Fix javadoc of Stores.{persistent,inMemory}SessionStore


--
[...truncated 2.71 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.strea

46 matches

Mail list logo