Re: [ANNOUNCE] New Kafka PMC Member: John Roesler

[Matthias J. Sax]

Congrats John! Well deserved!

-Matthias

On 8/11/20 2:14 PM, John Roesler wrote:
> Thanks for your kind words, all. This is truly an honor.
> -John
> 
> On Mon, 2020-08-10 at 16:26 -0700, Boyang Chen wrote:
>> Congrats Mr. John!
>>
>>
>> On Mon, Aug 10, 2020 at 3:02 PM Adam Bellemare 
>> wrote:
>>
>>> Congratulations John! You have been an excellent help to me and many
>>> others. I am pleased to see this!
>>>
 On Aug 10, 2020, at 5:54 PM, Bill Bejeck  wrote:

 Congrats!

> On Mon, Aug 10, 2020 at 4:52 PM Guozhang Wang 
>>> wrote:
> Congratulations!
>
>> On Mon, Aug 10, 2020 at 1:11 PM Jun Rao  wrote:
>>
>> Hi, Everyone,
>>
>> John Roesler has been a Kafka committer since Nov. 5, 2019. He has
> remained
>> active in the community since becoming a committer. It's my pleasure to
>> announce that John is now a member of Kafka PMC.
>>
>> Congratulations John!
>>
>> Jun
>> on behalf of Apache Kafka PMC
>>
>
> --
> -- Guozhang
>
> 



signature.asc
Description: OpenPGP digital signature

Jenkins build is back to normal : Kafka » kafka-2.1-jdk8 #3

[Apache Jenkins Server]

See 

Re: need permission to create KIP

[Bill Bejeck]

Done. Thanks for your interest in Apache Kafka

-Bill

On Fri, Aug 14, 2020 at 2:33 PM gyejun choi  wrote:

> I need a permission with this page for "Create KIP"
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>
> wiki userId : whsoul82
>
> thanks
>

Re: [DISCUSS] Apache Kafka 2.7.0 release

[John Roesler]

Thanks, Bill!
-John

On Thu, 2020-08-13 at 15:19 -0700, Ismael Juma wrote:
> Thanks for volunteering Bill. :)
> 
> Ismael
> 
> On Thu, Aug 13, 2020 at 3:13 PM Bill Bejeck  wrote:
> 
> > Hi All,
> > 
> > I'd like to volunteer to be the release manager for our next feature
> > release, 2.7. If there are no objections, I'll send out the release plan
> > soon.
> > 
> > Thanks,
> > Bill Bejeck
> > 

Re: [DISCUSS] KIP-639 Move nodeLevelSensor and storeLevelSensor methods from StreamsMetricsImpl to StreamsMetrics

[John Roesler]

Thanks Mohamed,

I think Bruno raised a good point in the ticket that the
node name is not well known from within the Processor at the
time of init(), so users would basically have to make up a
name to pass into the sensor. Maybe this is ok, but it
doesn't seem too nice.

Offhand, it seems like the options are:

1. To just expose the node name also in ProcessorContext
(such as with a nullable "processorNodeName()" method).

2. To close over the node name in the view of the context
and metrics that we pass to the Processor when we call
init(). The caller of init() is actually the ProcessorNode
itself, so it can easily insert this information into the
metrics before invoking Processor#init().

Offhand, it seems option 2 gives a simpler and better
experience. My concern would be that it might be "too
fancy". Also, if we favor this route, we should reconsider
many of the other arguments to those methods.

Thanks for the KIP!
-John

On Fri, 2020-08-14 at 01:37 +0100, Mohamed Chebbi wrote:
> KIP updated with the comments of Bruno Cardona.
> 
> Le 06/07/2020 à 22:36, Mohamed Chebbi a écrit :
> > Thank Bruno for your review.
> > 
> > Changes was added as you sugested.
> > 
> > Le 06/07/2020 à 14:57, Bruno Cadonna a écrit :
> > > Hi Mohamed,
> > > 
> > > Thank you for the KIP.
> > > 
> > > Comments regarding the KIP wiki:
> > > 
> > > 1. In section "Public Interface", you should state what you want to
> > > change in interface StreamsMetrics. In your case, you want to add two
> > > methods. You can find a good example how to describe this in KIP-444
> > > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%3A+Augment+metrics+for+Kafka+Streams).
> > >  
> > > 
> > > 
> > > 2. In section "Compatibility, Deprecation, and Migration Plan" you
> > > should state if anything is needed to keep backward compatibility.
> > > Since you just want to add two methods to the interface, nothing is
> > > needed. You should describe that under that section.
> > > 
> > > Regarding the KIP content, I left some comments on the corresponding
> > > Jira ticket.
> > > 
> > > Best,
> > > Bruno
> > > 
> > > 
> > > On Sun, Jul 5, 2020 at 3:48 AM Mohamed Chebbi  
> > > wrote:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-639%3A+Move+nodeLevelSensor+and+storeLevelSensor+methods+from+StreamsMetricsImpl+to+StreamsMetrics
> > > >  
> > > > 
> > > > 

Re: [EXTERNAL] Re: Request for access to create KIP

[Boyang Chen]

Done!

On Thu, Aug 13, 2020 at 2:25 PM Koushik Chitta
 wrote:

> Accountid: koushikchitta
>
> Cheers,
> Koushik
>
> -Original Message-
> From: Boyang Chen 
> Sent: Sunday, August 9, 2020 6:02 PM
> To: dev 
> Subject: [EXTERNAL] Re: Request for access to create KIP
>
> Have you created the account already? What's your account id?
>
> On Sat, Aug 8, 2020 at 4:36 PM Koushik Chitta
> 
> wrote:
>
> > Hi Team,
> >
> > Can you please grant me access to create KIP ?
> >
> > Thanks,
> > Koushik
> >
>

[jira] [Created] (KAFKA-10405) Flaky Test org.apache.kafka.streams.integration.PurgeRepartitionTopicIntegrationTest.shouldRestoreState

[Bill Bejeck (Jira)]

Bill Bejeck created KAFKA-10405:
---

 Summary: Flaky Test 
org.apache.kafka.streams.integration.PurgeRepartitionTopicIntegrationTest.shouldRestoreState
 Key: KAFKA-10405
 URL: https://issues.apache.org/jira/browse/KAFKA-10405
 Project: Kafka
  Issue Type: Test
  Components: streams
Reporter: Bill Bejeck


>From build [https://builds.apache.org/job/kafka-pr-jdk14-scala2.13/1979/]

 
{noformat}
org.apache.kafka.streams.integration.PurgeRepartitionTopicIntegrationTest > 
shouldRestoreState FAILED
14:25:19 java.lang.AssertionError: Condition not met within timeout 6. 
Repartition topic 
restore-test-KSTREAM-AGGREGATE-STATE-STORE-02-repartition not purged 
data after 6 ms.
14:25:19 at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:26)
14:25:19 at 
org.apache.kafka.test.TestUtils.lambda$waitForCondition$6(TestUtils.java:401)
14:25:19 at 
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:449)
14:25:19 at 
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:417)
14:25:19 at 
org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:398)
14:25:19 at 
org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:388)
14:25:19 at 
org.apache.kafka.streams.integration.PurgeRepartitionTopicIntegrationTest.shouldRestoreState(PurgeRepartitionTopicIntegrationTest.java:206){noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

need permission to create KIP

[gyejun choi]

I need a permission with this page for "Create KIP"
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals

wiki userId : whsoul82

thanks

Re: [DISCUSSION] Upgrade system tests to python 3

[Ismael Juma]

+1

On Fri, Aug 14, 2020 at 7:42 AM John Roesler  wrote:

> Thanks Nikolay,
>
> No objection. This would be very nice to have.
>
> Thanks,
> John
>
> On Fri, Aug 14, 2020, at 09:18, Nikolay Izhikov wrote:
> > Hello.
> >
> > > If anyone's interested in porting it to Python 3 it would be a good
> change.
> >
> > I’ve created a ticket [1] to upgrade system tests to python3.
> > Does someone have any additional inputs or objections for this change?
> >
> > [1] https://issues.apache.org/jira/browse/KAFKA-10402
> >
> >
> > > 1 июля 2020 г., в 00:26, Gokul Ramanan Subramanian <
> gokul24...@gmail.com> написал(а):
> > >
> > > Thanks Colin.
> > >
> > > While at the subject of system tests, there are a few times I see tests
> > > timed out (even on a large machine such as m5.4xlarge EC2 with Linux).
> Are
> > > there any knobs that system tests provide to control timeouts /
> throughputs
> > > across all tests?
> > > Thanks.
> > >
> > > On Tue, Jun 30, 2020 at 6:32 PM Colin McCabe 
> wrote:
> > >
> > >> Ducktape runs on Python 2.  You can't use it with Python 3, as you are
> > >> trying to do here.
> > >>
> > >> If anyone's interested in porting it to Python 3 it would be a good
> change.
> > >>
> > >> Otherwise, using docker as suggested here seems to be the best way to
> go.
> > >>
> > >> best,
> > >> Colin
> > >>
> > >> On Mon, Jun 29, 2020, at 02:14, Gokul Ramanan Subramanian wrote:
> > >>> Hi.
> > >>>
> > >>> Has anyone had luck running Kafka system tests on a Mac. I have a
> MacOS
> > >>> Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the command
> > >>> *ducktape tests/kafkatest/tests* yields the following error, making
> it
> > >> look
> > >>> like some Python incompatibility issue.
> > >>>
> > >>> $ ducktape tests/kafkatest/tests
> > >>> Traceback (most recent call last):
> > >>>  File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape", line 11,
> in
> > >>> 
> > >>>load_entry_point('ducktape', 'console_scripts', 'ducktape')()
> > >>>  File
> > >>>
> > >>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> > >>> line 487, in load_entry_point
> > >>>return get_distribution(dist).load_entry_point(group, name)
> > >>>  File
> > >>>
> > >>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> > >>> line 2728, in load_entry_point
> > >>>return ep.load()
> > >>>  File
> > >>>
> > >>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> > >>> line 2346, in load
> > >>>return self.resolve()
> > >>>  File
> > >>>
> > >>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> > >>> line 2352, in resolve
> > >>>module = __import__(self.module_name, fromlist=['__name__'],
> > >>> level=0)
> > >>>  File
> > >>>
> > >>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
> > >>> line 127
> > >>>print "parameters are not valid json: " + str(e.message)
> > >>>  ^
> > >>> SyntaxError: invalid syntax
> > >>>
> > >>> I followed the instructions in tests/README.md to setup a cluster of
> 9
> > >>> worker machines. That worked well. When I ran *python setup.py
> develop*
> > >> to
> > >>> install the necessary dependencies (including ducktape), I got
> similar
> > >>> errors to above, but the overall command completed successfully.
> > >>>
> > >>> Any help appreciated.
> > >>>
> > >>> Thanks.
> > >>>
> > >>
> >
> >
>

Build failed in Jenkins: Kafka » kafka-2.0-jdk8 #2

[Apache Jenkins Server]

See 


Changes:


--
Started by user Ismael Juma
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H29 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress -- https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress -- https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git rev-parse refs/remotes/origin/2.0^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/2.0^{commit} # timeout=10
Checking out Revision 8c4d78ee2a85760fc73062b590c732a346d80502 
(refs/remotes/origin/2.0)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8c4d78ee2a85760fc73062b590c732a346d80502 # timeout=10
Commit message: "MINOR: Upgrade ducktape to 0.7.8 (#8879)"
 > git rev-list --no-walk 8c4d78ee2a85760fc73062b590c732a346d80502 # timeout=10
[kafka-2.0-jdk8] $ /bin/sh -xe /tmp/jenkins4402437334660952235.sh
+ rm -rf 
[kafka-2.0-jdk8] $ /bin/sh -xe /tmp/jenkins8305903453482636948.sh
+ export GRADLE_OPTS=-Xmx1024m
+ ./gradlew --no-daemon -PmaxParallelForks=1 
-PtestLoggingEvents=started,passed,skipped,failed -PxmlFindBugsReport=true 
clean testAll
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:31)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:108)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:61)
Caused by: java.lang.NoClassDefFoundError: 
org/gradle/api/internal/classpath/ModuleRegistry
at org.gradle.launcher.GradleMain.main(GradleMain.java:23)
... 7 more
Caused by: java.lang.ClassNotFoundException: 
org.gradle.api.internal.classpath.ModuleRegistry
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 8 more
Build step 'Execute shell' marked build as failure
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?

[jira] [Created] (KAFKA-10404) Flaky Test kafka.api.SaslSslConsumerTest.testCoordinatorFailover

[Bill Bejeck (Jira)]

Bill Bejeck created KAFKA-10404:
---

 Summary: Flaky Test 
kafka.api.SaslSslConsumerTest.testCoordinatorFailover
 Key: KAFKA-10404
 URL: https://issues.apache.org/jira/browse/KAFKA-10404
 Project: Kafka
  Issue Type: Test
  Components: core, unit tests
Reporter: Bill Bejeck


>From build [https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3829/]

 
{noformat}
kafka.api.SaslSslConsumerTest > testCoordinatorFailover FAILED
11:27:15 java.lang.AssertionError: expected: but 
was:
11:27:15 at org.junit.Assert.fail(Assert.java:89)
11:27:15 at org.junit.Assert.failNotEquals(Assert.java:835)
11:27:15 at org.junit.Assert.assertEquals(Assert.java:120)
11:27:15 at org.junit.Assert.assertEquals(Assert.java:146)
11:27:15 at 
kafka.api.AbstractConsumerTest.sendAndAwaitAsyncCommit(AbstractConsumerTest.scala:195)
11:27:15 at 
kafka.api.AbstractConsumerTest.ensureNoRebalance(AbstractConsumerTest.scala:302)
11:27:15 at 
kafka.api.BaseConsumerTest.testCoordinatorFailover(BaseConsumerTest.scala:76)
11:27:15 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:27:15 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
11:27:15 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:27:15 at java.lang.reflect.Method.invoke(Method.java:498)
11:27:15 at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
11:27:15 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
11:27:15 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
11:27:15 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
11:27:15 at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
11:27:15 at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
11:27:15 at 
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
11:27:15 at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
11:27:15 at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
11:27:15 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
11:27:15 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
11:27:15 at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
11:27:15 at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
11:27:15 at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
11:27:15 at 
org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
11:27:15 at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
11:27:15 at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
11:27:15 at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
11:27:15 at 
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
11:27:15 at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
11:27:15 at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
11:27:15 at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
11:27:15 at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
11:27:15 at 
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
11:27:15 at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
11:27:15 at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
11:27:15 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:27:15 at java.lang.reflect.Method.invoke(Method.java:498)
11:27:15 at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
11:27:15 at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
11:27:15 at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
11:27:15 at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
11:27:15 at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
11:27:15 at 

[jira] [Created] (KAFKA-10403) Replace scala collection by java collection in generating MBeans attributes

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-10403:
--

 Summary: Replace scala collection by java collection in generating 
MBeans attributes
 Key: KAFKA-10403
 URL: https://issues.apache.org/jira/browse/KAFKA-10403
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


It seems to me the metrics is a "kind" of public interface so users should be 
able to access metrics of kafka server without scala library.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSSION] KIP-619: Add internal topic creation support

[David Arthur]

Cheng,

Can you clarify a bit more what the difference is between regular topics
and internal topics (excluding  __consumer_offsets and
__transaction_state)? Reading your last message, if internal topics
(excluding the two) can be created, deleted, produced to, consumed from,
added to transactions, I'm failing to see what is different about them. Is
it simply that they are marked as "internal" so the application can treat
them differently?


In the "Compatibility, Deprecation, and Migration" section, we should
detail how users can overcome this incompatibility (i.e., changing the
config name on their topic and changing their application logic if
necessary).


Should we consider adding any configs to constrain the min isr and
replication factor for internal topics? If a topic is really internal and
fundamentally required for an application to function, it might need a more
stringent replication config. Our existing internal topics have their own
configs in server.properties with a comment saying as much.


Thanks!
David



On Tue, Jul 7, 2020 at 1:40 PM Cheng Tan  wrote:

> Hi Colin,
>
>
> Thanks for the comments. I’ve modified the KIP accordingly.
>
> > I think we need to understand which of these limitations we will carry
> forward and which we will not.  We also have the option of putting
> limitations just on consumer offsets, but not on other internal topics.
>
>
> In the proposal, I added details about this. I agree that cluster admin
> should use ACLs to apply the restrictions.
> Internal topic creation will be allowed.
> Internal topic deletion will be allowed except for` __consumer_offsets`
> and `__transaction_state`.
> Producing to internal topic partitions other than `__consumer_offsets` and
> `__transaction_state` will be allowed.
> Adding internal topic partitions to transactions will be allowed.
> > I think there are a fair number of compatibility concerns.  What's the
> result if someone tries to create a topic with the configuration internal =
> true right now?  Does it fail?  If not, that seems like a potential problem.
>
> I also added this compatibility issue in the "Compatibility, Deprecation,
> and Migration Plan" section.
>
> Please feel free to make any suggestions or comments regarding to my
> latest proposal. Thanks.
>
>
> Best, - Cheng Tan
>
>
>
>
>
>
> > On Jun 15, 2020, at 11:18 AM, Colin McCabe  wrote:
> >
> > Hi Cheng,
> >
> > The link from the main KIP page is an "edit link" meaning that it drops
> you into the editor for the wiki page.  I think the link you meant to use
> is a "view link" that will just take you to view the page.
> >
> > In general I'm not sure what I'm supposed to take away from the large
> UML diagram in the KIP.  This is just a description of the existing code,
> right?  Seems like we should remove this.
> >
> > I'm not sure why the controller classes are featured here since as far
> as I can tell, the controller doesn't need to care if a topic is internal.
> >
> >> Kafka and its upstream applications treat internal topics differently
> from
> >> non-internal topics. For example:
> >> * Kafka handles topic creation response errors differently for internal
> topics
> >> * Internal topic partitions cannot be added to a transaction
> >> * Internal topic records cannot be deleted
> >> * Appending to internal topics might get rejected
> >
> > I think we need to understand which of these limitations we will carry
> forward and which we will not.  We also have the option of putting
> limitations just on consumer offsets, but not on other internal topics.
> >
> > Taking it one by one:
> >
> >> * Kafka handles topic creation response errors differently for internal
> topics.
> >
> > Hmm.  Kafka doesn't currently allow you to create internal topics, so
> the difference here is that you always fail, right?  Or is there something
> else more subtle here?  Like do we specifically prevent you from creating
> topics named __consumer_offsets or something?  We need to spell this all
> out in the KIP.
> >
> >> * Internal topic partitions cannot be added to a transaction
> >
> > I don't think we should carry this limitation forward, or if we do, we
> should only do it for consumer-offsets.  Does anyone know why this
> limitation exists?
> >
> >> * Internal topic records cannot be deleted
> >
> > This seems like something that should be handled by ACLs rather than by
> treating internal topics specially.
> >
> >> * Appending to internal topics might get rejected
> >
> > We clearly need to use ACLs here rather than rejecting appends.
> Otherwise, how will external systems like KSQL, streams, etc. use this
> feature?  This is the kind of information we need to have in the KIP.
> >
> >> Public Interfaces
> >> 2. KafkaZkClient will have a new method getInternalTopics() which
> >> returns a set of internal topic name strings.
> >
> > KafkaZkClient isn't a public interface, so it doesn't need to be
> described here.
> >
> >> There are no compatibility concerns in this KIP.
> >
> 

Re: [DISCUSSION] Upgrade system tests to python 3

[John Roesler]

Thanks Nikolay,

No objection. This would be very nice to have. 

Thanks,
John 

On Fri, Aug 14, 2020, at 09:18, Nikolay Izhikov wrote:
> Hello.
> 
> > If anyone's interested in porting it to Python 3 it would be a good change.
> 
> I’ve created a ticket [1] to upgrade system tests to python3.
> Does someone have any additional inputs or objections for this change?
> 
> [1] https://issues.apache.org/jira/browse/KAFKA-10402
> 
> 
> > 1 июля 2020 г., в 00:26, Gokul Ramanan Subramanian  
> > написал(а):
> > 
> > Thanks Colin.
> > 
> > While at the subject of system tests, there are a few times I see tests
> > timed out (even on a large machine such as m5.4xlarge EC2 with Linux). Are
> > there any knobs that system tests provide to control timeouts / throughputs
> > across all tests?
> > Thanks.
> > 
> > On Tue, Jun 30, 2020 at 6:32 PM Colin McCabe  wrote:
> > 
> >> Ducktape runs on Python 2.  You can't use it with Python 3, as you are
> >> trying to do here.
> >> 
> >> If anyone's interested in porting it to Python 3 it would be a good change.
> >> 
> >> Otherwise, using docker as suggested here seems to be the best way to go.
> >> 
> >> best,
> >> Colin
> >> 
> >> On Mon, Jun 29, 2020, at 02:14, Gokul Ramanan Subramanian wrote:
> >>> Hi.
> >>> 
> >>> Has anyone had luck running Kafka system tests on a Mac. I have a MacOS
> >>> Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the command
> >>> *ducktape tests/kafkatest/tests* yields the following error, making it
> >> look
> >>> like some Python incompatibility issue.
> >>> 
> >>> $ ducktape tests/kafkatest/tests
> >>> Traceback (most recent call last):
> >>>  File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape", line 11, in
> >>> 
> >>>load_entry_point('ducktape', 'console_scripts', 'ducktape')()
> >>>  File
> >>> 
> >> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>> line 487, in load_entry_point
> >>>return get_distribution(dist).load_entry_point(group, name)
> >>>  File
> >>> 
> >> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>> line 2728, in load_entry_point
> >>>return ep.load()
> >>>  File
> >>> 
> >> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>> line 2346, in load
> >>>return self.resolve()
> >>>  File
> >>> 
> >> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>> line 2352, in resolve
> >>>module = __import__(self.module_name, fromlist=['__name__'],
> >>> level=0)
> >>>  File
> >>> 
> >> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
> >>> line 127
> >>>print "parameters are not valid json: " + str(e.message)
> >>>  ^
> >>> SyntaxError: invalid syntax
> >>> 
> >>> I followed the instructions in tests/README.md to setup a cluster of 9
> >>> worker machines. That worked well. When I ran *python setup.py develop*
> >> to
> >>> install the necessary dependencies (including ducktape), I got similar
> >>> errors to above, but the overall command completed successfully.
> >>> 
> >>> Any help appreciated.
> >>> 
> >>> Thanks.
> >>> 
> >> 
> 
>

[DISCUSSION] Upgrade system tests to python 3

[Nikolay Izhikov]

Hello.

> If anyone's interested in porting it to Python 3 it would be a good change.

I’ve created a ticket [1] to upgrade system tests to python3.
Does someone have any additional inputs or objections for this change?

[1] https://issues.apache.org/jira/browse/KAFKA-10402


> 1 июля 2020 г., в 00:26, Gokul Ramanan Subramanian  
> написал(а):
> 
> Thanks Colin.
> 
> While at the subject of system tests, there are a few times I see tests
> timed out (even on a large machine such as m5.4xlarge EC2 with Linux). Are
> there any knobs that system tests provide to control timeouts / throughputs
> across all tests?
> Thanks.
> 
> On Tue, Jun 30, 2020 at 6:32 PM Colin McCabe  wrote:
> 
>> Ducktape runs on Python 2.  You can't use it with Python 3, as you are
>> trying to do here.
>> 
>> If anyone's interested in porting it to Python 3 it would be a good change.
>> 
>> Otherwise, using docker as suggested here seems to be the best way to go.
>> 
>> best,
>> Colin
>> 
>> On Mon, Jun 29, 2020, at 02:14, Gokul Ramanan Subramanian wrote:
>>> Hi.
>>> 
>>> Has anyone had luck running Kafka system tests on a Mac. I have a MacOS
>>> Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the command
>>> *ducktape tests/kafkatest/tests* yields the following error, making it
>> look
>>> like some Python incompatibility issue.
>>> 
>>> $ ducktape tests/kafkatest/tests
>>> Traceback (most recent call last):
>>>  File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape", line 11, in
>>> 
>>>load_entry_point('ducktape', 'console_scripts', 'ducktape')()
>>>  File
>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>> line 487, in load_entry_point
>>>return get_distribution(dist).load_entry_point(group, name)
>>>  File
>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>> line 2728, in load_entry_point
>>>return ep.load()
>>>  File
>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>> line 2346, in load
>>>return self.resolve()
>>>  File
>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>> line 2352, in resolve
>>>module = __import__(self.module_name, fromlist=['__name__'],
>>> level=0)
>>>  File
>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
>>> line 127
>>>print "parameters are not valid json: " + str(e.message)
>>>  ^
>>> SyntaxError: invalid syntax
>>> 
>>> I followed the instructions in tests/README.md to setup a cluster of 9
>>> worker machines. That worked well. When I ran *python setup.py develop*
>> to
>>> install the necessary dependencies (including ducktape), I got similar
>>> errors to above, but the overall command completed successfully.
>>> 
>>> Any help appreciated.
>>> 
>>> Thanks.
>>> 
>> 

[jira] [Created] (KAFKA-10402) Upgrade python version in system tests

[Nikolay Izhikov (Jira)]

Nikolay Izhikov created KAFKA-10402:
---

 Summary: Upgrade python version in system tests
 Key: KAFKA-10402
 URL: https://issues.apache.org/jira/browse/KAFKA-10402
 Project: Kafka
  Issue Type: Improvement
Reporter: Nikolay Izhikov
Assignee: Nikolay Izhikov


Currently, system tests using python 2 which is outdated and not supported.

Since all dependency of system tests including ducktape supporting python 3 we 
can migrate system tests to python3.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10401) GroupMetadataManager ignores current_state_timestamp field for GROUP_METADATA_VALUE_SCHEMA_V3

[Marek (Jira)]

Marek created KAFKA-10401:
-

 Summary: GroupMetadataManager ignores current_state_timestamp 
field for GROUP_METADATA_VALUE_SCHEMA_V3
 Key: KAFKA-10401
 URL: https://issues.apache.org/jira/browse/KAFKA-10401
 Project: Kafka
  Issue Type: Bug
  Components: offset manager
Affects Versions: 2.5.1, 2.6.0, 2.4.1, 2.2.2, 2.1.1
Reporter: Marek


While reading group metadata information from ByteBuffer GroupMetadataManager 
reads current_state_timestamp only for group schema version 2. For all other 
versions this value is set to "None". 

Piece of code responsible for bug:

[https://github.com/apache/kafka/blob/2.6.0/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1412]

Restarting kafka forces GroupMetadataManager manager to reload group metadata 
from file basically causing 
[KIP-211|[https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals?src=breadcrumbs-parent]]
  to be only applicable for schema version 2. 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : Kafka » kafka-trunk-jdk11 #11

[Apache Jenkins Server]

See