[jira] [Resolved] (KAFKA-9996) upgrade zookeeper to 3.5.8 to address security vulnerabilities

[Ismael Juma (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-9996.

Fix Version/s: 2.5.1
   2.4.2
   2.6.0
   Resolution: Fixed

> upgrade zookeeper to 3.5.8 to address security vulnerabilities
> --
>
> Key: KAFKA-9996
> URL: https://issues.apache.org/jira/browse/KAFKA-9996
> Project: Kafka
>  Issue Type: Bug
>  Components: packaging
>Affects Versions: 2.5.0
>Reporter: Emanuele Maccherani
>Assignee: Ismael Juma
>Priority: Major
> Fix For: 2.6.0, 2.4.2, 2.5.1
>
>
> Kafka is now using zookeeper 3.5.7, which is affected by CVE-2020-8840 and 
> CVE-2020-9488. Those 2 are resolved in 3.5.8.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-623: Add "internal-topics" option to streams application reset tool

[Boyang Chen]

Sounds good, thank you! Moreover, I do feel it would be great to integrate
some of the motivation stuff into the option description, but we could
discuss more in the PR.

On Sat, Jul 18, 2020 at 3:24 AM Joel Wee  wrote:

> I agree Boyang, we should be able to specify both `—internal-topics` and
> `—dry-run` at the same time and this should display the topics that will be
> deleted. What I was trying to communicate in the description is that if you
> want to see all the topics that are valid as input into the option, then
> first do a dry-run without the `—internal-topics` option. I’ve rephrased it
> slightly which hopefully makes it clearer.
>
> --internal-topics   Comma-separated list of internal
> topics
>to delete. Must be a subset of the
>internal topics marked for deletion
> by
>the default behaviour (do a dry-run
> without this
>option to view these topics).
>
>
> - Joel
>
> On 11 Jul 2020, at 8:41 AM, Boyang Chen  > wrote:
>
> Thanks for the update!
>
> On Fri, Jul 3, 2020 at 3:18 PM Joel Wee  joel@outlook.com>> wrote:
>
> Thanks all for voting!
>
> I am closing the vote as accepted with three binding +1 votes (Boyang,
> Guozhang, John).
>
> Thanks John for the suggestion. I think that makes sense. I have updated
> the KIP to say that only topics flagged as internal by the tool can be
> deleted, and have also rephrased the option description to make this
> clearer:
>
> --internal-topics   Comma-separated list of internal
> topics
>to delete. Must be a subset of
> the
>internal topics marked for
> deletion by
>the default behaviour. To view
> these
>topics, do a dry-run without
> this
>option.
>
> Hmm, could you explain why we couldn't run with both --internal-topics and
> --dry-run? To me we could either display the exact set of internal topics
> specified to be deleted, or reject the same way as we are doing an actual
> reset.
>
>
> Thanks Guozhang for the suggestion. The updated option description should
> address your first point. By the “topology description” script are you
> referring to bin/kafka-topics.sh? It currently has the option to display
> all topics (including internal topics). Were you thinking about adding
> something to view just internal topics?
>
> Thanks Bruno for the suggestion. I will close this vote for now, and we
> can continue the discussion on another thread. (P.S. apologies for
> misspelling your name previously)
>
> - Joel
>
> On 1 Jul 2020, at 3:04 AM, Boyang Chen  reluctanthero...@gmail.com>>
> wrote:
>
> Hey Bruno,
>
> I agree adding a prompt would be a nice precaution, but it is not
> backward
> compatible as you suggested and could make the automation harder to
> achieve.
>
> If you want, we may consider starting a separate ticket to discuss
> whether
> adding a prompt to let users be aware of the topics that are about to
> delete. However, this is also inverting the assumptions we made about
> `--dry-run` mode, which would become useless to me once we are adding a
> prompt asking users whether they want to remove these topics completely,
> or
> do nothing at all.
>
> Back to KIP-623, I think this discussion could be held in orthogonal,
> which
> applies to more general considerations of reducing human errors, etc.
>
> Boyang
>
> On Tue, Jun 30, 2020 at 12:55 AM Bruno Cadonna  br...@confluent.io>>
> wrote:
>
> Hi,
>
> I have already brought this up in the discussion thread.
>
> Should we not run a dry-run in any case to avoid inadvertently
> deleting topics of other applications?
>
> I know it is a backward incompatible change if users use it in
> scripts, but I think it is still worth discussing it. I would to hear
> what committers think about it.
>
> Best,
> Bruno
>
> On Tue, Jun 30, 2020 at 5:33 AM Boyang Chen  
>
> wrote:
>
> Thanks John for the great suggestion. +1 for enforcing the prefix check
> for
> the `--internal-topics` list.
>
> On Mon, Jun 29, 2020 at 5:11 PM John Roesler  vvcep...@apache.org>>
> wrote:
>
> Oh, I meant to say, if that’s the case, then I’m +1 (binding) also :)
>
> Thanks again,
> John
>
> On Mon, Jun 29, 2020, at 19:09, John Roesler wrote:
> Thanks for the KIP, Joel!
>
> It seems like a good pattern to document would be to first run with
> —dry-run and without —internal-topics to list all potential topics,
> and
> then to use —internal-topics if needed to limit the internal topics
> to
> delete.
>
> Just to make sure, would we have a sanity check to forbid including
> arbitrary topics? I.e., it seems like —internal-topics should require
> all the topics to be prefixed with the app id. Is that right?

[jira] [Created] (KAFKA-10293) fix flaky streams/streams_eos_test.py

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-10293:
--

 Summary: fix flaky streams/streams_eos_test.py
 Key: KAFKA-10293
 URL: https://issues.apache.org/jira/browse/KAFKA-10293
 Project: Kafka
  Issue Type: Bug
Reporter: Chia-Ping Tsai


{quote}
Module: kafkatest.tests.streams.streams_eos_test
Class:  StreamsEosTest
Method: test_failure_and_recovery_complex
Arguments:
{
  "processing_guarantee": "exactly_once"
}
{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10292) fix flaky streams/streams_broker_bounce_test.py

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-10292:
--

 Summary: fix flaky streams/streams_broker_bounce_test.py
 Key: KAFKA-10292
 URL: https://issues.apache.org/jira/browse/KAFKA-10292
 Project: Kafka
  Issue Type: Bug
Reporter: Chia-Ping Tsai


{quote}
Module: kafkatest.tests.streams.streams_broker_bounce_test
Class:  StreamsBrokerBounceTest
Method: test_broker_type_bounce
Arguments:
{
  "broker_type": "leader",
  "failure_mode": "clean_bounce",
  "num_threads": 1,
  "sleep_time_secs": 120
}
{quote}

{quote}
Module: kafkatest.tests.streams.streams_broker_bounce_test
Class:  StreamsBrokerBounceTest
Method: test_broker_type_bounce
Arguments:
{
  "broker_type": "controller",
  "failure_mode": "hard_shutdown",
  "num_threads": 3,
  "sleep_time_secs": 120
}
{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : kafka-trunk-jdk8 #4723

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk11 #1650

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Improved code quality for various files. (#9037)


--
[...truncated 6.40 MB...]
org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:compileTestJava
> Task :streams:upgrade-system-tests-0101:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:testClasses
> Task :streams:upgrade-system-tests-0101:checkstyleTest
> Task :streams:upgrade-system-tests-0101:spotbugsMain 

[jira] [Created] (KAFKA-10291) fix flaky tools/log4j_appender_test.py

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-10291:
--

 Summary: fix flaky tools/log4j_appender_test.py
 Key: KAFKA-10291
 URL: https://issues.apache.org/jira/browse/KAFKA-10291
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai


{quote}
Module: kafkatest.tests.tools.log4j_appender_test
Class:  Log4jAppenderTest
Method: test_log4j_appender
Arguments:
{
  "security_protocol": "SSL"
}
{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk14 #299

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Improved code quality for various files. (#9037)


--
[...truncated 3.20 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

[jira] [Created] (KAFKA-10290) fix flaky core/compatibility_test_new_broker_test.py

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-10290:
--

 Summary: fix flaky core/compatibility_test_new_broker_test.py
 Key: KAFKA-10290
 URL: https://issues.apache.org/jira/browse/KAFKA-10290
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai


{quote}
Module: kafkatest.tests.core.compatibility_test_new_broker_test
Class:  ClientCompatibilityTestNewBroker
Method: test_compatibility
Arguments:
{
  "compression_types": [
"none"
  ],
  "consumer_version": "1.0.2",
  "producer_version": "1.0.2",
  "timestamp_type": "CreateTime"
}
{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10289) fix flaky connect/connect_distributed_test.py

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-10289:
--

 Summary: fix flaky connect/connect_distributed_test.py
 Key: KAFKA-10289
 URL: https://issues.apache.org/jira/browse/KAFKA-10289
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai


{quote}
Module: kafkatest.tests.connect.connect_distributed_test
Class:  ConnectDistributedTest
Method: test_broker_compatibility
Arguments:
{
  "auto_create_topics": false,
  "broker_version": "0.10.1.1",
  "connect_protocol": "compatible",
  "security_protocol": "PLAINTEXT"
}
{quote}


{quote}
Module: kafkatest.tests.connect.connect_distributed_test
Class:  ConnectDistributedTest
Method: test_broker_compatibility
Arguments:
{
  "auto_create_topics": false,
  "broker_version": "2.1.1",
  "connect_protocol": "compatible",
  "security_protocol": "PLAINTEXT"
}
{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10288) fix flaky client/client_compatibility_features_test.py

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-10288:
--

 Summary: fix flaky client/client_compatibility_features_test.py
 Key: KAFKA-10288
 URL: https://issues.apache.org/jira/browse/KAFKA-10288
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai


{quote}
Module: kafkatest.tests.client.client_compatibility_features_test
Class:  ClientCompatibilityFeaturesTest
Method: run_compatibility_test
Arguments:
{
  "broker_version": "0.10.0.1"
}
{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10287) fix flaky streams/streams_standby_replica_test.py

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-10287:
--

 Summary: fix flaky streams/streams_standby_replica_test.py
 Key: KAFKA-10287
 URL: https://issues.apache.org/jira/browse/KAFKA-10287
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


{quote}
Module: kafkatest.tests.streams.streams_standby_replica_test
Class:  StreamsStandbyTask
Method: test_standby_tasks_rebalance
{quote}

It pass occasionally on my local.  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-623: Add "internal-topics" option to streams application reset tool

[Joel Wee]

I agree Boyang, we should be able to specify both `—internal-topics` and 
`—dry-run` at the same time and this should display the topics that will be 
deleted. What I was trying to communicate in the description is that if you 
want to see all the topics that are valid as input into the option, then first 
do a dry-run without the `—internal-topics` option. I’ve rephrased it slightly 
which hopefully makes it clearer.

--internal-topics   Comma-separated list of internal topics
   to delete. Must be a subset of the
   internal topics marked for deletion by
   the default behaviour (do a dry-run 
without this
   option to view these topics).


- Joel

On 11 Jul 2020, at 8:41 AM, Boyang Chen 
mailto:reluctanthero...@gmail.com>> wrote:

Thanks for the update!

On Fri, Jul 3, 2020 at 3:18 PM Joel Wee 
mailto:joel@outlook.com>> wrote:

Thanks all for voting!

I am closing the vote as accepted with three binding +1 votes (Boyang,
Guozhang, John).

Thanks John for the suggestion. I think that makes sense. I have updated
the KIP to say that only topics flagged as internal by the tool can be
deleted, and have also rephrased the option description to make this
clearer:

--internal-topics   Comma-separated list of internal
topics
   to delete. Must be a subset of
the
   internal topics marked for
deletion by
   the default behaviour. To view
these
   topics, do a dry-run without
this
   option.

Hmm, could you explain why we couldn't run with both --internal-topics and
--dry-run? To me we could either display the exact set of internal topics
specified to be deleted, or reject the same way as we are doing an actual
reset.


Thanks Guozhang for the suggestion. The updated option description should
address your first point. By the “topology description” script are you
referring to bin/kafka-topics.sh? It currently has the option to display
all topics (including internal topics). Were you thinking about adding
something to view just internal topics?

Thanks Bruno for the suggestion. I will close this vote for now, and we
can continue the discussion on another thread. (P.S. apologies for
misspelling your name previously)

- Joel

On 1 Jul 2020, at 3:04 AM, Boyang Chen 
mailto:reluctanthero...@gmail.com>>
wrote:

Hey Bruno,

I agree adding a prompt would be a nice precaution, but it is not
backward
compatible as you suggested and could make the automation harder to
achieve.

If you want, we may consider starting a separate ticket to discuss
whether
adding a prompt to let users be aware of the topics that are about to
delete. However, this is also inverting the assumptions we made about
`--dry-run` mode, which would become useless to me once we are adding a
prompt asking users whether they want to remove these topics completely,
or
do nothing at all.

Back to KIP-623, I think this discussion could be held in orthogonal,
which
applies to more general considerations of reducing human errors, etc.

Boyang

On Tue, Jun 30, 2020 at 12:55 AM Bruno Cadonna 
mailto:br...@confluent.io>>
wrote:

Hi,

I have already brought this up in the discussion thread.

Should we not run a dry-run in any case to avoid inadvertently
deleting topics of other applications?

I know it is a backward incompatible change if users use it in
scripts, but I think it is still worth discussing it. I would to hear
what committers think about it.

Best,
Bruno

On Tue, Jun 30, 2020 at 5:33 AM Boyang Chen 
mailto:reluctanthero...@gmail.com>

wrote:

Thanks John for the great suggestion. +1 for enforcing the prefix check
for
the `--internal-topics` list.

On Mon, Jun 29, 2020 at 5:11 PM John Roesler 
mailto:vvcep...@apache.org>>
wrote:

Oh, I meant to say, if that’s the case, then I’m +1 (binding) also :)

Thanks again,
John

On Mon, Jun 29, 2020, at 19:09, John Roesler wrote:
Thanks for the KIP, Joel!

It seems like a good pattern to document would be to first run with
—dry-run and without —internal-topics to list all potential topics,
and
then to use —internal-topics if needed to limit the internal topics
to
delete.

Just to make sure, would we have a sanity check to forbid including
arbitrary topics? I.e., it seems like —internal-topics should require
all the topics to be prefixed with the app id. Is that right?

Thanks,
John

On Mon, Jun 29, 2020, at 18:25, Guozhang Wang wrote:
Thanks Joel, the KIP lgtm.

A minor suggestion is to explain where users can get the list of
internal
topics of a given application, and maybe also add it as part of the
helper
scripts, for example via topology description.

Overall, I'm +1 as well (binding).

Guozhang


On Sat, Jun 27, 2020 at 4:33 AM Joel Wee 
mailto:joel@outlook.com>>
wrote:

[jira] [Created] (KAFKA-10286) Connect system tests should wait for workers to join group

[Greg Harris (Jira)]

Greg Harris created KAFKA-10286:
---

 Summary: Connect system tests should wait for workers to join group
 Key: KAFKA-10286
 URL: https://issues.apache.org/jira/browse/KAFKA-10286
 Project: Kafka
  Issue Type: Test
  Components: KafkaConnect
Affects Versions: 2.6.0
Reporter: Greg Harris
Assignee: Greg Harris


There are a few flakey test failures for {{connect_distributed_test}} in which 
one of the workers does not join the group quickly, and the test fails in the 
following manner:
 # The test starts each of the connect workers, and waits for their REST APIs 
to become available
 # All workers start up, complete plugin scanning, and start their REST API
 # At least one worker kicks off an asynchronous job to join the group that 
hangs for a yet unknown reason (30s timeout)
 # The test continues without all of the members joined
 # The test makes a call to the REST api that it expects to succeed, and gets 
an error
 # The test fails without the worker ever joining the group

Instead of allowing the test to fail in this manner, we could wait for each 
worker to join the group with the existing 60s startup timeout. This change 
would go into effect for all system tests using the 
{{ConnectDistributedService}}, currently just {{connect_distributed_test}} and 
{{connect_rest_test}}. 

Alternatively we could retry the operation that failed, or ensure that we use a 
known-good worker to continue the test, but these would require more involved 
code changes. The existing wait-for-startup logic is the most natural place to 
fix this issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)