[jira] [Created] (KAFKA-12693) Consecutive rebalances with zombie instances may cause corrupted changelogs

[Guozhang Wang (Jira)]

Guozhang Wang created KAFKA-12693:
-

 Summary: Consecutive rebalances with zombie instances may cause 
corrupted changelogs
 Key: KAFKA-12693
 URL: https://issues.apache.org/jira/browse/KAFKA-12693
 Project: Kafka
  Issue Type: Bug
Reporter: Guozhang Wang


When an instance (or thread within an instance) of Kafka Streams has a soft 
failure and the group coordinator triggers a rebalance, that instance would 
temporarily become a "zombie writer". That is, this instance does not know 
there's already a new rebalance and hence its partitions have been migrated 
out, until it tries to commit and then got notified of the illegal-generation 
error and realize itself is the "zombie" already. During this period until the 
commit, this zombie may still be writing data to the changelogs of the migrated 
tasks as the new owner has already taken over and also writing to the 
changelogs.

When EOS is enabled, this would not be a problem: when the zombie tries to 
commit and got notified that it's fenced, its zombie appends would be aborted. 
With EOS disabled, though, such shared writes would be interleaved on the 
changelogs where a zombie append may arrive later after the new writer's 
append, effectively overwriting that new append.

Note that such interleaving writes do not necessarily cause corrupted data: as 
long as the new producer keep appending after the old zombie stops, and all the 
corrupted keys are overwritten again by the new values, then it is fine. 
However, if there are consecutive rebalances where right after the changelogs 
are corrupted by zombie writers, and before the new writer can overwrite them 
again, the task gets migrated again and needs to be restored from changelogs, 
the old values would be restored instead of the new values, effectively causing 
data loss.

Although this should be a rare event, we should fix it asap still. One idea is 
to have producers get a PID even under ALOS: that is, we set the transactional 
id in the producer config, but did not trigger any txn APIs; when there are 
zombie producers, they would then be immediately fenced on appends and hence 
there's no interleaved appends. I think this may require a KIP still, since 
today one has to call initTxn in order to register and get the PID.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #52

[Apache Jenkins Server]

See 

Re: [VOTE} KIP-733: change Kafka Streams default replication factor config

[Matthias J. Sax]

Thanks Ismael,

just to clarify, any app that is running and is regularly upgraded is
not affected: As pointed out in the KIP, if there is a running app that
does not overwrite the default, and the app is upgraded, it won't be
affected because the repartition/changelog topics already exist.

Second, as the default value is currently 1, for production application
I would assume that most people change the default anyway and thus won't
be affected either. Thus, if one deploys a new production app, or resets
a production app with reasonable config overwrites, they won't be affected.

Hence, the impact on production deployment should basically be
non-existent, and for those rare cases were user would be affected it
should be just a small hick-up because they would either deploy a new
application (maybe annoying but no harm done; and actually, they should
detect in staging) or would have reset an existing app for data
reprocessing, ie, they did some manual cleanup and might hit this corner
case only when re-deploying an previously stopped app, ie, an app that
is currently offline anyway. Again, no harm done, just some delay to
redeploy the app.

Also, we point the change out in the upgrade notes. Obviously, not
everybody might read them, but in the end there are multiple guards and
if a user really has issues, it sounds like a very rare corner case.

Last, while it might be possible to do so, the question is really, which
value should we pick?

Thus, overall I personally don't see any need to cover this case
automatically. It might not be too hard to write code and handle this
case, but I would rather keep it simple as it should really not impact
users in any critical way.


Thoughts?


-Matthias

On 4/18/21 8:00 PM, Ismael Juma wrote:
> Is the following accurate? Do we have data that not many users would be
> affected?
> 
> "We also believe that 2.3.0 broker a sufficiently old such that not too
> many user may be affected"
> 
> I wonder if we should fallback to an actual value if the brokers are 2.3.x
> or older.
> 
> Ismael
> 

[jira] [Created] (KAFKA-12692) Enforce config types in ConfigurationControlManager

[Ryan Dielhenn (Jira)]

Ryan Dielhenn created KAFKA-12692:
-

 Summary: Enforce config types in ConfigurationControlManager
 Key: KAFKA-12692
 URL: https://issues.apache.org/jira/browse/KAFKA-12692
 Project: Kafka
  Issue Type: Task
  Components: config
Reporter: Ryan Dielhenn
Assignee: Ryan Dielhenn


Enforce config types when using the new alterConfigs and 
incrementalAlterConfigs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12691) TaskMetadata timeSinceIdlingStarted not reporting correctly

[Walker Carlson (Jira)]

Walker Carlson created KAFKA-12691:
--

 Summary: TaskMetadata timeSinceIdlingStarted not reporting 
correctly
 Key: KAFKA-12691
 URL: https://issues.apache.org/jira/browse/KAFKA-12691
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Walker Carlson
Assignee: Walker Carlson


TaskMetadata timeSinceIdlingStarted not reporting correctly. It takes into 
account suspended but not the call to is processable. To fix this we need to 
record when the first time it is not processable. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[Sophie Blee-Goldman]

Thanks all. This KIP has passed with eight +1 votes:

+1 (binding)
Myself
Bruno
Ismael
John
Guozhang
Matthias

+1 (non-binding)
Israel
Luke

I'll prepare a PR for this shortly. I have also filed KAFKA-12689
 and KAFKA-12690
 for the removal
of these configs and method in 4.0

-Sophie


On Fri, Apr 16, 2021 at 6:20 AM Luke Chen  wrote:

> Hi Sophie,
> +1 (non-binding)
> Thanks for the KIP!
>
> Luke
>
>
> Matthias J. Sax  於 2021年4月16日 週五 上午11:21 寫道：
>
> > +1 (binding)
> >
> > On 4/15/21 12:56 PM, Israel Ekpo wrote:
> > > 1+ I agree.
> > >
> > > I think besides just merging the changes, specific attention should be
> > > brought to the KIP in the 3.0 release through blogs and tutorials to
> make
> > > the community more aware of the change in 3.0, upcoming removal of
> > > deprecated features in 4.0, it's prerequisites (broker version etc) and
> > > benefits
> > >
> > > Thanks for working on this
> > >
> > > On Thu, Apr 15, 2021 at 2:43 PM Guozhang Wang 
> > wrote:
> > >
> > >> +1 as well (binding). Thanks Sophie!
> > >>
> > >> On Thu, Apr 15, 2021 at 7:29 AM Bruno Cadonna 
> > wrote:
> > >>
> > >>> Sophie,
> > >>>
> > >>> Thank you for the KIP!
> > >>>
> > >>> +1 (binding)
> > >>>
> > >>> Best,
> > >>> Bruno
> > >>>
> > >>> On 15.04.21 01:59, Sophie Blee-Goldman wrote:
> >  Hey all,
> > 
> >  I'd like to kick off the vote on KIP-732, to deprecate eos-alpha in
> > >> Kafka
> >  Streams and migrate away from the "eos-beta" terminology by
> replacing
> > >> it
> >  with "eos-v2" to shore up user confidence in this feature.
> > 
> > 
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-732%3A+Deprecate+eos-alpha+and+replace+eos-beta+with+eos-v2
> > 
> >  Please respond on the discussion thread if you have any
> late-breaking
> >  concerns or questions.
> > 
> >  Thanks!
> >  Sophie
> > 
> > >>>
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >
> >
>

[jira] [Created] (KAFKA-12690) Remove deprecated Producer#sendOffsetsToTransaction

[A. Sophie Blee-Goldman (Jira)]

A. Sophie Blee-Goldman created KAFKA-12690:
--

 Summary: Remove deprecated Producer#sendOffsetsToTransaction
 Key: KAFKA-12690
 URL: https://issues.apache.org/jira/browse/KAFKA-12690
 Project: Kafka
  Issue Type: Task
  Components: producer 
Reporter: A. Sophie Blee-Goldman
 Fix For: 4.0.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12689) Remove deprecated EOS configs

[A. Sophie Blee-Goldman (Jira)]

A. Sophie Blee-Goldman created KAFKA-12689:
--

 Summary: Remove deprecated EOS configs
 Key: KAFKA-12689
 URL: https://issues.apache.org/jira/browse/KAFKA-12689
 Project: Kafka
  Issue Type: Task
  Components: streams
Reporter: A. Sophie Blee-Goldman
 Fix For: 4.0.0


In 
[KIP-732|https://cwiki.apache.org/confluence/display/KAFKA/KIP-732%3A+Deprecate+eos-alpha+and+replace+eos-beta+with+eos-v2]
 we deprecated the EXACTLY_ONCE and EXACTLY_ONCE_BETA configs in StreamsConfig, 
to be removed in 4.0



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: Kafka » kafka-2.6-jdk8 #118

[Apache Jenkins Server]

See 


Changes:

[A. Sophie Blee-Goldman] Bump version to 2.6.2

[A. Sophie Blee-Goldman] Bump version to 2.6.3-SNAPSHOT


--
[...truncated 3.18 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > 

Re: [kafka-clients] Re: [ANNOUNCE] Apache Kafka 2.8.0

[Adam Bellemare]

Wow, 2.8! I remember 0.8.2.0 :)

Thanks for all the work everyone, and thank you John for putting together
the release!

On Mon, Apr 19, 2021 at 3:48 PM Israel Ekpo  wrote:

> This is fantastic news!
>
> Thanks everyone for contributing and thanks John for managing the release.
>
> On Mon, Apr 19, 2021 at 1:10 PM Guozhang Wang  wrote:
>
> > This is great! Thanks to everyone who has contributed to the release.
> >
> > On Mon, Apr 19, 2021 at 9:36 AM John Roesler 
> wrote:
> >
> >> The Apache Kafka community is pleased to announce the
> >> release for Apache Kafka 2.8.0
> >>
> >> Kafka 2.8.0 includes a number of significant new features.
> >> Here is a summary of some notable changes:
> >>
> >> * Early access of replace ZooKeeper with a self-managed
> >> quorum
> >> * Add Describe Cluster API
> >> * Support mutual TLS authentication on SASL_SSL listeners
> >> * JSON request/response debug logs
> >> * Limit broker connection creation rate
> >> * Topic identifiers
> >> * Expose task configurations in Connect REST API
> >> * Update Streams FSM to clarify ERROR state meaning
> >> * Extend StreamJoined to allow more store configs
> >> * More convenient TopologyTestDriver construtors
> >> * Introduce Kafka-Streams-specific uncaught exception
> >> handler
> >> * API to start and shut down Streams threads
> >> * Improve TimeWindowedDeserializer and TimeWindowedSerde to
> >> handle window size
> >> * Improve timeouts and retries in Kafka Streams
> >>
> >> All of the changes in this release can be found in the
> >> release notes:
> >> https://www.apache.org/dist/kafka/2.8.0/RELEASE_NOTES.html
> >>
> >>
> >> You can download the source and binary release (Scala 2.12
> >> and 2.13) from:
> >> https://kafka.apache.org/downloads#2.8.0
> >>
> >> 
> >> ---
> >>
> >>
> >> Apache Kafka is a distributed streaming platform with four
> >> core APIs:
> >>
> >>
> >> ** The Producer API allows an application to publish a
> >> stream records to one or more Kafka topics.
> >>
> >> ** The Consumer API allows an application to subscribe to
> >> one or more topics and process the stream of records
> >> produced to them.
> >>
> >> ** The Streams API allows an application to act as a stream
> >> processor, consuming an input stream from one or more topics
> >> and producing an output stream to one or more output topics,
> >> effectively transforming the input streams to output
> >> streams.
> >>
> >> ** The Connector API allows building and running reusable
> >> producers or consumers that connect Kafka topics to existing
> >> applications or data systems. For example, a connector to a
> >> relational database might capture every change to a table.
> >>
> >>
> >> With these APIs, Kafka can be used for two broad classes of
> >> application:
> >>
> >> ** Building real-time streaming data pipelines that reliably
> >> get data between systems or applications.
> >>
> >> ** Building real-time streaming applications that transform
> >> or react to the streams of data.
> >>
> >>
> >> Apache Kafka is in use at large and small companies
> >> worldwide, including Capital One, Goldman Sachs, ING,
> >> LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York
> >> Times, Uber, Yelp, and Zalando, among others.
> >>
> >> A big thank you for the following 128 contributors to this
> >> release!
> >>
> >> 17hao, abc863377, Adem Efe Gencer, Alexander Iskuskov, Alok
> >> Nikhil, Anastasia Vela, Andrew Lee, Andrey Bozhko, Andrey
> >> Falko, Andy Coates, Andy Wilkinson, Ankit Kumar, APaMio,
> >> Arjun Satish, ArunParthiban-ST, A. Sophie Blee-Goldman,
> >> Attila Sasvari, Benoit Maggi, bertber, bill, Bill Bejeck,
> >> Bob Barrett, Boyang Chen, Brajesh Kumar, Bruno Cadonna,
> >> Cheng Tan, Chia-Ping Tsai, Chris Egerton, CHUN-HAO TANG,
> >> Colin Patrick McCabe, Colin P. Mccabe, Cyrus Vafadari, David
> >> Arthur, David Jacot, David Mao, dengziming, Dhruvil Shah,
> >> Dima Reznik, Dongjoon Hyun, Dongxu Wang, Emre Hasegeli,
> >> feyman2016, fml2, Gardner Vickers, Geordie, Govinda Sakhare,
> >> Greg Harris, Guozhang Wang, Gwen Shapira, Hamza Slama,
> >> high.lee, huxi, Igor Soarez, Ilya Ganelin, Ismael Juma, Ivan
> >> Ponomarev, Ivan Yurchenko, jackyoh, James Cheng, James
> >> Yuzawa, Jason Gustafson, Jesse Gorzinski, Jim Galasyn, John
> >> Roesler, Jorge Esteban Quilcate Otoya, José Armando García
> >> Sancio, Julien Chanaud, Julien Jean Paul Sirocchi, Justine
> >> Olshan, Kengo Seki, Kowshik Prakasam, leah, Lee Dongjin,
> >> Levani Kokhreidze, Lev Zemlyanov, Liju John, Lincong Li,
> >> Lucas Bradstreet, Luke Chen, Manikumar Reddy, Marco Aurelio
> >> Lotz, mathieu, Matthew Wong, Matthias J. Sax, Matthias
> >> Merdes, Michael Bingham, Michael G. Noll, Mickael Maison,
> >> Montyleo, mowczare, Nikolay, Nikolay Izhikov, Ning Zhang,
> >> Nitesh Mor, Okada Haruki, panguncle, parafiend, Patrick
> >> Dignan, Prateek Agarwal, Prithvi, Rajini Sivaram, Raman
> >> Verma, Ramesh 

Re: [kafka-clients] Re: [ANNOUNCE] Apache Kafka 2.8.0

[Israel Ekpo]

This is fantastic news!

Thanks everyone for contributing and thanks John for managing the release.

On Mon, Apr 19, 2021 at 1:10 PM Guozhang Wang  wrote:

> This is great! Thanks to everyone who has contributed to the release.
>
> On Mon, Apr 19, 2021 at 9:36 AM John Roesler  wrote:
>
>> The Apache Kafka community is pleased to announce the
>> release for Apache Kafka 2.8.0
>>
>> Kafka 2.8.0 includes a number of significant new features.
>> Here is a summary of some notable changes:
>>
>> * Early access of replace ZooKeeper with a self-managed
>> quorum
>> * Add Describe Cluster API
>> * Support mutual TLS authentication on SASL_SSL listeners
>> * JSON request/response debug logs
>> * Limit broker connection creation rate
>> * Topic identifiers
>> * Expose task configurations in Connect REST API
>> * Update Streams FSM to clarify ERROR state meaning
>> * Extend StreamJoined to allow more store configs
>> * More convenient TopologyTestDriver construtors
>> * Introduce Kafka-Streams-specific uncaught exception
>> handler
>> * API to start and shut down Streams threads
>> * Improve TimeWindowedDeserializer and TimeWindowedSerde to
>> handle window size
>> * Improve timeouts and retries in Kafka Streams
>>
>> All of the changes in this release can be found in the
>> release notes:
>> https://www.apache.org/dist/kafka/2.8.0/RELEASE_NOTES.html
>>
>>
>> You can download the source and binary release (Scala 2.12
>> and 2.13) from:
>> https://kafka.apache.org/downloads#2.8.0
>>
>> 
>> ---
>>
>>
>> Apache Kafka is a distributed streaming platform with four
>> core APIs:
>>
>>
>> ** The Producer API allows an application to publish a
>> stream records to one or more Kafka topics.
>>
>> ** The Consumer API allows an application to subscribe to
>> one or more topics and process the stream of records
>> produced to them.
>>
>> ** The Streams API allows an application to act as a stream
>> processor, consuming an input stream from one or more topics
>> and producing an output stream to one or more output topics,
>> effectively transforming the input streams to output
>> streams.
>>
>> ** The Connector API allows building and running reusable
>> producers or consumers that connect Kafka topics to existing
>> applications or data systems. For example, a connector to a
>> relational database might capture every change to a table.
>>
>>
>> With these APIs, Kafka can be used for two broad classes of
>> application:
>>
>> ** Building real-time streaming data pipelines that reliably
>> get data between systems or applications.
>>
>> ** Building real-time streaming applications that transform
>> or react to the streams of data.
>>
>>
>> Apache Kafka is in use at large and small companies
>> worldwide, including Capital One, Goldman Sachs, ING,
>> LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York
>> Times, Uber, Yelp, and Zalando, among others.
>>
>> A big thank you for the following 128 contributors to this
>> release!
>>
>> 17hao, abc863377, Adem Efe Gencer, Alexander Iskuskov, Alok
>> Nikhil, Anastasia Vela, Andrew Lee, Andrey Bozhko, Andrey
>> Falko, Andy Coates, Andy Wilkinson, Ankit Kumar, APaMio,
>> Arjun Satish, ArunParthiban-ST, A. Sophie Blee-Goldman,
>> Attila Sasvari, Benoit Maggi, bertber, bill, Bill Bejeck,
>> Bob Barrett, Boyang Chen, Brajesh Kumar, Bruno Cadonna,
>> Cheng Tan, Chia-Ping Tsai, Chris Egerton, CHUN-HAO TANG,
>> Colin Patrick McCabe, Colin P. Mccabe, Cyrus Vafadari, David
>> Arthur, David Jacot, David Mao, dengziming, Dhruvil Shah,
>> Dima Reznik, Dongjoon Hyun, Dongxu Wang, Emre Hasegeli,
>> feyman2016, fml2, Gardner Vickers, Geordie, Govinda Sakhare,
>> Greg Harris, Guozhang Wang, Gwen Shapira, Hamza Slama,
>> high.lee, huxi, Igor Soarez, Ilya Ganelin, Ismael Juma, Ivan
>> Ponomarev, Ivan Yurchenko, jackyoh, James Cheng, James
>> Yuzawa, Jason Gustafson, Jesse Gorzinski, Jim Galasyn, John
>> Roesler, Jorge Esteban Quilcate Otoya, José Armando García
>> Sancio, Julien Chanaud, Julien Jean Paul Sirocchi, Justine
>> Olshan, Kengo Seki, Kowshik Prakasam, leah, Lee Dongjin,
>> Levani Kokhreidze, Lev Zemlyanov, Liju John, Lincong Li,
>> Lucas Bradstreet, Luke Chen, Manikumar Reddy, Marco Aurelio
>> Lotz, mathieu, Matthew Wong, Matthias J. Sax, Matthias
>> Merdes, Michael Bingham, Michael G. Noll, Mickael Maison,
>> Montyleo, mowczare, Nikolay, Nikolay Izhikov, Ning Zhang,
>> Nitesh Mor, Okada Haruki, panguncle, parafiend, Patrick
>> Dignan, Prateek Agarwal, Prithvi, Rajini Sivaram, Raman
>> Verma, Ramesh Krishnan M, Randall Hauch, Richard
>> Fussenegger, Rohan, Rohit Deshpande, Ron Dagostino, Samuel
>> Cantero, Sanket Fajage, Scott Hendricks, Shao Yang Hong,
>> ssugar, Stanislav Kozlovski, Stanislav Vodetskyi, tang7526,
>> Thorsten Hake, Tom Bentley, vamossagar12, Viktor Somogyi-
>> Vass, voffcheg109, Walker Carlson, wenbingshen, wycc,
>> xakassi, Xavier Léauté, Yilong Chang, zhangyue19921010

[jira] [Created] (KAFKA-12688) Sources and maven artifact in runtime classpath

[Cosmin Giurgiu (Jira)]

Cosmin Giurgiu created KAFKA-12688:
--

 Summary: Sources and maven artifact in runtime classpath
 Key: KAFKA-12688
 URL: https://issues.apache.org/jira/browse/KAFKA-12688
 Project: Kafka
  Issue Type: Bug
  Components: build, config
Affects Versions: 2.8.0
Reporter: Cosmin Giurgiu


Started new kafka-2.8.0 with scala 2.13 and while checking the runtime I found 
the 
 * kafka_2.13-2.8.0-sources.jar
 * maven-artifact-3.6.3.jar

in kafka's classpath:
{code:java}
java [...] -cp 
[...]/home/kafka/kafka/bin/../libs/kafka_2.13-2.8.0-sources.jar:[...]:/home/kafka/kafka/bin/../libs/maven-artifact-3.6.3.jar:[...]
 kafka.Kafka /home/kafka/kafka/config/server.properties{code}
Is there a reason for these jars to be there ?

I believe these should not be included in classpath at all.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [kafka-site] bbejeck merged pull request #349: Add Bill Bejeck as PMC member

[GitBox]

bbejeck merged pull request #349:
URL: https://github.com/apache/kafka-site/pull/349


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] bbejeck opened a new pull request #349: Add Bill Bejeck as PMC member

[GitBox]

bbejeck opened a new pull request #349:
URL: https://github.com/apache/kafka-site/pull/349


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[RESULTS] [VOTE] Release Kafka version 2.6.2

[Sophie Blee-Goldman]

This vote passes with 5 +1 votes (3 bindings) and no 0 or -1 votes.

+1 votes
PMC Members:
* Guozhang Wang
* John Roesler
* Bill Bejeck

Committers:
* Myself (Sophie Blee-Goldman)

Community:
* Tom Bentley

0 votes
* No votes

-1 votes
* No votes

Vote 
thread:https://lists.apache.org/thread.html/rec46cefee599dbf18af92aaf49d7addeb27dafef726c58dc9e71da13%40%3Cdev.kafka.apache.org%3E

I'll continue with the release process and the release announcement
will follow in the next few days.


Cheers,

Sophie

Re: [DISCUSS] KIP-494 Connect REST Endpoint for Transformations (SMTs) and other Plugins

[Cyrus Vafadari]

Thanks! Anyone else from the community with final thoughts before going to
vote?

On Mon, Apr 19, 2021 at 4:16 AM Tom Bentley  wrote:

> Hi Cyrus,
>
> That seems reasonable to me.
>
> On Thu, Apr 15, 2021 at 6:44 PM Cyrus Vafadari 
> wrote:
>
> > What do you think of "type" field remaining in, and returning
> > "transformation", "converter" or whatever type it is. This way the schema
> > remains consistent, and you can programmatically understand what plugins
> > are returned on the holistic "GET /plugins" endpoint? It will be slightly
> > redundant in the case where you specify the plugin-type as a path
> > parameter.
> >
> > On Thu, Apr 15, 2021 at 1:13 PM Tom Bentley  wrote:
> >
> > > Hi Cyrus,
> > >
> > > Re 2: A very minor thing but while type=source|sink for a connector, it
> > > doesn't makes sense for the other plugin types, but so the json for
> those
> > > plugins should omit that property rather than have type=null.
> > >
> > > Apart from that it seems reasonable to me. Thanks again,
> > >
> > > Tom
> > >
> > > On Thu, Apr 15, 2021 at 3:49 PM Cyrus Vafadari 
> > > wrote:
> > >
> > > > Hi Tom,
> > > >
> > > > Thanks for taking the time to respond with thoughtful questions:
> > > >
> > > > 1. I propose HTTP-400, will update the KIP to reflect this proposal
> > > > 2. I think the schema should be of the same format. It is quite
> > minimal,
> > > so
> > > > there is room to add fields in the future without breaking
> > compatibility.
> > > > I will update the KIP to specify.
> > > > 3. Yes, that's right.
> > > > 4. I personally don't see the value in deprecating since it's a
> simple
> > > > alias (interface and impl will both be simple changes). I would be
> > > > comfortable kicking that can down the road if/when there is a need
> for
> > an
> > > > actual breaking change. That way we can keep the scope and diff here
> > > tight.
> > > > But iff the community feels strongly that deprecating is the right
> > thing
> > > to
> > > > do, I'm happy to update the KIP to propose deprecating.
> > > >
> > > > Thanks again!
> > > >
> > > > On Wed, Apr 14, 2021 at 9:38 AM Tom Bentley 
> > wrote:
> > > >
> > > > > Hi Cyrus,
> > > > >
> > > > > Thanks for the KIP. A few questions:
> > > > >
> > > > > 1. What status code does the /plugins/{plugin-type} endpoint return
> > > when
> > > > an
> > > > > unknown type is given?
> > > > > 2. The result of /connector-plugins is a list of objects with
> > 'class',
> > > > > 'type' and 'version' properties. Presumably /plugins/connector is
> the
> > > > same,
> > > > > but what is the schema for the other plugin types?
> > > > > 3. You're not proposing an equivalent of the
> > > > > /connector-plugins/{connectorType}/config/validate endpoint for
> > > > > non-connector types, which I think makes sense, because you would
> > > > validate
> > > > > an SMT's config via its used by a connector, so the existing
> endpoint
> > > > > suffices, right?
> > > > > 4. Would /connector-plugins eventually be deprecated in favour of
> > > > > /plugins/connector, or do we expect it to remain in the API
> > > indefinitely
> > > > > and accept that /connector-plugins and /plugins/connector provide
> > > > identical
> > > > > responses? If we had the intention to deprecate in the future maybe
> > we
> > > > > should add a /plugins/connector/config/validate endpoint now?
> > > > >
> > > > > Many thanks,
> > > > >
> > > > > Tom
> > > > >
> > > > > On Tue, Apr 13, 2021 at 6:46 PM Cyrus Vafadari <
> cvafad...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > As the number of connect plugins, SMT's, etc have grown, I wanted
> > to
> > > > bump
> > > > > > this thread to see if there is more interest in adding a Connect
> > REST
> > > > > > endpoint to get the current plugins in the worker.
> > > > > >
> > > > > > Thanks all, and thanks Chris for the initial feedback on this.
> > > > > >
> > > > > > On Mon, Feb 17, 2020 at 12:56 PM Cyrus Vafadari <
> > cvafad...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for the feedback, Chris.
> > > > > > >
> > > > > > > WRT Worker-plugins, I see your point that they aren't very
> useful
> > > for
> > > > > > > people trying to create connectors to see what exists already,
> > but
> > > I
> > > > > > would
> > > > > > > find them useful for things like making assertions against a
> > docker
> > > > > image
> > > > > > > to confirm the pluginpaths/classpaths are configured correctly.
> > It
> > > is
> > > > > > more
> > > > > > > useful for "sanity check" kinds of things than
> > exploring/browsing.
> > > > That
> > > > > > > said, I don't feel too strongly and if more feedback from the
> > > > community
> > > > > > > thinks they are better excluded then we can remove them.
> > > > > > >
> > > > > > > WRT camelCase vs hyphens, I see your point here, will update
> the
> > > KIP.
> > > > > > >
> > > > > > > WRT compatibility -- Good question -- I would expect that yes,
> > new
> > > > > > plugins
> > 

Re: [ANNOUNCE] Apache Kafka 2.8.0

[Guozhang Wang]

This is great! Thanks to everyone who has contributed to the release.

On Mon, Apr 19, 2021 at 9:36 AM John Roesler  wrote:

> The Apache Kafka community is pleased to announce the
> release for Apache Kafka 2.8.0
>
> Kafka 2.8.0 includes a number of significant new features.
> Here is a summary of some notable changes:
>
> * Early access of replace ZooKeeper with a self-managed
> quorum
> * Add Describe Cluster API
> * Support mutual TLS authentication on SASL_SSL listeners
> * JSON request/response debug logs
> * Limit broker connection creation rate
> * Topic identifiers
> * Expose task configurations in Connect REST API
> * Update Streams FSM to clarify ERROR state meaning
> * Extend StreamJoined to allow more store configs
> * More convenient TopologyTestDriver construtors
> * Introduce Kafka-Streams-specific uncaught exception
> handler
> * API to start and shut down Streams threads
> * Improve TimeWindowedDeserializer and TimeWindowedSerde to
> handle window size
> * Improve timeouts and retries in Kafka Streams
>
> All of the changes in this release can be found in the
> release notes:
> https://www.apache.org/dist/kafka/2.8.0/RELEASE_NOTES.html
>
>
> You can download the source and binary release (Scala 2.12
> and 2.13) from:
> https://kafka.apache.org/downloads#2.8.0
>
> 
> ---
>
>
> Apache Kafka is a distributed streaming platform with four
> core APIs:
>
>
> ** The Producer API allows an application to publish a
> stream records to one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to
> one or more topics and process the stream of records
> produced to them.
>
> ** The Streams API allows an application to act as a stream
> processor, consuming an input stream from one or more topics
> and producing an output stream to one or more output topics,
> effectively transforming the input streams to output
> streams.
>
> ** The Connector API allows building and running reusable
> producers or consumers that connect Kafka topics to existing
> applications or data systems. For example, a connector to a
> relational database might capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of
> application:
>
> ** Building real-time streaming data pipelines that reliably
> get data between systems or applications.
>
> ** Building real-time streaming applications that transform
> or react to the streams of data.
>
>
> Apache Kafka is in use at large and small companies
> worldwide, including Capital One, Goldman Sachs, ING,
> LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York
> Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 128 contributors to this
> release!
>
> 17hao, abc863377, Adem Efe Gencer, Alexander Iskuskov, Alok
> Nikhil, Anastasia Vela, Andrew Lee, Andrey Bozhko, Andrey
> Falko, Andy Coates, Andy Wilkinson, Ankit Kumar, APaMio,
> Arjun Satish, ArunParthiban-ST, A. Sophie Blee-Goldman,
> Attila Sasvari, Benoit Maggi, bertber, bill, Bill Bejeck,
> Bob Barrett, Boyang Chen, Brajesh Kumar, Bruno Cadonna,
> Cheng Tan, Chia-Ping Tsai, Chris Egerton, CHUN-HAO TANG,
> Colin Patrick McCabe, Colin P. Mccabe, Cyrus Vafadari, David
> Arthur, David Jacot, David Mao, dengziming, Dhruvil Shah,
> Dima Reznik, Dongjoon Hyun, Dongxu Wang, Emre Hasegeli,
> feyman2016, fml2, Gardner Vickers, Geordie, Govinda Sakhare,
> Greg Harris, Guozhang Wang, Gwen Shapira, Hamza Slama,
> high.lee, huxi, Igor Soarez, Ilya Ganelin, Ismael Juma, Ivan
> Ponomarev, Ivan Yurchenko, jackyoh, James Cheng, James
> Yuzawa, Jason Gustafson, Jesse Gorzinski, Jim Galasyn, John
> Roesler, Jorge Esteban Quilcate Otoya, José Armando García
> Sancio, Julien Chanaud, Julien Jean Paul Sirocchi, Justine
> Olshan, Kengo Seki, Kowshik Prakasam, leah, Lee Dongjin,
> Levani Kokhreidze, Lev Zemlyanov, Liju John, Lincong Li,
> Lucas Bradstreet, Luke Chen, Manikumar Reddy, Marco Aurelio
> Lotz, mathieu, Matthew Wong, Matthias J. Sax, Matthias
> Merdes, Michael Bingham, Michael G. Noll, Mickael Maison,
> Montyleo, mowczare, Nikolay, Nikolay Izhikov, Ning Zhang,
> Nitesh Mor, Okada Haruki, panguncle, parafiend, Patrick
> Dignan, Prateek Agarwal, Prithvi, Rajini Sivaram, Raman
> Verma, Ramesh Krishnan M, Randall Hauch, Richard
> Fussenegger, Rohan, Rohit Deshpande, Ron Dagostino, Samuel
> Cantero, Sanket Fajage, Scott Hendricks, Shao Yang Hong,
> ssugar, Stanislav Kozlovski, Stanislav Vodetskyi, tang7526,
> Thorsten Hake, Tom Bentley, vamossagar12, Viktor Somogyi-
> Vass, voffcheg109, Walker Carlson, wenbingshen, wycc,
> xakassi, Xavier Léauté, Yilong Chang, zhangyue19921010
>
> We welcome your help and feedback. For more information on
> how to report problems, and to get involved, visit the
> project website at https://kafka.apache.org/
>
> Thank you!
>
>
> Regards,
>
> John Roesler
>
>

-- 
-- Guozhang

[GitHub] [kafka-site] rhauch merged pull request #348: Add Randall Hauch as PMC member

[GitBox]

rhauch merged pull request #348:
URL: https://github.com/apache/kafka-site/pull/348


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] rhauch opened a new pull request #348: Add Randall Hauch as PMC member

[GitBox]

rhauch opened a new pull request #348:
URL: https://github.com/apache/kafka-site/pull/348


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[ANNOUNCE] Apache Kafka 2.8.0

[John Roesler]

The Apache Kafka community is pleased to announce the
release for Apache Kafka 2.8.0

Kafka 2.8.0 includes a number of significant new features.
Here is a summary of some notable changes:

* Early access of replace ZooKeeper with a self-managed
quorum
* Add Describe Cluster API
* Support mutual TLS authentication on SASL_SSL listeners
* JSON request/response debug logs
* Limit broker connection creation rate
* Topic identifiers
* Expose task configurations in Connect REST API
* Update Streams FSM to clarify ERROR state meaning
* Extend StreamJoined to allow more store configs
* More convenient TopologyTestDriver construtors
* Introduce Kafka-Streams-specific uncaught exception
handler
* API to start and shut down Streams threads
* Improve TimeWindowedDeserializer and TimeWindowedSerde to
handle window size
* Improve timeouts and retries in Kafka Streams

All of the changes in this release can be found in the
release notes:
https://www.apache.org/dist/kafka/2.8.0/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.12
and 2.13) from:
https://kafka.apache.org/downloads#2.8.0


---


Apache Kafka is a distributed streaming platform with four
core APIs:


** The Producer API allows an application to publish a
stream records to one or more Kafka topics.

** The Consumer API allows an application to subscribe to
one or more topics and process the stream of records
produced to them.

** The Streams API allows an application to act as a stream
processor, consuming an input stream from one or more topics
and producing an output stream to one or more output topics,
effectively transforming the input streams to output
streams.

** The Connector API allows building and running reusable
producers or consumers that connect Kafka topics to existing
applications or data systems. For example, a connector to a
relational database might capture every change to a table.


With these APIs, Kafka can be used for two broad classes of
application:

** Building real-time streaming data pipelines that reliably
get data between systems or applications.

** Building real-time streaming applications that transform
or react to the streams of data.


Apache Kafka is in use at large and small companies
worldwide, including Capital One, Goldman Sachs, ING,
LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York
Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 128 contributors to this
release!

17hao, abc863377, Adem Efe Gencer, Alexander Iskuskov, Alok
Nikhil, Anastasia Vela, Andrew Lee, Andrey Bozhko, Andrey
Falko, Andy Coates, Andy Wilkinson, Ankit Kumar, APaMio,
Arjun Satish, ArunParthiban-ST, A. Sophie Blee-Goldman,
Attila Sasvari, Benoit Maggi, bertber, bill, Bill Bejeck,
Bob Barrett, Boyang Chen, Brajesh Kumar, Bruno Cadonna,
Cheng Tan, Chia-Ping Tsai, Chris Egerton, CHUN-HAO TANG,
Colin Patrick McCabe, Colin P. Mccabe, Cyrus Vafadari, David
Arthur, David Jacot, David Mao, dengziming, Dhruvil Shah,
Dima Reznik, Dongjoon Hyun, Dongxu Wang, Emre Hasegeli,
feyman2016, fml2, Gardner Vickers, Geordie, Govinda Sakhare,
Greg Harris, Guozhang Wang, Gwen Shapira, Hamza Slama,
high.lee, huxi, Igor Soarez, Ilya Ganelin, Ismael Juma, Ivan
Ponomarev, Ivan Yurchenko, jackyoh, James Cheng, James
Yuzawa, Jason Gustafson, Jesse Gorzinski, Jim Galasyn, John
Roesler, Jorge Esteban Quilcate Otoya, José Armando García
Sancio, Julien Chanaud, Julien Jean Paul Sirocchi, Justine
Olshan, Kengo Seki, Kowshik Prakasam, leah, Lee Dongjin,
Levani Kokhreidze, Lev Zemlyanov, Liju John, Lincong Li,
Lucas Bradstreet, Luke Chen, Manikumar Reddy, Marco Aurelio
Lotz, mathieu, Matthew Wong, Matthias J. Sax, Matthias
Merdes, Michael Bingham, Michael G. Noll, Mickael Maison,
Montyleo, mowczare, Nikolay, Nikolay Izhikov, Ning Zhang,
Nitesh Mor, Okada Haruki, panguncle, parafiend, Patrick
Dignan, Prateek Agarwal, Prithvi, Rajini Sivaram, Raman
Verma, Ramesh Krishnan M, Randall Hauch, Richard
Fussenegger, Rohan, Rohit Deshpande, Ron Dagostino, Samuel
Cantero, Sanket Fajage, Scott Hendricks, Shao Yang Hong,
ssugar, Stanislav Kozlovski, Stanislav Vodetskyi, tang7526,
Thorsten Hake, Tom Bentley, vamossagar12, Viktor Somogyi-
Vass, voffcheg109, Walker Carlson, wenbingshen, wycc,
xakassi, Xavier Léauté, Yilong Chang, zhangyue19921010

We welcome your help and feedback. For more information on
how to report problems, and to get involved, visit the
project website at https://kafka.apache.org/

Thank you!


Regards,

John Roesler

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #51

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-12687) Add request cancellation to BrokerToControllerChannelManager

[David Arthur (Jira)]

David Arthur created KAFKA-12687:


 Summary: Add request cancellation to 
BrokerToControllerChannelManager
 Key: KAFKA-12687
 URL: https://issues.apache.org/jira/browse/KAFKA-12687
 Project: Kafka
  Issue Type: Improvement
Reporter: David Arthur


It would be nice if the broker-to-controller channel manager had the ability to 
cancel inflight requests. Since we process requests serially, this could avoid 
wasting time waiting on requests which are destined to fail.

One example is inflight ISR update requests after a leader election. The 
inflight request has the old leader epoch and we know it will fail. Currently, 
we have to wait for this request to complete before we can enqueue any new ISR 
changes for the partition.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12686) Race condition in AlterIsr response handling

[David Arthur (Jira)]

David Arthur created KAFKA-12686:


 Summary: Race condition in AlterIsr response handling
 Key: KAFKA-12686
 URL: https://issues.apache.org/jira/browse/KAFKA-12686
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.8.0, 2.7.0
Reporter: David Arthur
Assignee: David Arthur
 Fix For: 3.0.0


In Partition.scala, there is a race condition between the handling of an 
AlterIsrResponse and a LeaderAndIsrRequest. This is a pretty rare scenario and 
would involve the AlterIsrResponse being delayed for some time, but it is 
possible. This was observed in a test environment when lots of ISR and 
leadership changes were happening due to broker restarts.

When the leader handles the LeaderAndIsr, it calls Partition#makeLeader which 
overrides the {{isrState}} variable and clears the pending ISR items via 
{{AlterIsrManager#clearPending(TopicPartition)}}. 

The bug is that AlterIsrManager does not check its inflight state before 
clearing pending items. The way AlterIsrManager is designed, it retains 
inflight items in the pending items collection until the response is processed 
(to allow for retries). The result is that an inflight item is inadvertently 
removed from this collection.

Since the inflight item is cleared from the collection, AlterIsrManager allows 
for new AlterIsrItem-s to be enqueued for this partition even though it has an 
inflight AlterIsrItem. By allowing an update to be enqueued, Partition will 
transition its {{isrState}} to one of the inflight states (PendingIsrExpand, 
PendingIsrShrink, etc). Once the inflight partition's response is handled, it 
will fail to update the {{isrState}} due to detecting changes since the request 
was sent (which is by design). However, after the response callback is run, 
AlterIsrManager will clear the partitions that it saw in the response from the 
unsent items collection. This includes the newly added (and unsent) update.

The result is that Partition has a "inflight" isrState but AlterIsrManager does 
not have an unsent item for this partition. This prevents any further ISR 
updates on the partition until the next leader election (when {{isrState}} is 
reset).

If this bug is encountered, the workaround is to force a leader election which 
will reset the partition's state.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #50

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-12685) ReplicaStateMachine attempts invalid transition

[Tom Bentley (Jira)]

Tom Bentley created KAFKA-12685:
---

 Summary: ReplicaStateMachine attempts invalid transition
 Key: KAFKA-12685
 URL: https://issues.apache.org/jira/browse/KAFKA-12685
 Project: Kafka
  Issue Type: Bug
  Components: replication
Affects Versions: 2.7.0
Reporter: Tom Bentley
 Attachments: invalid-transition.log

The ReplicaStateMachine tried to perform the invalid transition {{NewReplica}} 
-> {{NewReplica}}, in a cluster which was being rolling restarted at the same 
time as the problem partition was being reassigned (first removing and then 
re-adding the replica).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-494 Connect REST Endpoint for Transformations (SMTs) and other Plugins

[Tom Bentley]

Hi Cyrus,

That seems reasonable to me.

On Thu, Apr 15, 2021 at 6:44 PM Cyrus Vafadari  wrote:

> What do you think of "type" field remaining in, and returning
> "transformation", "converter" or whatever type it is. This way the schema
> remains consistent, and you can programmatically understand what plugins
> are returned on the holistic "GET /plugins" endpoint? It will be slightly
> redundant in the case where you specify the plugin-type as a path
> parameter.
>
> On Thu, Apr 15, 2021 at 1:13 PM Tom Bentley  wrote:
>
> > Hi Cyrus,
> >
> > Re 2: A very minor thing but while type=source|sink for a connector, it
> > doesn't makes sense for the other plugin types, but so the json for those
> > plugins should omit that property rather than have type=null.
> >
> > Apart from that it seems reasonable to me. Thanks again,
> >
> > Tom
> >
> > On Thu, Apr 15, 2021 at 3:49 PM Cyrus Vafadari 
> > wrote:
> >
> > > Hi Tom,
> > >
> > > Thanks for taking the time to respond with thoughtful questions:
> > >
> > > 1. I propose HTTP-400, will update the KIP to reflect this proposal
> > > 2. I think the schema should be of the same format. It is quite
> minimal,
> > so
> > > there is room to add fields in the future without breaking
> compatibility.
> > > I will update the KIP to specify.
> > > 3. Yes, that's right.
> > > 4. I personally don't see the value in deprecating since it's a simple
> > > alias (interface and impl will both be simple changes). I would be
> > > comfortable kicking that can down the road if/when there is a need for
> an
> > > actual breaking change. That way we can keep the scope and diff here
> > tight.
> > > But iff the community feels strongly that deprecating is the right
> thing
> > to
> > > do, I'm happy to update the KIP to propose deprecating.
> > >
> > > Thanks again!
> > >
> > > On Wed, Apr 14, 2021 at 9:38 AM Tom Bentley 
> wrote:
> > >
> > > > Hi Cyrus,
> > > >
> > > > Thanks for the KIP. A few questions:
> > > >
> > > > 1. What status code does the /plugins/{plugin-type} endpoint return
> > when
> > > an
> > > > unknown type is given?
> > > > 2. The result of /connector-plugins is a list of objects with
> 'class',
> > > > 'type' and 'version' properties. Presumably /plugins/connector is the
> > > same,
> > > > but what is the schema for the other plugin types?
> > > > 3. You're not proposing an equivalent of the
> > > > /connector-plugins/{connectorType}/config/validate endpoint for
> > > > non-connector types, which I think makes sense, because you would
> > > validate
> > > > an SMT's config via its used by a connector, so the existing endpoint
> > > > suffices, right?
> > > > 4. Would /connector-plugins eventually be deprecated in favour of
> > > > /plugins/connector, or do we expect it to remain in the API
> > indefinitely
> > > > and accept that /connector-plugins and /plugins/connector provide
> > > identical
> > > > responses? If we had the intention to deprecate in the future maybe
> we
> > > > should add a /plugins/connector/config/validate endpoint now?
> > > >
> > > > Many thanks,
> > > >
> > > > Tom
> > > >
> > > > On Tue, Apr 13, 2021 at 6:46 PM Cyrus Vafadari 
> > > > wrote:
> > > >
> > > > > Hello all,
> > > > >
> > > > > As the number of connect plugins, SMT's, etc have grown, I wanted
> to
> > > bump
> > > > > this thread to see if there is more interest in adding a Connect
> REST
> > > > > endpoint to get the current plugins in the worker.
> > > > >
> > > > > Thanks all, and thanks Chris for the initial feedback on this.
> > > > >
> > > > > On Mon, Feb 17, 2020 at 12:56 PM Cyrus Vafadari <
> cvafad...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Thanks for the feedback, Chris.
> > > > > >
> > > > > > WRT Worker-plugins, I see your point that they aren't very useful
> > for
> > > > > > people trying to create connectors to see what exists already,
> but
> > I
> > > > > would
> > > > > > find them useful for things like making assertions against a
> docker
> > > > image
> > > > > > to confirm the pluginpaths/classpaths are configured correctly.
> It
> > is
> > > > > more
> > > > > > useful for "sanity check" kinds of things than
> exploring/browsing.
> > > That
> > > > > > said, I don't feel too strongly and if more feedback from the
> > > community
> > > > > > thinks they are better excluded then we can remove them.
> > > > > >
> > > > > > WRT camelCase vs hyphens, I see your point here, will update the
> > KIP.
> > > > > >
> > > > > > WRT compatibility -- Good question -- I would expect that yes,
> new
> > > > > plugins
> > > > > > would match this format. I will add this explicitly in the KIP
> for
> > > > > clarity.
> > > > > > I do think it's a valuable, simple feature to have the worker
> able
> > to
> > > > > > report new plugins.
> > > > > >
> > > > > > Thanks again!
> > > > > >
> > > > > > On Wed, Feb 12, 2020 at 4:44 PM Christopher Egerton <
> > > > chr...@confluent.io
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Cyrus,
> > > > > >>
> 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 2.8 #18

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 462722 lines...]
[2021-04-19T06:06:41.638Z] > Task :connect:json:jar UP-TO-DATE
[2021-04-19T06:06:41.638Z] > Task :connect:json:javadoc SKIPPED
[2021-04-19T06:06:41.638Z] > Task :connect:json:javadocJar
[2021-04-19T06:06:41.638Z] > Task :connect:api:javadocJar
[2021-04-19T06:06:41.638Z] > Task :connect:json:compileTestJava UP-TO-DATE
[2021-04-19T06:06:41.638Z] > Task :connect:json:testClasses UP-TO-DATE
[2021-04-19T06:06:41.638Z] > Task :streams:compileJava UP-TO-DATE
[2021-04-19T06:06:41.638Z] > Task :streams:classes UP-TO-DATE
[2021-04-19T06:06:41.638Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2021-04-19T06:06:41.638Z] > Task :connect:api:testClasses UP-TO-DATE
[2021-04-19T06:06:41.638Z] > Task :connect:json:testJar
[2021-04-19T06:06:41.638Z] > Task :connect:json:testSrcJar
[2021-04-19T06:06:41.638Z] > Task :connect:json:signArchives SKIPPED
[2021-04-19T06:06:41.638Z] > Task :connect:api:testJar
[2021-04-19T06:06:41.638Z] > Task :connect:api:testSrcJar
[2021-04-19T06:06:41.638Z] > Task :connect:api:signArchives SKIPPED
[2021-04-19T06:06:42.677Z] > Task :connect:json:install
[2021-04-19T06:06:47.111Z] > Task :clients:javadoc
[2021-04-19T06:06:47.111Z] > Task :clients:javadocJar
[2021-04-19T06:06:48.200Z] > Task :clients:srcJar
[2021-04-19T06:06:48.200Z] > Task :streams:javadoc
[2021-04-19T06:06:48.200Z] > Task :streams:javadocJar
[2021-04-19T06:06:48.200Z] > Task :clients:testJar
[2021-04-19T06:06:49.237Z] > Task :clients:testSrcJar
[2021-04-19T06:06:49.237Z] > Task :clients:signArchives SKIPPED
[2021-04-19T06:06:49.237Z] > Task :clients:install
[2021-04-19T06:06:49.237Z] > Task :connect:api:install
[2021-04-19T06:06:49.287Z] [Checks API] No suitable checks publisher found.
[Pipeline] echo
[2021-04-19T06:06:49.288Z] Skipping Kafka Streams archetype test for Java 11
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2021-04-19T06:07:09.091Z] > Task :core:compileScala
[2021-04-19T06:08:13.690Z] > Task :core:classes
[2021-04-19T06:08:13.690Z] > Task :core:copyDependantLibs
[2021-04-19T06:08:13.690Z] > Task :core:jar
[2021-04-19T06:08:13.690Z] > Task :core:compileTestJava NO-SOURCE
[2021-04-19T06:08:38.281Z] > Task :core:compileTestScala
[2021-04-19T06:09:13.732Z] > Task :core:testClasses
[2021-04-19T06:09:13.732Z] > Task :streams:copyDependantLibs
[2021-04-19T06:09:13.732Z] > Task :streams:jar UP-TO-DATE
[2021-04-19T06:09:13.732Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2021-04-19T06:09:13.732Z] > Task :streams:test-utils:classes UP-TO-DATE
[2021-04-19T06:09:13.732Z] > Task :streams:test-utils:copyDependantLibs
[2021-04-19T06:09:13.732Z] > Task :streams:test-utils:jar UP-TO-DATE
[2021-04-19T06:09:44.056Z] > Task :streams:compileTestJava
[2021-04-19T06:10:41.188Z] > Task :streams:testClasses
[2021-04-19T06:10:41.188Z] > Task :streams:testJar
[2021-04-19T06:10:41.188Z] > Task :streams:testSrcJar
[2021-04-19T06:10:41.188Z] > Task :streams:signArchives SKIPPED
[2021-04-19T06:10:41.188Z] > Task :streams:install
[2021-04-19T06:10:41.188Z] 
[2021-04-19T06:10:41.188Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 7.0.
[2021-04-19T06:10:41.188Z] Use '--warning-mode all' to show the individual 
deprecation warnings.
[2021-04-19T06:10:41.188Z] See 
https://docs.gradle.org/6.8.1/userguide/command_line_interface.html#sec:command_line_warnings
[2021-04-19T06:10:41.188Z] 
[2021-04-19T06:10:41.188Z] BUILD SUCCESSFUL in 4m 13s
[2021-04-19T06:10:41.188Z] 69 actionable tasks: 34 executed, 35 up-to-date
[Pipeline] sh
[2021-04-19T06:10:43.881Z] + grep ^version= gradle.properties
[2021-04-19T06:10:43.881Z] + cut -d= -f 2
[Pipeline] dir
[2021-04-19T06:10:44.578Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_2.8/streams/quickstart
[Pipeline] {
[Pipeline] sh
[2021-04-19T06:10:46.745Z] + mvn clean install -Dgpg.skip
[2021-04-19T06:10:47.690Z] [INFO] Scanning for projects...
[2021-04-19T06:10:47.690Z] [INFO] 

[2021-04-19T06:10:47.690Z] [INFO] Reactor Build Order:
[2021-04-19T06:10:47.690Z] [INFO] 
[2021-04-19T06:10:47.690Z] [INFO] Kafka Streams :: Quickstart   
 [pom]
[2021-04-19T06:10:47.690Z] [INFO] streams-quickstart-java   
 [maven-archetype]
[2021-04-19T06:10:47.690Z] [INFO] 
[2021-04-19T06:10:47.690Z] [INFO] < 
org.apache.kafka:streams-quickstart >-
[2021-04-19T06:10:47.690Z] [INFO] Building Kafka Streams :: Quickstart 
2.8.1-SNAPSHOT[1/2]
[2021-04-19T06:10:47.690Z] [INFO] [ pom