[jira] [Resolved] (KAFKA-10502) Threadlocal may can not set null,because it may create a memory leak

[Guozhang Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-10502.
---
Fix Version/s: 2.7.0
   Resolution: Fixed

> Threadlocal  may can not set null,because it may create a memory leak
> -
>
> Key: KAFKA-10502
> URL: https://issues.apache.org/jira/browse/KAFKA-10502
> Project: Kafka
>  Issue Type: Bug
>Reporter: huangyiming
>Priority: Major
> Fix For: 2.7.0
>
>
> When setting Threadlocal to null it may create a memory leak, you can see the 
> link:
> [https://stackoverflow.com/questions/12424838/threadlocal-remove], so I think 
> we should invoke its remove instead 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #115

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 20 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Re: [DISCUSS] KIP-671: Shutdown Streams Application when appropriate exception is thrown

[Guozhang Wang]

I think single-threaded clients may be common in practice, and what
Matthias raised is a valid concern.

We had a related discussion in KIP-663, that maybe we can tweak the
`UncaughtExceptionExceptionHandler` a bit such that instead of just
registered users' function into the individual threads, we trigger them
BEFORE the thread dies in the "catch (Exception)" block. It was proposed
originally to make sure that in the function if a user calls
localThreadMetadata() the dying / throwing thread would still be included,
but maybe we could consider this reason as well.


Guozhang


On Fri, Sep 25, 2020 at 3:20 PM Matthias J. Sax  wrote:

> I am wondering about the usage pattern of this new method.
>
> As already discussed, the method only works if there is at least one
> running thread... Do we have any sense how many apps actually run
> multi-threaded vs single-threaded? It seems that the feature might be
> quite limited without having a handler that is called _before_ the
> thread dies? However, for this case, I am wondering if it might be
> easier to just return a enum type from such a handler instead of calling
> `KakfaStreams#initiateClosingAllClients()`?
>
> In general, it seems that there is some gap between the case of stopping
> all instances from "outside" (as proposed in the KIP), vs from "inside"
> (what I though was the original line of thinking for this KIP?).
>
> For the network partitioning case, should we at least shutdown all local
> threads? It might be sufficient that only one thread sends the "shutdown
> signal" while all others just shut down directly? Why should the other
> thread wait for shutdown signal for a rebalance? Or should we recommend
> to call `initiateClosingAllClients()` followed to `close()` to make sure
> that at least the local threads stop (what might be a little bit odd)?
>
> -Matthias
>
> On 9/24/20 7:51 AM, John Roesler wrote:
> > Hello all,
> >
> > Thanks for bringing this up, Bruno. It’s a really good point that a
> disconnected node would miss the signal and then resurrect a single-node
> “zombie cluster” when it reconnects.
> >
> > Offhand, I can’t think of a simple and reliable way to distinguish this
> case from one in which an operator starts a node manually after a prior
> shutdown signal. Can you? Right now, I’m inclined to agree with Walker that
> we should leave this as a problem for the future.
> >
> > It should certainly be mentioned in the kip, and it also deserves
> special mention in our javadoc and html docs for this feature.
> >
> > Thanks!
> > John
> >
> > On Wed, Sep 23, 2020, at 17:49, Walker Carlson wrote:
> >> Bruno,
> >>
> >> I think that we can't guarantee that the message will get
> >> propagated perfectly in every case of, say network partitioning, though
> it
> >> will work for many cases. So I would say it's best effort and I will
> >> mention it in the kip.
> >>
> >> As for when to use it I think we can discuss if this will be
> >> sufficient when we come to it, as long as we document its capabilities.
> >>
> >> I hope this answers your question,
> >>
> >> Walker
> >>
> >> On Tue, Sep 22, 2020 at 12:33 AM Bruno Cadonna 
> wrote:
> >>
> >>> Walker,
> >>>
> >>> I am sorry, but I still have a comment on the KIP although you have
> >>> already started voting.
> >>>
> >>> What happens when a consumer of the group skips the rebalancing that
> >>> propagates the shutdown request? Do you give a guarantee that all Kafka
> >>> Streams clients are shutdown or is it best effort? If it is best
> effort,
> >>> I guess the proposed method might not be used in critical cases where
> >>> stopping record consumption may prevent or limit damage. I am not
> saying
> >>> that it must be a guarantee, but this question should be answered in
> the
> >>> KIP, IMO.
> >>>
> >>> Best,
> >>> Bruno
> >>>
> >>> On 22.09.20 01:14, Walker Carlson wrote:
>  The error code right now is the assignor error, 2 is coded for
> shutdown
>  but it could be expanded to encode the causes or for other errors that
> >>> need
>  to be communicated. For example we can add error code 3 to close the
> >>> thread
>  but leave the client in an error state if we choose to do so in the
> >>> future.
> 
>  On Mon, Sep 21, 2020 at 3:43 PM Boyang Chen <
> reluctanthero...@gmail.com>
>  wrote:
> 
> > Thanks for the KIP Walker.
> >
> > In the KIP we mentioned "In order to communicate the shutdown request
> >>> from
> > one client to the others we propose to update the
> SubcriptionInfoData to
> > include a short field which will encode an error code.", is there a
> > dedicated error code that we should define here, or it is
> case-by-case?
> >
> > On Mon, Sep 21, 2020 at 1:38 PM Walker Carlson <
> wcarl...@confluent.io>
> > wrote:
> >
> >> I am changing the name to "Add method to Shutdown entire Streams
> >> Application" since we are no longer using an Exception, it seems
> more
> >> appropriate.
> >>
> >> 

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #113

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 18 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #114

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 19 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #112

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 17 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #106

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 11 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #107

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 12 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #109

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 14 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #110

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 15 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #108

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 13 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #111

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 16 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #93

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10502: Use Threadlocal.remote to avoid leak on TimestampRouter 
(#9304)


--
[...truncated 3.35 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #95

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 0 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #92

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: add docs for 2.7 TRACE-level e2e latency metrics (#9339)


--
[...truncated 3.35 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
PASSED

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #96

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 1 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Jenkins build is back to normal : Kafka » kafka-trunk-jdk8 #91

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #117

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 22 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

[Guozhang Wang]

Hello Jose,

Thanks for the KIP. Overall it looks great. I have a few meta / minor
question, or maybe just clarifications below:

Meta:

1. I want to clarify that if only the active controller would generate
snapshots, OR would any voter in the quorum would generate snapshots, OR
would even observers generate snapshots? Originally I thought it was the
latter case, but I think reading through the doc I got confused by some
paragraphs. E.g. you mentioned snapshots are generated by the Controller
module, and observers would not have that module.

2. Following on Jun's previous comment: currently the __consumer_metadata
log is replicated on ALL brokers since all voters and observers would
replicate that topic. I know this may be out of the scope of this KIP but I
think maybe only letting the voters to replicate (and periodically
truncate) the log while observers only maintain the in-memory state and
snapshots is a good trade-off here, assuming snapshot loading is relatively
fast.

3. When a raft client is in the middle of loading a snapshot, should it
reject any vote / begin-/end-/describe-quorum requests at the time? More
generally, while a snapshot is being loaded, how should we treat the
current state of the client when handling Raft requests.

Minor:

4."All of the live replicas (followers and observers) have replicated LBO".
Today the raft layer does not yet maintain LBO across all replicas, is this
information kept in the controller layer? I'm asking because I do not see
relevant docs in KIP-631 and hence a bit confused which layer is
responsible for bookkeeping the LBOs of all replicas.

5. "Followers and observers will increase their log begin offset to the
value sent on the fetch response as long as the local Kafka Controller and
Metadata Cache has generated a snapshot with an end offset greater than or
equal to the new log begin offset." Not sure I follow this: 1) If observers
do not generate snapshots since they do not have a Controller module on
them, then it is possible that observers do not have any snapshots at all
if they do not get one from the leader, in that case they would never
truncate the logs locally; 2) could you clarify on "value sent on the fetch
response", are you referring to the "HighWatermark", or "LogStartOffset" in
the schema, or some other fields?

6. The term "SnapshotId" is introduced without definition at first. My
understanding is it's defined as a combo of , could you
clarify if this is the case? BTW I think the term "endOffset" is a term
used per log, and maybe calling the part of the SnapshotId "nextOffset" is
better since that offset is likely already filled with a record.

7. This is a very nit one: "If the latest snapshot has an epoch E and end
offset O and is it newer than the LEO of the replicated log, then the
replica must set the LBO and LEO to O." On wiki `O` and `0` looks very much
the same  and that confused me a couple of times... I'd suggest we phrase
any of such occussions to "an epoch e1 and offset o1". Also for LEO since
we would not really know what would be its epoch (since it may be bumped)
when comparing we only care about the offset and not about the epoch right?
If yes, please clarify that in the doc as well.

8. "LEO - log end offset - the largest offset and epoch that has been
written to disk." I think LEO is the "next" offset to be written to the log
right? Also it seems consistent with your diagrams.

9. "... will send a vote request and response as if they had an empty log."
Not sure I completely follow this, do you mean that they will set
"LastOffsetEpoch/LastOffset" as "-1/0" when sending a vote request, and
upon receiving a vote request it would compare the
request's "LastOffsetEpoch/LastOffset" with "-1/0" as well?

10. In the FetchSnapshot response schema, just to clarify the "Position" :
"The byte position within the snapshot." is referring to the starting byte
position of the returned snapshot data, right?


Thanks,
Guozhang

On Fri, Sep 25, 2020 at 4:42 PM Jun Rao  wrote:

> Hi, Jose,
>
> Thanks for the reply. A few more comments below.
>
> 20. Good point on metadata cache. I think we need to make a decision
> consistently. For example, if we decide that dedicated voter nodes don't
> serve metadata requests, then we don't need to expose the voters host/port
> to the client. Which KIP should make this decision?
>
> 31. controller.snapshot.minimum.records: For a compacted topic, we use a
> ratio instead of the number of records to determine when to compact. This
> has some advantages. For example, if we use
> controller.snapshot.minimum.records and set it to 1000, then it will
> trigger the generation of a new snapshot when the existing snapshot is
> either 10MB or 1GB. Intuitively, the larger the snapshot, the more
> expensive it is to write to disk. So, we want to wait for more data to be
> accumulated before generating the next snapshot. The ratio based setting
> achieves this. For instance, a 50% ratio requires 10MB/1GB more data to 

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #118

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 23 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #116

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 21 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #98

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 3 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #99

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 4 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #101

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 6 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #97

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 2 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #100

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 5 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #105

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 10 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #102

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 7 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #103

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 8 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #104

[Apache Jenkins Server]

See 


Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H48 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
ERROR: Cannot delete workspace: Unable to delete 
' Tried 3 
times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 14184 
additional exceptions)
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did leafNodes run? 
For example, 

 is 1 day 9 hr old

Not sending mail to unregistered user git...@hugo-hirsch.de