[jira] [Created] (KAFKA-12849) Consider migrating TaskMetadata to interface with internal implementation

[A. Sophie Blee-Goldman (Jira)]

A. Sophie Blee-Goldman created KAFKA-12849:
--

 Summary: Consider migrating TaskMetadata to interface with 
internal implementation
 Key: KAFKA-12849
 URL: https://issues.apache.org/jira/browse/KAFKA-12849
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: A. Sophie Blee-Goldman


In KIP-740 we had to go through a deprecation cycle in order to change the 
constructor from the original one which accepted the taskId parameter as a 
string, to the new one which takes a TaskId object directly. We had considered 
just changing the signature directly without deprecation as this was never 
intended to be instantiated by users, rather it just acts as a pass-through 
metadata class. Sort of by definition if there is no reason to ever instantiate 
it, this seems to indicate it may be better suited as a public interface with 
the implementation and constructor as internal APIs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12848) Add some basic benchmarks for Kafka Streams

[A. Sophie Blee-Goldman (Jira)]

A. Sophie Blee-Goldman created KAFKA-12848:
--

 Summary: Add some basic benchmarks for Kafka Streams
 Key: KAFKA-12848
 URL: https://issues.apache.org/jira/browse/KAFKA-12848
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: A. Sophie Blee-Goldman


As the title suggests, we often want to test out improvements or verify that a 
bugfix does not introduce a serious regression. While there are existing 
benchmarks that are run for quality assurance by various contributors, there 
are no publicly available benchmarks for Kafka Streams in AK itself.

It would be great if we had a simple jmh suite (or something) with various 
Streams features which could be run on a one-off basis by developers.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-699: Update FindCoordinator to resolve multiple Coordinators at a time

[Sanjana Kaundinya]

Hi Mickael,
Just took a look at your draft PR, seems like your implementation is looking to 
change the way we call all consumer group APIs to leverage the new 
AdminApiHandler introduced by Jason as part of KIP-664. It also seems like 
someone already got started to rewrite the ListOffsets API using this new 
infrastructure with this PR: https://github.com/apache/kafka/pull/10467/files 
:). In any case this is very helpful and certainly changes my plans on how I 
intended to implement the KIP. I can volunteer to help review your PR once 
you’re done with it, since it will give me a good idea for my PR. I was also 
hoping to get KIP-709 into 3.0 before the code freeze, but given that we are 
now rewriting ListOffsets API, I think I’d better wait until you get your KIP 
in. Are you planning on getting KIP-699 into 3.0?
Cheers,
Sanjana
On May 23, 2021, 11:35 PM -0700, Sanjana Kaundinya , 
wrote:
> Hi Mickael,
> Thank you for your response! Will take a look at the draft PR later this week.
> Cheers,
> Sanjana
> On May 21, 2021, 10:26 AM -0700, Mickael Maison , 
> wrote:
> > Hi Sanjana,
> >
> > Again I'm sorry for the delay.
> > I've opened a draft PR which hopefully will let you make progress on
> > KIP-709. Let me know if you have any questions.
> >
> > Thanks
> >
> > On Thu, May 20, 2021 at 10:10 PM Sanjana Kaundinya  
> > wrote:
> > >
> > > Hi Mickael,
> > >
> > > Apologies for pinging on this again, but just wondering if you got a 
> > > chance to make any progress on this? Would love to take a look and help 
> > > with the reviews if you have.
> > >
> > > Thanks,
> > > Sanjana
> > > On May 7, 2021, 3:33 AM -0700, Mickael Maison , 
> > > wrote:
> > > > Hi Sanjana,
> > > >
> > > > Sorry for the late reply.
> > > > I hope to have a draft PR next week.
> > > >
> > > > Thanks
> > > >
> > > > On Tue, May 4, 2021 at 11:11 PM Sanjana Kaundinya 
> > > >  wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Just wanted to ping the thread to inquire if you’ve made any progress 
> > > > > on the implementation. I was planning on doing the implementation for 
> > > > > KIP-709 and seeing how KIP-699 is something that’s tightly coupled 
> > > > > and related to KIP-709 I thought I’d ask what the status of this KIP 
> > > > > was.
> > > > >
> > > > > Thanks,
> > > > > Sanjana
> > > > > On Apr 13, 2021, 8:44 AM -0700, Rajini Sivaram 
> > > > > , wrote:
> > > > > > Thanks Mickael!
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > > On Tue, Apr 13, 2021 at 4:40 PM Mickael Maison 
> > > > > > 
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Rajini!
> > > > > > > Yes it would be good to get it into 3.0. I've got some of it
> > > > > > > implemented already, I'll try to get a PR out in the next couple 
> > > > > > > of
> > > > > > > weeks.
> > > > > > >
> > > > > > > I'm closing this vote. This KIP has been accepted with 3 +1 
> > > > > > > binding
> > > > > > > votes from David, Tom and Rajini.
> > > > > > >
> > > > > > > On Tue, Apr 13, 2021 at 1:49 PM Rajini Sivaram 
> > > > > > > 
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Mickael,
> > > > > > > >
> > > > > > > > +1 (binding)
> > > > > > > >
> > > > > > > > Thanks for the KIP!. Looks like this KIP has sufficient votes. 
> > > > > > > > It will be
> > > > > > > > good to get this into 3.0 along with KIP-709. Will you have 
> > > > > > > > time to work
> > > > > > > on
> > > > > > > > this? Please let us know if we can help with the implementation 
> > > > > > > > or
> > > > > > > reviews.
> > > > > > > >
> > > > > > > > Thank you,
> > > > > > > >
> > > > > > > > Rajini
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Mar 18, 2021 at 4:00 PM Tom Bentley 
> > > > > > > >  wrote:
> > > > > > > >
> > > > > > > > > Hi Mickael,
> > > > > > > > >
> > > > > > > > > I'd like to re-cast my vote as +1 (binding) now I'm a 
> > > > > > > > > committer.
> > > > > > > > >
> > > > > > > > > Thanks again,
> > > > > > > > >
> > > > > > > > > Tom
> > > > > > > > >
> > > > > > > > > On Tue, Mar 2, 2021 at 9:46 AM David Jacot 
> > > > > > > > > 
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks for the KIP, Mickael. +1 (binding)
> > > > > > > > > >
> > > > > > > > > > On Mon, Mar 1, 2021 at 11:53 AM Tom Bentley 
> > > > > > > > > > 
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1 (non-binding), thanks Mickael.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Feb 25, 2021 at 6:32 PM Mickael Maison <
> > > > > > > mimai...@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > I'd like to start a vote on KIP-699 to support 
> > > > > > > > > > > > resolving multiple
> > > > > > > > > > > > coordinators at a time:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > 

Re: [DISCUSS] KIP-739: Block Less on KafkaProducer#send

[Nakamura]

Hey Colin,

For the metadata case, what would fixing the bug look like?  I agree that
we should fix it, but I don't have a clear picture in my mind of what
fixing it should look like.  Can you elaborate?

Best,
Moses

On Mon, May 24, 2021 at 1:54 PM Colin McCabe  wrote:

> Hi all,
>
> I agree that we should give users the option of having a fully async API,
> but I don't think external thread pools or queues are the right direction
> to go here. They add performance overheads and don't address the root
> causes of the problem.
>
> There are basically two scenarios where we block, currently. One is when
> we are doing a metadata fetch. I think this is clearly a bug, or at least
> an implementation limitation. From the user's point of view, the fact that
> we are doing a metadata fetch is an implementation detail that really
> shouldn't be exposed like this. We have talked about fixing this in the
> past. I think we just should spend the time to do it.
>
> The second scenario is where the client has produced too much data in too
> little time. This could happen if there is a network glitch, or the server
> is slower than expected. In this case, the behavior is intentional and not
> a bug. To understand this, think about what would happen if we didn't
> block. We would start buffering more and more data in memory, until finally
> the application died with an out of memory error. That would be frustrating
> for users and wouldn't add to the usability of Kafka.
>
> We could potentially have an option to handle the out-of-memory scenario
> differently by returning an error code immediately rather than blocking.
> Applications would have to be rewritten to handle this properly, but it is
> a possibility. I suspect that most of them wouldn't use this, but we could
> offer it as a possibility for async purists (which might include certain
> frameworks). The big problem the users would have to solve is what to do
> with the record that they were unable to produce due to the buffer full
> issue.
>
> best,
> Colin
>
>
> On Thu, May 20, 2021, at 10:35, Nakamura wrote:
> > >
> > > My suggestion was just do this in multiple steps/phases, firstly let's
> fix
> > > the issue of send being misleadingly asynchronous (i.e. internally its
> > > blocking) and then later one we can make the various
> > > threadpools configurable with a sane default.
> >
> > I like that approach. I updated the "Which thread should be responsible
> for
> > waiting" part of KIP-739 to add your suggestion as my recommended
> approach,
> > thank you!  If no one else has major concerns about that approach, I'll
> > move the alternatives to "rejected alternatives".
> >
> > On Thu, May 20, 2021 at 7:26 AM Matthew de Detrich
> >  wrote:
> >
> > > @
> > >
> > > Nakamura
> > > On Wed, May 19, 2021 at 7:35 PM Nakamura  wrote:
> > >
> > > > @Ryanne:
> > > > In my mind's eye I slightly prefer the throwing the "cannot enqueue"
> > > > exception to satisfying the future immediately with the "cannot
> enqueue"
> > > > exception?  But I agree, it would be worth doing more research.
> > > >
> > > > @Matthew:
> > > >
> > > > > 3. Using multiple thread pools is definitely recommended for
> different
> > > > > types of tasks, for serialization which is CPU bound you definitely
> > > would
> > > > > want to use a bounded thread pool that is fixed by the number of
> CPU's
> > > > (or
> > > > > something along those lines).
> > > > > https://gist.github.com/djspiewak/46b543800958cf61af6efa8e072bfd5c
> is
> > > a
> > > > > very good guide on this topic
> > > > I think this guide is good in general, but I would be hesitant to
> follow
> > > > its guidance re: offloading serialization without benchmarking it.
> My
> > > > understanding is that context-switches have gotten much cheaper, and
> that
> > > > gains from cache locality are small, but they're not nothing.
> Especially
> > > > if the workload has a very small serialization cost, I wouldn't be
> > > shocked
> > > > if it made it slower.  I feel pretty strongly that we should do more
> > > > research here before unconditionally encouraging serialization in a
> > > > threadpool.  If people think it's important to do it here (eg if we
> think
> > > > it would mean another big API change) then we should start thinking
> about
> > > > what benchmarking we can do to gain higher confidence in this kind of
> > > > change.  However, I don't think it would change semantics as
> > > substantially
> > > > as we're proposing here, so I would vote for pushing this to a
> subsequent
> > > > KIP.
> > > >
> > > Of course, its all down to benchmarking, benchmarking and benchmarking.
> > > Ideally speaking you want to use all of the resources available to
> you, so
> > > if you have a bottleneck in serialization and you have many cores free
> then
> > > using multiple cores may be more appropriate than a single core.
> Typically
> > > I would expect that using a single thread to do serialization is
> likely to
> > > be the most 

[jira] [Reopened] (KAFKA-9295) KTableKTableForeignKeyInnerJoinMultiIntegrationTest#shouldInnerJoinMultiPartitionQueryable

[A. Sophie Blee-Goldman (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

A. Sophie Blee-Goldman reopened KAFKA-9295:
---

Guess there is still something else going on here yet. At this point I think we 
can mostly rule out fiddling with the configs but I don't have any guesses on 
where to look next. It would be nice if we could get real logs from a run that 
reproduced this, but unfortunately all the actual Streams content is truncated.

[~showuon] maybe you can look into turning the zookeeper and kafka logs down to 
WARN or even ERROR so that we have some hope of viewing the relevant parts of 
the logs? I tried to do that a while back but clearly it didn't work, and I 
didn't have time to revisit it

> KTableKTableForeignKeyInnerJoinMultiIntegrationTest#shouldInnerJoinMultiPartitionQueryable
> --
>
> Key: KAFKA-9295
> URL: https://issues.apache.org/jira/browse/KAFKA-9295
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, unit tests
>Affects Versions: 2.4.0, 2.6.0
>Reporter: Matthias J. Sax
>Assignee: Luke Chen
>Priority: Critical
>  Labels: flaky-test
> Fix For: 3.0.0
>
>
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/27106/testReport/junit/org.apache.kafka.streams.integration/KTableKTableForeignKeyInnerJoinMultiIntegrationTest/shouldInnerJoinMultiPartitionQueryable/]
> {quote}java.lang.AssertionError: Did not receive all 1 records from topic 
> output- within 6 ms Expected: is a value equal to or greater than <1> 
> but: <0> was less than <1> at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.lambda$waitUntilMinKeyValueRecordsReceived$1(IntegrationTestUtils.java:515)
>  at 
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:417)
>  at 
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:385)
>  at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:511)
>  at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:489)
>  at 
> org.apache.kafka.streams.integration.KTableKTableForeignKeyInnerJoinMultiIntegrationTest.verifyKTableKTableJoin(KTableKTableForeignKeyInnerJoinMultiIntegrationTest.java:200)
>  at 
> org.apache.kafka.streams.integration.KTableKTableForeignKeyInnerJoinMultiIntegrationTest.shouldInnerJoinMultiPartitionQueryable(KTableKTableForeignKeyInnerJoinMultiIntegrationTest.java:183){quote}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #161

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » kafka-2.7-jdk8 #156

[Apache Jenkins Server]

See 


Changes:

[Tom Bentley] KAFKA-10846: Grow buffer in FileSourceStreamTask only when needed 
(#9735)


--
[...truncated 6.91 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

subscribe to kafka dev

[姜家俊]

subscribe to kafka dev mailing list

[VOTE] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

Hi Kafka dev,

I'd like to kick-off the voting for KIP-719: Add Log4J2 Appender.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender

Best,
Dongjin

-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

Bumping up the discussion thread.

Recently, I updated the document of KIP-653: Upgrade log4j to log4j2

(accepted)
and KIP-719: Add Log4J2 Appender

(under
discussion) reflecting the recent changes to our codebase. Especially:

1. KIP-653 document

now
explains which modules will be migrated and why.
2. KIP-719 document

now
explains not only the log4j2-appender plan but also upgrading the omitted
modules in KIP-653 into log4j2.

As you can see here, those two KIPs are the different parts of the same
problem. I believe the community will have a good grasp on why both KIPs
are best if released altogether.

I will open the voting thread now, and please leave a vote if you are
interested in this issue.

Best,
Dongjin

On Tue, Mar 2, 2021 at 5:00 PM Dongjin Lee  wrote:

> Hi Kafka dev,
>
> I would like to start the discussion of KIP-719: Add Log4J2 Appender.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
>
> All kinds of feedbacks are greatly appreciated!
>
> Best,
> Dongjin
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck: speakerdeck.com/dongjin
> *
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Re: [DISCUSS] Apache Kafka 3.0.0 release

[Dongjin Lee]

Hi Konstantine,

Please Add the following KIPs into Kafka 3.0 plan. In the case of KIP-653,
it was passed already but not included in the 2.8.0 release.

- KIP-653: Upgrade log4j to log4j2

- KIP-719: Add Log4J2 Appender


I also updated the proposal documents reflecting the recent changes to our
codebase. Especially:

1. KIP-653 document

now explains which modules will be migrated and why.
2. KIP-719 document

now explains not only the log4j2-appender plan but also upgrading the
omitted modules in KIP-653 into log4j2.

As you can see here, those two KIPs are the different parts of the same
problem. I believe the community will have a good grasp on why both KIPs
are best if released altogether.

Best,
Dongjin

On Fri, May 14, 2021 at 8:51 AM Konstantine Karantasis
 wrote:

> Hi all,
>
> Lots of good progress recently towards the 3.0.0 release and given that we
> are now a little less than a month away from the first milestone, I wanted
> to post a status update and a reminder of the upcoming dates for this major
> release.
>
> According to the current release plan for Apache Kafka 3.0.0:
>
> KIP Freeze is 09 Jun 2021
>
> Feature Freeze is 16 Jun 2021
>
> Code Freeze is 30 Jun 2021
>
> and at least two weeks of stabilization will follow Code Freeze.
>
> Since the initial announcement, the list of adopted KIPs targeting 3.0.0
> has grown with 10 additional KIPs. Several more are under discussion and
> voting.
>
> New KIPs will be accepted up until the KIP Freeze and of course the actual
> set of features that will make it to 3.0.0 will be finalized right after
> Feature Freeze.
>
> With that in mind, if you want to assist the 3.0.0 release process, please
> make sure that the adopted KIPs which you aim to include in this major
> release have the right number in the “Release” column of the table in the
> KIP wiki and that their respective Jira tickets include 3.0.0 in the "Fix
> Version/s" label.
>
> Kafka Improvement Proposals:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>
> Release Plan 3.0.0:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.0.0
>
> Regards,
>
> Konstantine
>
>
> On Wed, Mar 10, 2021 at 10:28 AM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> >
> > Thank you and hi again.
> >
> > I just published a release plan at:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> >
> > I have included all the currently approved KIPs. I'm expecting this list
> > to grow as we approach KIP freeze.
> >
> > The KIP freeze date for Apache Kafka 3.0 is June 9th, 2021.
> >
> > Please let me know if you have any objections.
> >
> > Regards,
> > Konstantine
> >
> >
> > On Wed, Feb 24, 2021 at 9:45 AM Chia-Ping Tsai 
> > wrote:
> >
> >> Thanks for taking over this hard job! +1
> >>
> >> On 2021/02/23 08:02:09, Konstantine Karantasis <
> konstant...@confluent.io>
> >> wrote:
> >> > Hi all,
> >> >
> >> > Given that we seem to reach an agreement that the feature release
> after
> >> the
> >> > upcoming 2.8.0 will be 3.0.0, I'd like to volunteer to be the release
> >> > manager for the Apache Kafka 3.0.0 release.
> >> >
> >> > It's a major release, so I thought it'd be helpful to start planning a
> >> bit
> >> > in advance.
> >> >
> >> > If there are no objections, I'll start working on a release plan in
> the
> >> > next few days.
> >> >
> >> > Best,
> >> > Konstantine
> >> >
> >>
> >
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Re: [VOTE] KIP-741: Change default serde to be null

[Dongjin Lee]

Hi Bruno,

Oh, thanks for pointing out my mistake. Yes, the KIP is not passed yet and
here is the updated status:

- Binding: Guozhang Wang, Sophie Blee-Goldman, John Roesler, Bruno Cadonna
(+4)
- Non-binding: Walker Carlson, Lee Dongjin (+2)

Thanks,
Dongjin

On Tue, May 25, 2021 at 9:12 PM Bruno Cadonna  wrote:

> Hi Leah,
>
> Thanks for the KIP!
>
> +1 (binding)
>
> Best,
> Bruno
>
> On 25.05.21 14:02, Bruno Cadonna wrote:
> > Hi Dongjin,
> >
> > voting for a KIP needs to remain open for at least 72 hours [1].
> > According to the date and time of the first message to this thread 72
> > hours havn't passed, yet. Theoretically, there could still be -1 votes
> > coming in.
> >
> > Best,
> > Bruno
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-Process
> >
> >
> >
> > On 25.05.21 13:13, Dongjin Lee wrote:
> >> +1 (non-binding).
> >>
> >> As of present:
> >>
> >> - Binding: Guozhang Wang, Sophie Blee-Goldman, John Roesler (+3)
> >> - Non-binding: Walker Carlson, Lee Dongjin (+2)
> >>
> >> This KIP is now passed.
> >>
> >> Thanks,
> >> Dongjin
> >>
> >> On Tue, May 25, 2021 at 10:12 AM John Roesler 
> >> wrote:
> >>
> >>> +1 (binding) from me. Thanks for the KIP!
> >>> -John
> >>>
> >>> On Mon, May 24, 2021, at 18:10, Sophie Blee-Goldman wrote:
>  +1 binding
> 
>  thanks for the KIP
>  -Sophie
> 
>  On Mon, May 24, 2021 at 2:02 PM Walker Carlson
>   wrote:
> 
> > +1 (non-binding) from me, Leah
> >
> > Walker
> >
> > On Mon, May 24, 2021 at 1:51 PM Leah Thomas
> >>> 
> > wrote:
> >
> >> Hi,
> >>
> >> I'd like to kick-off voting for KIP-741: Change default serde to be
> >>> null.
> >> <
> >>
> >
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-741%3A+Change+default+serde+to+be+null
> >>>
> >>>
> >> The
> >> discussion is linked on the KIP for context.
> >>
> >> Cheers,
> >> Leah
> >>
> >
> 
> >>>
> >>
> >>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck: speakerdeck.com/dongjin
> *
>

[jira] [Resolved] (KAFKA-12800) Configure jackson to to reject trailing input in the generator

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-12800.
-
Fix Version/s: 3.0.0
   Resolution: Fixed

Author: Nathan Lincoln

> Configure jackson to to reject trailing input in the generator
> --
>
> Key: KAFKA-12800
> URL: https://issues.apache.org/jira/browse/KAFKA-12800
> Project: Kafka
>  Issue Type: Task
>  Components: generator
>Reporter: Nathan Lincoln
>Priority: Minor
> Fix For: 3.0.0
>
>
> The ObjectMapper instance that parses the schema JSONs will successfully 
> parse, even if there is trailing input at the end of the file. This the 
> default behavior on Jackson, but JSON parsers in other languages may reject 
> these files. 
> The only instance of this should have been fixed with KAFKA-12794, and 
> configuring jackson to reject this in the future is simple - just enable 
> [FAIL_ON_TRAILING_TOKENS|https://fasterxml.github.io/jackson-databind/javadoc/2.9/com/fasterxml/jackson/databind/DeserializationFeature.html#FAIL_ON_TRAILING_TOKENS]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-741: Change default serde to be null

[Bruno Cadonna]

Hi Leah,

Thanks for the KIP!

+1 (binding)

Best,
Bruno

On 25.05.21 14:02, Bruno Cadonna wrote:

Hi Dongjin,

voting for a KIP needs to remain open for at least 72 hours [1]. 
According to the date and time of the first message to this thread 72 
hours havn't passed, yet. Theoretically, there could still be -1 votes 
coming in.


Best,
Bruno

[1] 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-Process 




On 25.05.21 13:13, Dongjin Lee wrote:

+1 (non-binding).

As of present:

- Binding: Guozhang Wang, Sophie Blee-Goldman, John Roesler (+3)
- Non-binding: Walker Carlson, Lee Dongjin (+2)

This KIP is now passed.

Thanks,
Dongjin

On Tue, May 25, 2021 at 10:12 AM John Roesler  
wrote:



+1 (binding) from me. Thanks for the KIP!
-John

On Mon, May 24, 2021, at 18:10, Sophie Blee-Goldman wrote:

+1 binding

thanks for the KIP
-Sophie

On Mon, May 24, 2021 at 2:02 PM Walker Carlson
 wrote:


+1 (non-binding) from me, Leah

Walker

On Mon, May 24, 2021 at 1:51 PM Leah Thomas



wrote:


Hi,

I'd like to kick-off voting for KIP-741: Change default serde to be

null.

<



https://cwiki.apache.org/confluence/display/KAFKA/KIP-741%3A+Change+default+serde+to+be+null 




The
discussion is linked on the KIP for context.

Cheers,
Leah

Re: [VOTE] KIP-741: Change default serde to be null

[Bruno Cadonna]

Hi Dongjin,

voting for a KIP needs to remain open for at least 72 hours [1]. 
According to the date and time of the first message to this thread 72 
hours havn't passed, yet. Theoretically, there could still be -1 votes 
coming in.


Best,
Bruno

[1] 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-Process



On 25.05.21 13:13, Dongjin Lee wrote:

+1 (non-binding).

As of present:

- Binding: Guozhang Wang, Sophie Blee-Goldman, John Roesler (+3)
- Non-binding: Walker Carlson, Lee Dongjin (+2)

This KIP is now passed.

Thanks,
Dongjin

On Tue, May 25, 2021 at 10:12 AM John Roesler  wrote:


+1 (binding) from me. Thanks for the KIP!
-John

On Mon, May 24, 2021, at 18:10, Sophie Blee-Goldman wrote:

+1 binding

thanks for the KIP
-Sophie

On Mon, May 24, 2021 at 2:02 PM Walker Carlson
 wrote:


+1 (non-binding) from me, Leah

Walker

On Mon, May 24, 2021 at 1:51 PM Leah Thomas



wrote:


Hi,

I'd like to kick-off voting for KIP-741: Change default serde to be

null.

<




https://cwiki.apache.org/confluence/display/KAFKA/KIP-741%3A+Change+default+serde+to+be+null



The
discussion is linked on the KIP for context.

Cheers,
Leah

Re: [VOTE] KIP-741: Change default serde to be null

[Dongjin Lee]

+1 (non-binding).

As of present:

- Binding: Guozhang Wang, Sophie Blee-Goldman, John Roesler (+3)
- Non-binding: Walker Carlson, Lee Dongjin (+2)

This KIP is now passed.

Thanks,
Dongjin

On Tue, May 25, 2021 at 10:12 AM John Roesler  wrote:

> +1 (binding) from me. Thanks for the KIP!
> -John
>
> On Mon, May 24, 2021, at 18:10, Sophie Blee-Goldman wrote:
> > +1 binding
> >
> > thanks for the KIP
> > -Sophie
> >
> > On Mon, May 24, 2021 at 2:02 PM Walker Carlson
> >  wrote:
> >
> > > +1 (non-binding) from me, Leah
> > >
> > > Walker
> > >
> > > On Mon, May 24, 2021 at 1:51 PM Leah Thomas
> 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I'd like to kick-off voting for KIP-741: Change default serde to be
> null.
> > > > <
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-741%3A+Change+default+serde+to+be+null
> > > > >
> > > > The
> > > > discussion is linked on the KIP for context.
> > > >
> > > > Cheers,
> > > > Leah
> > > >
> > >
> >
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Re: [VOTE] KIP-743: Remove config value 0.10.0-2.4 of Streams built-in metrics version config

[Bruno Cadonna]

Hi,

Thank you for discussing and voting on KIP-743!

I close this voting thread. KIP-743 is accepted with 4 binding +1 
(Guozhang, Sophie, John, and myself).


Best,
Bruno


On 25.05.21 05:05, John Roesler wrote:

+1 (binding)

Thanks, Bruno!
-John

On Fri, May 21, 2021, at 10:58, Sophie Blee-Goldman wrote:

+1 (binding)

Thanks Bruno!

On Fri, May 21, 2021 at 8:06 AM Guozhang Wang  wrote:


+1, thanks!

On Fri, May 21, 2021 at 12:45 AM Bruno Cadonna  wrote:


Hi,

I'd like to start a vote on KIP-743 that proposes to remove config value
0.10.0-2.4 from Streams config built.in.metrics.version.

https://cwiki.apache.org/confluence/x/uIfOCg

Best,
Bruno




--
-- Guozhang

[jira] [Created] (KAFKA-12847) Dockerfile needed for kafka system tests needs changes

[Abhijit Mane (Jira)]

Abhijit Mane created KAFKA-12847:


 Summary: Dockerfile needed for kafka system tests needs changes
 Key: KAFKA-12847
 URL: https://issues.apache.org/jira/browse/KAFKA-12847
 Project: Kafka
  Issue Type: Bug
  Components: system tests
Affects Versions: 2.7.1, 2.8.0
 Environment: Issue tested in environments below but is independent of 
h/w arch. or Linux flavor: -
1.) RHEL-8.3 on x86_64 
2.) RHEL-8.3 on IBM Power (ppc64le)
3.) apache/kafka branch tested: trunk (master)
Reporter: Abhijit Mane
 Attachments: Dockerfile.upstream

Hello,

I tried apache/kafka system tests as per documentation: -

(_https://github.com/apache/kafka/tree/trunk/tests#readme_)

=
PROBLEM
~~

1.) As root user, clone kafka github repo and start "kafka system tests"
 # git clone https://github.com/apache/kafka.git
 # cd kafka
 # ./gradlew clean systemTestLibs
 # bash tests/docker/run_tests.sh 
 
2.) Dockerfile issue - 
_https://github.com/apache/kafka/blob/trunk/tests/docker/*Dockerfile*_
 This file has an *UID* entry as shown below: -
---
ARG *UID*="1000"
RUN useradd -u $*UID* ducker

// {color:#de350b}*Error during docker build*{color} => useradd: UID 0 is not 
unique, root user id is 0
---
 I ran everything as root which means the built-in bash environment variable 
'UID' always

resolves to 0 and can't be changed. Hence, the docker build fails.
 
3.) Next, as root, as per README, I ran: -

server:/kafka> *bash tests/docker/run_tests.sh*

The ducker tool builds the container images & switches to user '*ducker*' 
inside the container

& maps kafka root dir ('kafka') from host to '/opt/kafka-dev' in the container.

Ref: *https://github.com/apache/kafka/blob/trunk/tests/docker/ducker-ak*

Ex:  docker run -d *-v "${kafka_dir}:/opt/kafka-dev"* 

This fails as the 'ducker' user has *no write permissions* to create files 
under 'kafka' root dir. Hence, it needs to be made writeable.

// *chmod -R a+w kafka* 
 -- needed as container is run as 'ducker' and needs write access since kafka 
root volume from host is mapped to container as "/opt/kafka-dev" where the 
'ducker' user writes logs
=


 
=
*FIXES needed*
~
 1.) Dockerfile - 
https://github.com/apache/kafka/blob/trunk/tests/docker/Dockerfile
 Change 'UID' to '*UID_DUCKER*'.

This won't conflict with built in bash env. var UID and the docker image build 
should succeed.
---
ARG *UID_DUCKER*="1000"
RUN useradd -u $*UID_DUCKER* ducker

// *{color:#57d9a3}No Error{color}* => No conflict with built-in UID
---

2.) README needs an update where we must ensure the kafka root dir from where 
the tests 
 are launched is writeable to allow the 'ducker' user to create results/logs.
 # chmod -R a+w kafka

With this, I was able to get the docker images built and system tests started 
successfully.
=
 
Also, I wonder whether or not upstream Dockerfile & System tests are part of 
CI/CD and get tested for every PR. If so, this issue should have been caught.

 

*Question to kafka SME*
-
Do you believe this is a valid problem with the Dockerfile and the fix is 
acceptable? 
Please let me know and I am happy to submit a PR with this fix.

Thanks,
Abhijit



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12846) why need this logic in Consumer‘s Fetch logic it should remove?

[yws (Jira)]

yws created KAFKA-12846:
---

 Summary: why need this logic in Consumer‘s Fetch logic   it should 
remove?
 Key: KAFKA-12846
 URL: https://issues.apache.org/jira/browse/KAFKA-12846
 Project: Kafka
  Issue Type: Wish
Reporter: yws
 Fix For: 2.3.0


package: org.apache.kafka.clients.consumer.internals
class: Fetcher

else {
// this case shouldn't usually happen because we 
only send one fetch at a time per partition,
// but it might conceivably happen in some rare 
cases (such as partition leader changes).
// we have to copy to a new list because the old 
one may be immutable
List> newRecords = new 
ArrayList<>(records.size() + currentRecords.size());
newRecords.addAll(currentRecords);
newRecords.addAll(records);
fetched.put(partition, newRecords);
}
recordsRemaining -= records.size();
}


I just cannot think of the case that it will goes to the else logic,  who can 
illustrate it？ it's useless logic in my opinion, looking forward to reply!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)