[jira] [Created] (KAFKA-16196) Cast transform doesn't handle invalid whole value casts gracefully

[Yash Mayya (Jira)]

Yash Mayya created KAFKA-16196:
--

 Summary: Cast transform doesn't handle invalid whole value casts 
gracefully
 Key: KAFKA-16196
 URL: https://issues.apache.org/jira/browse/KAFKA-16196
 Project: Kafka
  Issue Type: Bug
  Components: connect
Reporter: Yash Mayya
Assignee: Yash Mayya


The Cast transform currently doesn't handle invalid whole value casts 
gracefully. A whole value cast is configured like {{{"spec": "int8"}}} as 
opposed to a field level cast like {{{}{"spec": "field1:int8"{.

 

If an invalid field level cast is specified (for instance - {{{}{"spec": 
"field1:invalid"{), this results in a {{ConfigException}} being thrown here 
- 
[https://github.com/apache/kafka/blob/5f410ceb04878ca44d2d007655155b5303a47907/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java#L416]
 which is handled gracefully as a validation error here - 
[https://github.com/apache/kafka/blob/5f410ceb04878ca44d2d007655155b5303a47907/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java#L605-L609]

 

However, invalid whole value casts aren't handled appropriately and result in 
an 
{{IllegalArgumentException}} being thrown, which surfaces as an uncaught 
exception and a {{500 Internal Server Error}} response from the connector 
create / update / config validation REST API endpoint.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

[José Armando García Sancio]

Hi all,

I have updated the KIP to include information on how KRaft controller
automatic joining will work.

Thanks,
-- 
-José

Re: Kafka trunk test & build stability

[Justine Olshan]

It looks like there was some server maintenance that shut down Jenkins.
Upon coming back up, the builds were expired but unable to stop.

They all had similar logs:

Cancelling nested steps due to timeoutCancelling nested steps due to
timeoutBody did not finish within grace period; terminating with
extreme prejudiceBody did not finish within grace period; terminating
with extreme prejudicePausing (Preparing for shutdown)
Resuming build at Thu Jan 25 06:56:23 UTC 2024 after Jenkins restart
Resuming build at Thu Jan 25 09:45:03 UTC 2024 after Jenkins restart
Pausing (Preparing for shutdown)
Resuming build at Thu Jan 25 10:37:41 UTC 2024 after Jenkins
restartTimeout expired 7 hr 39 min agoTimeout expired 7 hr 39 min
agoCancelling nested steps due to timeoutCancelling nested steps due
to timeout*02:37:41*  Waiting for reconnection of builds41 before
proceeding with build*02:37:41*  Waiting for reconnection of builds32
before proceeding with buildStill pausedBody did not finish within
grace period; terminating with extreme prejudiceBody did not finish
within grace period; terminating with extreme prejudice


I forcibly killed the builds running over one day to free resources. I
believe the rest are running as expected now.

Justine

On Thu, Jan 25, 2024 at 10:22 AM Justine Olshan 
wrote:

> Hey folks -- I noticed some builds have been running for a day or more. I
> thought we limited builds to 8 hours. Any ideas why this is happening?
>
>
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/activity/
> I tried to abort the build for PR-15257, and it also still seems to be
> running.
>
> Justine
>
> On Sun, Jan 14, 2024 at 6:25 AM Qichao Chu 
> wrote:
>
>> Hi Divij and all,
>>
>> Regarding the speeding up of the build & de-flaking tests, LinkedIn has
>> done some great work which we probably can borrow ideas from.
>> In the LinkedIn/Kafka repo, we can see one of their most recent PRs
>>  only took < 9
>> min(unit
>> test) + < 12 min (integration-test) + < 9 (code check) = < 30 min to
>> finish
>> all the checks:
>>
>>1. Similar to what David(mumrah) has mentioned/experimented with, the
>>LinkedIn team used GitHub Actions, which displayed the results in a
>> cleaner
>>way directly from GitHub.
>>2. Each top-level package is checked separately to increase the
>>concurrency. To further boost the speed for integration tests, the
>> tests
>>inside one package are divided into sub-groups(A-Z) based on their
>>names(see this job
>> for
>>details).
>>3. Once the tests are running at a smaller granularity with a decent
>>runner, heavy integration tests are less likely to be flaky, and flaky
>>tests are easier to catch.
>>
>>
>> --
>> Qichao
>>
>>
>> On Wed, Jan 10, 2024 at 2:57 PM Divij Vaidya 
>> wrote:
>>
>> > Hey folks
>> >
>> > We seem to have a handle on the OOM issues with the multiple fixes
>> > community members made. In
>> > https://issues.apache.org/jira/browse/KAFKA-16052,
>> > you can see the "before" profile in the description and the "after"
>> profile
>> > in the latest comment to see the difference. To prevent future
>> recurrence,
>> > we have an ongoing solution at
>> https://github.com/apache/kafka/pull/15101
>> > and after that we will start another once to get rid of mockito mocks at
>> > the end of every test suite using a similar extension. Note that this
>> > doesn't solve the flaky test problems in the trunk but it removes the
>> > aspect of build failures due to OOM (one of the many problems).
>> >
>> > To fix the flaky test problem, we probably need to run our tests in a
>> > separate CI environment (like Apache Beam does) instead of sharing the 3
>> > hosts that run our CI with many many other Apache projects. This
>> assumption
>> > is based on the fact that the tests are less flaky when running on
>> laptops
>> > / powerful EC2 machines. One of the avenues to get funding for these
>> > Kafka-only hosts is
>> >
>> >
>> https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/
>> > . I will start the conversation on this one with AWS & Apache Infra in
>> the
>> > next 1-2 months.
>> >
>> > --
>> > Divij Vaidya
>> >
>> >
>> >
>> > On Tue, Jan 9, 2024 at 9:21 PM Colin McCabe  wrote:
>> >
>> > > Sorry, but to put it bluntly, the current build setup isn't good
>> enough
>> > at
>> > > partial rebuilds that build caching would make sense. All Kafka devs
>> have
>> > > had the experience of needing to clean the build directory in order to
>> > get
>> > > a valid build. The scala code esspecially seems to have this issue.
>> > >
>> > > regards,
>> > > Colin
>> > >
>> > >
>> > > On Tue, Jan 2, 2024, at 07:00, Nick Telford wrote:
>> > > > Addendum: I've opened a PR with what I believe are the changes
>> > necessary
>> > > to
>> > > > enable Remote Build Caching, if you choose to go that route:
>> 

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

[José Armando García Sancio]

Hi Jason, Colin and Luke,

I updated the KIP based on your feedback and my comments. Here is what
has changed: 
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=217391519=54=52

Here is an outline of all of the protocol changes:
1. Removed the AddVoterRecord and RemoveVoterRecord control records
and replaced their functionality with VotersRecord instead.
2. VoterUuid in LeaderChangeMessage is now a uuid instead of a int32.
3. Added TimeoutMs field to the AddVoterRequest.
4. Added ClusterId field to the RemoveVoterRequest.
5. Made the ReplicaUuid field in FetchRequest a tagged field.

All of the protocol changes are also documented in this GitHub diff:
https://github.com/apache/kafka/compare/trunk...jsancio:kafka:kraft-membership-protocol

I haven't updated the KIP for adding support for controllers
auto-joining the voters set. I'll do that in the next revision to the
KIP.

Thanks,
-- 
-José

Re: [DISCUSS] KIP-939: Support Participation in 2PC

[Jun Rao]

Hi, Artem,

Thanks for the reply. A few more comments.

20. I am a bit confused by how we set keepPreparedTxn. From the KIP, I got
the following (1) to start 2pc, we call
InitProducerId(keepPreparedTxn=false); (2) when the producer fails and
needs to do recovery, it calls InitProducerId(keepPreparedTxn=true); (3)
Admin.forceTerminateTransaction() calls
InitProducerId(keepPreparedTxn=false).
20.1 In (1), when a producer calls InitProducerId(false) with 2pc enabled,
and there is an ongoing txn, should the server return an error to the
InitProducerId request? If so, what would be the error code?
20.2 How do we distinguish between (1) and (3)? It's the same API call but
(1) doesn't abort ongoing txn and (2) does.
20.3 The usage in (1) seems unintuitive. 2pc implies keeping the ongoing
txn. So, setting keepPreparedTxn to false to start 2pc seems counter
intuitive.

21. TransactionLogValue: Do we need some field to identify whether this is
written for 2PC so that ongoing txn is never auto aborted?

22. "8. InitProducerId(true); TC STATE: Ongoing, ProducerId=42,
ProducerEpoch=MAX-1, PrevProducerId=-1, NextProducerId=73,
NextProducerEpoch=MAX; RESPONSE ProducerId=73, Epoch=MAX-1,
OngoingTxnProducerId=42, OngoingTxnEpoch=MAX-1"
It seems in the above example, Epoch in RESPONSE should be MAX to match
NextProducerEpoch?

23. It's a bit weird that Enable2Pc is driven by a config
while KeepPreparedTxn is from an API param. Should we make them more
consistent since they seem related?

24. "9. Commit; REQUEST: ProducerId=73, ProducerEpoch=MAX-1; TC STATE:
PrepareCommit, ProducerId=42, ProducerEpoch=MAX, PrevProducerId=73,
NextProducerId=85, NextProducerEpoch=0; RESPONSE ProducerId=85, Epoch=0,
When a commit request is sent, it uses the latest ProducerId and
ProducerEpoch."
The step where we use the next produceId to commit an old txn works, but
can be confusing. It's going to be hard for people implementing this new
client protocol to figure out when to use the current or the new producerId
in the EndTxnRequest. One potential way to improve this is to extend
EndTxnRequest with a new field like expectedNextProducerId. Then we can
always use the old produceId in the existing field, but set
expectedNextProducerId to bypass the fencing logic when needed.

Thanks,

Jun



On Mon, Dec 18, 2023 at 2:06 PM Artem Livshits
 wrote:

> Hi Jun,
>
> Thank you for the comments.
>
> > 10. For the two new fields in Enable2Pc and KeepPreparedTxn ...
>
> I added a note that all combinations are valid.  Enable2Pc=false &
> KeepPreparedTxn=true could be potentially useful for backward compatibility
> with Flink, when the new version of Flink that implements KIP-319 tries to
> work with a cluster that doesn't authorize 2PC.
>
> > 11.  InitProducerIdResponse: If there is no ongoing txn, what will
> OngoingTxnProducerId and OngoingTxnEpoch be set?
>
> I added a note that they will be set to -1.  The client then will know that
> there is no ongoing txn and .completeTransaction becomes a no-op (but still
> required before .send is enabled).
>
> > 12. ListTransactionsRequest related changes: It seems those are already
> covered by KIP-994?
>
> Removed from this KIP.
>
> > 13. TransactionalLogValue ...
>
> This is now updated to work on top of KIP-890.
>
> > 14. "Note that the (producerId, epoch) pair that corresponds to the
> ongoing transaction ...
>
> This is now updated to work on top of KIP-890.
>
> > 15. active-transaction-total-time-max : ...
>
> Updated.
>
> > 16. "transaction.two.phase.commit.enable The default would be ‘false’.
> If it’s ‘false’, 2PC functionality is disabled even if the ACL is set ...
>
> Disabling 2PC effectively removes all authorization to use it, hence I
> thought TRANSACTIONAL_ID_AUTHORIZATION_FAILED would be appropriate.
>
> Do you suggest using a different error code for 2PC authorization vs some
> other authorization (e.g. TRANSACTIONAL_ID_2PC_AUTHORIZATION_FAILED) or a
> different code for disabled vs. unauthorised (e.g.
> TWO_PHASE_COMMIT_DISABLED) or both?
>
> > 17. completeTransaction(). We expect this to be only used during
> recovery.
>
> It can also be used if, say, a commit to the database fails and the result
> is inconclusive, e.g.
>
> 1. Begin DB transaction
> 2. Begin Kafka transaction
> 3. Prepare Kafka transaction
> 4. Commit DB transaction
> 5. The DB commit fails, figure out the state of the transaction by reading
> the PreparedTxnState from DB
> 6. Complete Kafka transaction with the PreparedTxnState.
>
> > 18. "either prepareTransaction was called or initTransaction(true) was
> called": "either" should be "neither"?
>
> Updated.
>
> > 19. Since InitProducerId always bumps up the epoch, it creates a
> situation ...
>
> InitProducerId only bumps the producer epoch, the ongoing transaction epoch
> stays the same, no matter how many times the InitProducerId is called
> before the transaction is completed.  Eventually the epoch may overflow,
> and then a new producer id would be allocated, but the 

Re: Kafka trunk test & build stability

[Justine Olshan]

Hey folks -- I noticed some builds have been running for a day or more. I
thought we limited builds to 8 hours. Any ideas why this is happening?

https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/activity/
I tried to abort the build for PR-15257, and it also still seems to be
running.

Justine

On Sun, Jan 14, 2024 at 6:25 AM Qichao Chu  wrote:

> Hi Divij and all,
>
> Regarding the speeding up of the build & de-flaking tests, LinkedIn has
> done some great work which we probably can borrow ideas from.
> In the LinkedIn/Kafka repo, we can see one of their most recent PRs
>  only took < 9 min(unit
> test) + < 12 min (integration-test) + < 9 (code check) = < 30 min to finish
> all the checks:
>
>1. Similar to what David(mumrah) has mentioned/experimented with, the
>LinkedIn team used GitHub Actions, which displayed the results in a
> cleaner
>way directly from GitHub.
>2. Each top-level package is checked separately to increase the
>concurrency. To further boost the speed for integration tests, the tests
>inside one package are divided into sub-groups(A-Z) based on their
>names(see this job
> for
>details).
>3. Once the tests are running at a smaller granularity with a decent
>runner, heavy integration tests are less likely to be flaky, and flaky
>tests are easier to catch.
>
>
> --
> Qichao
>
>
> On Wed, Jan 10, 2024 at 2:57 PM Divij Vaidya 
> wrote:
>
> > Hey folks
> >
> > We seem to have a handle on the OOM issues with the multiple fixes
> > community members made. In
> > https://issues.apache.org/jira/browse/KAFKA-16052,
> > you can see the "before" profile in the description and the "after"
> profile
> > in the latest comment to see the difference. To prevent future
> recurrence,
> > we have an ongoing solution at
> https://github.com/apache/kafka/pull/15101
> > and after that we will start another once to get rid of mockito mocks at
> > the end of every test suite using a similar extension. Note that this
> > doesn't solve the flaky test problems in the trunk but it removes the
> > aspect of build failures due to OOM (one of the many problems).
> >
> > To fix the flaky test problem, we probably need to run our tests in a
> > separate CI environment (like Apache Beam does) instead of sharing the 3
> > hosts that run our CI with many many other Apache projects. This
> assumption
> > is based on the fact that the tests are less flaky when running on
> laptops
> > / powerful EC2 machines. One of the avenues to get funding for these
> > Kafka-only hosts is
> >
> >
> https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/
> > . I will start the conversation on this one with AWS & Apache Infra in
> the
> > next 1-2 months.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Tue, Jan 9, 2024 at 9:21 PM Colin McCabe  wrote:
> >
> > > Sorry, but to put it bluntly, the current build setup isn't good enough
> > at
> > > partial rebuilds that build caching would make sense. All Kafka devs
> have
> > > had the experience of needing to clean the build directory in order to
> > get
> > > a valid build. The scala code esspecially seems to have this issue.
> > >
> > > regards,
> > > Colin
> > >
> > >
> > > On Tue, Jan 2, 2024, at 07:00, Nick Telford wrote:
> > > > Addendum: I've opened a PR with what I believe are the changes
> > necessary
> > > to
> > > > enable Remote Build Caching, if you choose to go that route:
> > > > https://github.com/apache/kafka/pull/15109
> > > >
> > > > On Tue, 2 Jan 2024 at 14:31, Nick Telford 
> > > wrote:
> > > >
> > > >> Hi everyone,
> > > >>
> > > >> Regarding building a "dependency graph"... Gradle already has this
> > > >> information, albeit fairly coarse-grained. You might be able to get
> > some
> > > >> considerable improvement by configuring the Gradle Remote Build
> Cache.
> > > It
> > > >> looks like it's currently disabled explicitly:
> > > >> https://github.com/apache/kafka/blob/trunk/settings.gradle#L46
> > > >>
> > > >> The trick is to have trunk builds write to the cache, and PR builds
> > only
> > > >> read from it. This way, any PR based on trunk should be able to
> cache
> > > not
> > > >> only the compilation, but also the tests from dependent modules that
> > > >> haven't changed (e.g. for a PR that only touches the connect/streams
> > > >> modules).
> > > >>
> > > >> This would probably be preferable to having to hand-maintain some
> > > >> rules/dependency graph in the CI configuration, and it's quite
> > > >> straight-forward to configure.
> > > >>
> > > >> Bonus points if the Remote Build Cache is readable publicly,
> enabling
> > > >> contributors to benefit from it locally.
> > > >>
> > > >> Regards,
> > > >> Nick
> > > >>
> > > >> On Tue, 2 Jan 2024 at 13:00, Lucas Brutschy  > > .invalid>
> > > >> wrote:
> > > >>
> > > >>> Thanks for all the work that has 

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

[Mickael Maison]

Hi Luke,

The reason vary for each plugin, I've added details to most plugins in
the table.
The plugins without an explanation are all from Streams. I admit I
don't know these interfaces enough to decide if it makes sense making
them closeable and instrumenting them. It would be nice to get some
input from Streams contributors to know.

Thanks,
Mickael

On Thu, Jan 25, 2024 at 5:50 PM Mickael Maison  wrote:
>
> Hi Tom,
>
> Thanks for taking a look at the KIP!
>
> 1. Yes I considered several names (see the previous messages in the
> discussion). KIP-608, which this KIP superseeds, used "monitor()" for
> the method name. I find "withMetrics()" to be nicer due to the way the
> method should be used. That said, I'm not attached to the name so if
> more people prefer "monitor()", or can come up with a better name, I'm
> happy to make the change. I updated the javadoc to clarify the usage
> and mention when to close the PluginMetrics instance.
>
> 2. Yes I added a note to the PluginMetrics interface
>
> 3. I used this exception to follow the behavior of Metrics.addMetric()
> which throws IllegalArgumentException if a metric with the same name
> already exist.
>
> 4. I added details to the javadoc
>
> Thanks,
> Mickael
>
>
> On Thu, Jan 25, 2024 at 10:32 AM Luke Chen  wrote:
> >
> > Hi Mickael,
> >
> > Thanks for the KIP.
> > The motivation and solution makes sense to me.
> >
> > Just one question:
> > If we could extend `closable` for Converter plugin, why don't we do that
> > for the "Unsupported Plugins" without close method?
> > I don't say we must do that in this KIP, but maybe you could add the reason
> > in the "rejected alternatives".
> >
> > Thanks.
> > Luke
> >
> > On Thu, Jan 25, 2024 at 3:46 PM Slathia p  wrote:
> >
> > > Hi Team,
> > >
> > >
> > >
> > > Greetings,
> > >
> > >
> > >
> > > Apologies for the delay in reply as I was down with flu.
> > >
> > >
> > >
> > > We actually reached out to you for IT/ SAP/ Oracle/ Infor / Microsoft
> > > “VOTEC IT SERVICE PARTNERSHIP”  “IT SERVICE OUTSOURCING” “ “PARTNER 
> > > SERVICE
> > > SUBCONTRACTING”
> > >
> > >
> > >
> > > We have very attractive newly introduce reasonably price PARTNER IT
> > > SERVICE ODC SUBCONTRACTING MODEL in USA, Philippines, India and Singapore
> > > etc with White Label Model.
> > >
> > >
> > >
> > > Our LOW COST IT SERVICE ODC MODEL eliminate the cost of expensive employee
> > > payroll, Help partner to get profit more than 50% on each project.. ..We
> > > really mean it.
> > >
> > >
> > >
> > > We are already working with platinum partner like NTT DATA, NEC Singapore,
> > > Deloitte, Hitachi consulting. ACCENTURE, Abeam Singapore etc.
> > >
> > >
> > >
> > > Are u keen to understand VOTEC IT SERVICE MODEL PARTNERSHIP offerings?
> > >
> > >
> > >
> > > Let us know your availability this week OR Next week?? We can arrange
> > > discussion with Partner Manager.
> > > > On 01/25/2024 9:56 AM +08 Tom Bentley  wrote:
> > > >
> > > >
> > > > Hi Mickael,
> > > >
> > > > Thanks for the KIP! I can tell a lot of thought went into this. I have a
> > > > few comments, but they're all pretty trivial and aimed at making the
> > > > correct use of this API clearer to implementors.
> > > >
> > > > 1. Configurable and Reconfigurable both use a verb in the imperative 
> > > > mood
> > > > for their method name. Monitorable doesn't, which initially seemed a bit
> > > > inconsistent to me, but I think your intention is to allow plugins to
> > > > merely retain a reference to the PluginMetrics, and allow registering
> > > > metrics at any later point? If that's the case you could add something
> > > like
> > > > "Plugins can register and unregister metrics using the given
> > > PluginMetrics
> > > > at any point in their lifecycle prior to their close method being 
> > > > called"
> > > > to the javadoc to make clear how this can be used.
> > > > 2. I assume PluginMetrics will be thread-safe? We should document that 
> > > > as
> > > > part of the contract.
> > > > 3. I don't think IAE is quite right for duplicate metrics. In this case
> > > the
> > > > arguments themselves are fine, it's the current state of the
> > > PluginMetrics
> > > > which causes the problem. If the earlier point about plugins being
> > > allowed
> > > > to register and unregister metrics at any point is correct then this
> > > > exception could be thrown after configuration time. That being the case 
> > > > I
> > > > think a new exception type might be clearer.
> > > > 4. You define some semantics for PluginMetrics.close(): It might be a
> > > good
> > > > idea to override the inherited method and add that as javadoc.
> > > > 5. You say "It will be the responsibility of the plugin that creates
> > > > metrics to call close() of the PluginMetrics instance they were given to
> > > > remove their metrics." But you don't provide any guidance to users about
> > > > when they need to do this. I guess that they should be doing this in
> > > their
> > > > plugin's close method (and 

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

[Mickael Maison]

Hi Tom,

Thanks for taking a look at the KIP!

1. Yes I considered several names (see the previous messages in the
discussion). KIP-608, which this KIP superseeds, used "monitor()" for
the method name. I find "withMetrics()" to be nicer due to the way the
method should be used. That said, I'm not attached to the name so if
more people prefer "monitor()", or can come up with a better name, I'm
happy to make the change. I updated the javadoc to clarify the usage
and mention when to close the PluginMetrics instance.

2. Yes I added a note to the PluginMetrics interface

3. I used this exception to follow the behavior of Metrics.addMetric()
which throws IllegalArgumentException if a metric with the same name
already exist.

4. I added details to the javadoc

Thanks,
Mickael


On Thu, Jan 25, 2024 at 10:32 AM Luke Chen  wrote:
>
> Hi Mickael,
>
> Thanks for the KIP.
> The motivation and solution makes sense to me.
>
> Just one question:
> If we could extend `closable` for Converter plugin, why don't we do that
> for the "Unsupported Plugins" without close method?
> I don't say we must do that in this KIP, but maybe you could add the reason
> in the "rejected alternatives".
>
> Thanks.
> Luke
>
> On Thu, Jan 25, 2024 at 3:46 PM Slathia p  wrote:
>
> > Hi Team,
> >
> >
> >
> > Greetings,
> >
> >
> >
> > Apologies for the delay in reply as I was down with flu.
> >
> >
> >
> > We actually reached out to you for IT/ SAP/ Oracle/ Infor / Microsoft
> > “VOTEC IT SERVICE PARTNERSHIP”  “IT SERVICE OUTSOURCING” “ “PARTNER SERVICE
> > SUBCONTRACTING”
> >
> >
> >
> > We have very attractive newly introduce reasonably price PARTNER IT
> > SERVICE ODC SUBCONTRACTING MODEL in USA, Philippines, India and Singapore
> > etc with White Label Model.
> >
> >
> >
> > Our LOW COST IT SERVICE ODC MODEL eliminate the cost of expensive employee
> > payroll, Help partner to get profit more than 50% on each project.. ..We
> > really mean it.
> >
> >
> >
> > We are already working with platinum partner like NTT DATA, NEC Singapore,
> > Deloitte, Hitachi consulting. ACCENTURE, Abeam Singapore etc.
> >
> >
> >
> > Are u keen to understand VOTEC IT SERVICE MODEL PARTNERSHIP offerings?
> >
> >
> >
> > Let us know your availability this week OR Next week?? We can arrange
> > discussion with Partner Manager.
> > > On 01/25/2024 9:56 AM +08 Tom Bentley  wrote:
> > >
> > >
> > > Hi Mickael,
> > >
> > > Thanks for the KIP! I can tell a lot of thought went into this. I have a
> > > few comments, but they're all pretty trivial and aimed at making the
> > > correct use of this API clearer to implementors.
> > >
> > > 1. Configurable and Reconfigurable both use a verb in the imperative mood
> > > for their method name. Monitorable doesn't, which initially seemed a bit
> > > inconsistent to me, but I think your intention is to allow plugins to
> > > merely retain a reference to the PluginMetrics, and allow registering
> > > metrics at any later point? If that's the case you could add something
> > like
> > > "Plugins can register and unregister metrics using the given
> > PluginMetrics
> > > at any point in their lifecycle prior to their close method being called"
> > > to the javadoc to make clear how this can be used.
> > > 2. I assume PluginMetrics will be thread-safe? We should document that as
> > > part of the contract.
> > > 3. I don't think IAE is quite right for duplicate metrics. In this case
> > the
> > > arguments themselves are fine, it's the current state of the
> > PluginMetrics
> > > which causes the problem. If the earlier point about plugins being
> > allowed
> > > to register and unregister metrics at any point is correct then this
> > > exception could be thrown after configuration time. That being the case I
> > > think a new exception type might be clearer.
> > > 4. You define some semantics for PluginMetrics.close(): It might be a
> > good
> > > idea to override the inherited method and add that as javadoc.
> > > 5. You say "It will be the responsibility of the plugin that creates
> > > metrics to call close() of the PluginMetrics instance they were given to
> > > remove their metrics." But you don't provide any guidance to users about
> > > when they need to do this. I guess that they should be doing this in
> > their
> > > plugin's close method (and that's why you're only adding Monitorable to
> > > plugins which implement Closeable and AutoCloseable), but if that's the
> > > case then let's say so in the Javadoc.
> > >
> > > Thanks again,
> > >
> > > Tom
> > >
> > > On Wed, 13 Dec 2023 at 06:09, Mickael Maison 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I've not received any feedback since I updated the KIP.
> > > > I'll wait a few more days and if there's no further feedback I'll
> > start a
> > > > vote.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Tue, Nov 7, 2023 at 6:29 PM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > A bit later than initially 

[jira] [Resolved] (KAFKA-16003) The znode /config/topics is not updated during KRaft migration in "dual-write" mode

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-16003.

Fix Version/s: 3.8.0
   Resolution: Fixed

> The znode /config/topics is not updated during KRaft migration in 
> "dual-write" mode
> ---
>
> Key: KAFKA-16003
> URL: https://issues.apache.org/jira/browse/KAFKA-16003
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 3.6.1
>Reporter: Paolo Patierno
>Assignee: Mickael Maison
>Priority: Major
> Fix For: 3.8.0
>
>
> I tried the following scenario ...
> I have a ZooKeeper-based cluster and create a my-topic-1 topic (without 
> specifying any specific configuration for it). The correct znodes are created 
> under /config/topics and /brokers/topics.
> I start a migration to KRaft but not moving forward from "dual write" mode. 
> While in this mode, I create a new my-topic-2 topic (still without any 
> specific config). I see that a new znode is created under /brokers/topics but 
> NOT under /config/topics. It seems that the KRaft controller is not updating 
> this information in ZooKeeper during the dual-write. The controller log shows 
> that the write to ZooKeeper was done, but not everything I would say:
> {code:java}
> 2023-12-13 10:23:26,229 TRACE [KRaftMigrationDriver id=3] Create Topic 
> my-topic-2, ID Macbp8BvQUKpzmq2vG_8dA. Transitioned migration state from 
> ZkMigrationLeadershipState{kraftControllerId=3, kraftControllerEpoch=7, 
> kraftMetadataOffset=445, kraftMetadataEpoch=7, 
> lastUpdatedTimeMs=1702462785587, migrationZkVersion=236, controllerZkEpoch=3, 
> controllerZkVersion=3} to ZkMigrationLeadershipState{kraftControllerId=3, 
> kraftControllerEpoch=7, kraftMetadataOffset=445, kraftMetadataEpoch=7, 
> lastUpdatedTimeMs=1702462785587, migrationZkVersion=237, controllerZkEpoch=3, 
> controllerZkVersion=3} 
> (org.apache.kafka.metadata.migration.KRaftMigrationDriver) 
> [controller-3-migration-driver-event-handler]
> 2023-12-13 10:23:26,229 DEBUG [KRaftMigrationDriver id=3] Made the following 
> ZK writes when handling KRaft delta: {CreateTopic=1} 
> (org.apache.kafka.metadata.migration.KRaftMigrationDriver) 
> [controller-3-migration-driver-event-handler] {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16195) metadata.log.dir failure shouldn't shutdown broker in ZK mode

[Gaurav Narula (Jira)]

Gaurav Narula created KAFKA-16195:
-

 Summary: metadata.log.dir failure shouldn't shutdown broker in ZK 
mode
 Key: KAFKA-16195
 URL: https://issues.apache.org/jira/browse/KAFKA-16195
 Project: Kafka
  Issue Type: Bug
  Components: jbod
Affects Versions: 3.7.0
Reporter: Gaurav Narula


{{ReplicaManager::handleLogDirFailure}} shuts down the broker if the directory 
that has failed is the same as {{config.metadataLogDir}} 
[[0]|https://github.com/apache/kafka/blob/f1924353126fdf6aad2ba1f8d0c22dade59360b1/core/src/main/scala/kafka/server/ReplicaManager.scala#L2601].
 If unset, it defaults to the first log directory configured even in ZK mode.

The check should be amended to ensure the broker is either in Migration mode or 
Kraft mode and the broker shouldn't shut down when running in ZK mode.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16194) KafkaConsumer.groupMetadata() should be correct when first records are returned

[David Jacot (Jira)]

David Jacot created KAFKA-16194:
---

 Summary: KafkaConsumer.groupMetadata() should be correct when 
first records are returned
 Key: KAFKA-16194
 URL: https://issues.apache.org/jira/browse/KAFKA-16194
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot


The following code returns records before the group metadata is updated. This 
fails the first transactions ever run by the Producer/Consumer.

 
{code:java}
Producer txnProducer = new KafkaProducer<>(txnProducerProps);
Consumer consumer = new KafkaConsumer<>(consumerProps);

txnProducer.initTransactions();
System.out.println("Init transactions called");

try {
txnProducer.beginTransaction();
System.out.println("Begin transactions called");

consumer.subscribe(Collections.singletonList("input"));
System.out.println("Consumer subscribed to topic -> KIP848-topic-2 ");

ConsumerRecords records = 
consumer.poll(Duration.ofSeconds(10));
System.out.println("Returned " + records.count() + " records.");

// Process and send txn messages.
for (ConsumerRecord processedRecord : records) {
txnProducer.send(new ProducerRecord<>("output", processedRecord.key(), 
"Processed: " + processedRecord.value()));
}

ConsumerGroupMetadata groupMetadata = consumer.groupMetadata();
System.out.println("Group metadata inside test" + groupMetadata);

Map offsetsToCommit = new HashMap<>();
for (ConsumerRecord record : records) {
offsetsToCommit.put(new TopicPartition(record.topic(), 
record.partition()),
new OffsetAndMetadata(record.offset() + 1));
}
System.out.println("Offsets to commit" + offsetsToCommit);
// Send offsets to transaction with ConsumerGroupMetadata.
txnProducer.sendOffsetsToTransaction(offsetsToCommit, groupMetadata);
System.out.println("Send offsets to transaction done");

// Commit the transaction.
txnProducer.commitTransaction();
System.out.println("Commit transaction done");
} catch (ProducerFencedException | OutOfOrderSequenceException | 
AuthorizationException e) {
e.printStackTrace();
txnProducer.close();
} catch (KafkaException e) {
e.printStackTrace();
txnProducer.abortTransaction();
} finally {
txnProducer.close();
consumer.close();
} {code}
The issue seems to be that while it waits in `poll`, the event to update the 
group metadata is not processed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

[Luke Chen]

Hi Mickael,

Thanks for the KIP.
The motivation and solution makes sense to me.

Just one question:
If we could extend `closable` for Converter plugin, why don't we do that
for the "Unsupported Plugins" without close method?
I don't say we must do that in this KIP, but maybe you could add the reason
in the "rejected alternatives".

Thanks.
Luke

On Thu, Jan 25, 2024 at 3:46 PM Slathia p  wrote:

> Hi Team,
>
>
>
> Greetings,
>
>
>
> Apologies for the delay in reply as I was down with flu.
>
>
>
> We actually reached out to you for IT/ SAP/ Oracle/ Infor / Microsoft
> “VOTEC IT SERVICE PARTNERSHIP”  “IT SERVICE OUTSOURCING” “ “PARTNER SERVICE
> SUBCONTRACTING”
>
>
>
> We have very attractive newly introduce reasonably price PARTNER IT
> SERVICE ODC SUBCONTRACTING MODEL in USA, Philippines, India and Singapore
> etc with White Label Model.
>
>
>
> Our LOW COST IT SERVICE ODC MODEL eliminate the cost of expensive employee
> payroll, Help partner to get profit more than 50% on each project.. ..We
> really mean it.
>
>
>
> We are already working with platinum partner like NTT DATA, NEC Singapore,
> Deloitte, Hitachi consulting. ACCENTURE, Abeam Singapore etc.
>
>
>
> Are u keen to understand VOTEC IT SERVICE MODEL PARTNERSHIP offerings?
>
>
>
> Let us know your availability this week OR Next week?? We can arrange
> discussion with Partner Manager.
> > On 01/25/2024 9:56 AM +08 Tom Bentley  wrote:
> >
> >
> > Hi Mickael,
> >
> > Thanks for the KIP! I can tell a lot of thought went into this. I have a
> > few comments, but they're all pretty trivial and aimed at making the
> > correct use of this API clearer to implementors.
> >
> > 1. Configurable and Reconfigurable both use a verb in the imperative mood
> > for their method name. Monitorable doesn't, which initially seemed a bit
> > inconsistent to me, but I think your intention is to allow plugins to
> > merely retain a reference to the PluginMetrics, and allow registering
> > metrics at any later point? If that's the case you could add something
> like
> > "Plugins can register and unregister metrics using the given
> PluginMetrics
> > at any point in their lifecycle prior to their close method being called"
> > to the javadoc to make clear how this can be used.
> > 2. I assume PluginMetrics will be thread-safe? We should document that as
> > part of the contract.
> > 3. I don't think IAE is quite right for duplicate metrics. In this case
> the
> > arguments themselves are fine, it's the current state of the
> PluginMetrics
> > which causes the problem. If the earlier point about plugins being
> allowed
> > to register and unregister metrics at any point is correct then this
> > exception could be thrown after configuration time. That being the case I
> > think a new exception type might be clearer.
> > 4. You define some semantics for PluginMetrics.close(): It might be a
> good
> > idea to override the inherited method and add that as javadoc.
> > 5. You say "It will be the responsibility of the plugin that creates
> > metrics to call close() of the PluginMetrics instance they were given to
> > remove their metrics." But you don't provide any guidance to users about
> > when they need to do this. I guess that they should be doing this in
> their
> > plugin's close method (and that's why you're only adding Monitorable to
> > plugins which implement Closeable and AutoCloseable), but if that's the
> > case then let's say so in the Javadoc.
> >
> > Thanks again,
> >
> > Tom
> >
> > On Wed, 13 Dec 2023 at 06:09, Mickael Maison 
> > wrote:
> >
> > > Hi,
> > >
> > > I've not received any feedback since I updated the KIP.
> > > I'll wait a few more days and if there's no further feedback I'll
> start a
> > > vote.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Tue, Nov 7, 2023 at 6:29 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > A bit later than initially planned I finally restarted looking at
> this
> > > KIP.
> > > >
> > > > I made a few significant changes to the proposed APIs.
> > > > I considered Chris' idea of automatically removing metrics but
> decided
> > > > to leave that responsibility to the plugins. All plugins that will
> > > > support this feature have close/stop methods and will need to close
> > > > their PluginMetrics instance. This simplifies the required changes a
> > > > lot and I think it's not a big ask on users implementing plugins.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Tue, May 30, 2023 at 11:32 AM Mickael Maison
> > > >  wrote:
> > > > >
> > > > > Hi Jorge,
> > > > >
> > > > > There are a few issues with the current proposal. Once 3.5 is out,
> I
> > > > > plan to start looking at this again.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Mon, May 15, 2023 at 3:19 PM Jorge Esteban Quilcate Otoya
> > > > >  wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > Just to check the status of this KIP as it looks very useful. I
> can
> > 

[jira] [Resolved] (KAFKA-7957) Flaky Test DynamicBrokerReconfigurationTest#testMetricsReporterUpdate

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-7957.
---
Resolution: Fixed

> Flaky Test DynamicBrokerReconfigurationTest#testMetricsReporterUpdate
> -
>
> Key: KAFKA-7957
> URL: https://issues.apache.org/jira/browse/KAFKA-7957
> Project: Kafka
>  Issue Type: Bug
>  Components: core, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Mickael Maison
>Priority: Blocker
>  Labels: flaky-test
> Fix For: 3.8.0
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all 
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/18/]
> {quote}java.lang.AssertionError: Messages not sent at 
> kafka.utils.TestUtils$.fail(TestUtils.scala:356) at 
> kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:766) at 
> kafka.server.DynamicBrokerReconfigurationTest.startProduceConsume(DynamicBrokerReconfigurationTest.scala:1270)
>  at 
> kafka.server.DynamicBrokerReconfigurationTest.testMetricsReporterUpdate(DynamicBrokerReconfigurationTest.scala:650){quote}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[ziming deng]

The KIP is accepted with 4 binding votes (Chirs, Divij Vaidya, David and Luke) 
and 4 unbinding votes(Kamal, Federico, Andrew and Николай Ижиков).
Thank you all !

> On Jan 24, 2024, at 20:44, Николай Ижиков  wrote:
> 
> +1 (non-binding)
> 
>> 24 янв. 2024 г., в 12:48, Chris Egerton  написал(а):
>> 
>> Thanks Ziming! +1 (binding)
>> 
>> On Wed, Jan 24, 2024, 03:23 ziming deng  wrote:
>> 
>>> Thank you for reminding this, David,
>>> 
>>> I have mentioned this in the [Compatibility] section as a following work.
>>> 
>>> --,
>>> Best,
>>> Ziming
>>> 
 On Jan 23, 2024, at 15:17, David Jacot 
>>> wrote:
 
 Hi Chris, Ziming,
 
 Thanks for the clarification. I am glad that it does not impact the tool.
 It may be worth adding a note about it in the KIP to avoid the same
 question in the future.
 
 Otherwise, I am +1 (binding). Thanks for driving this!
 
 Best,
 David
 
 On Tue, Jan 23, 2024 at 6:07 AM ziming deng 
 wrote:
 
> Hello David,
> 
> Thanks for reminding this, as Chirs explained, the tools I’m trying to
> update only support set/delete configs, and I’m just make a way for
> append/subtract configs in the future, so this would not be affected by
> KAFKA-10140, and it would be a little overkill to support
>>> append/subtract
> configs or solve KAFKA-10140 here, so let’s leave it right now, I'm
>>> happy
> to pick it after finishing this KIP.
> 
> --,
> Ziming
> 
>> On Jan 22, 2024, at 18:23, David Jacot 
> wrote:
>> 
>> Hi Ziming,
>> 
>> Thanks for driving this. I wanted to bring KAFKA-10140
>>  to your attention.
> It
>> looks like the incremental API does not work for configuring plugins. I
>> think that we need to cover this in the KIP.
>> 
>> Best,
>> David
>> 
>> On Mon, Jan 22, 2024 at 10:13 AM Andrew Schofield <
>> andrew_schofield_j...@outlook.com> wrote:
>> 
>>> +1 (non-binding)
>>> 
>>> Thanks,
>>> Andrew
>>> 
 On 22 Jan 2024, at 07:29, Federico Valeri 
> wrote:
 
 +1 (non binding)
 
 Thanks.
 
 On Mon, Jan 22, 2024 at 7:03 AM Luke Chen  wrote:
> 
> Hi Ziming,
> 
> +1(binding) from me.
> 
> Thanks.
> Luke
> 
> On Mon, Jan 22, 2024 at 11:50 AM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
> 
>> +1 (non-binding)
>> 
>> On Mon, Jan 22, 2024 at 8:34 AM ziming deng <
> dengziming1...@gmail.com>
>> wrote:
>> 
>>> Hello everyone,
>>> I'd like to initiate a vote for KIP-1011.
>>> This KIP is about replacing alterConfigs with
> incrementalAlterConfigs
>>> when updating broker configs using kafka-configs.sh, this is
>>> similar
>>> to
>>> what we have done in KIP-894.
>>> 
>>> KIP link:
>>> KIP-1011: Use incrementalAlterConfigs when updating broker configs
> by
>>> kafka-configs.sh - Apache Kafka - Apache Software Foundation
>>> <
>>> 
> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
 
>>> cwiki.apache.org
>>> <
>>> 
> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
 
>>> [image: favicon.ico]
>>> <
>>> 
> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
 
>>> <
>>> 
> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
 
>>> 
>>> Discussion thread:
>>> 
>>> 
>>> lists.apache.org
>>> >> >> >> 
>>> 
>>> --,
>>> Best,
>>> Ziming
>>> 
>>> 
>>> 
> 
> 
>>> 
>>> 
>