Re: KRaft Observer Nodes

[Paolo Patierno]

Hi Sanaa,
yes it's enough NOT having the "controller" role to be just an "observer".
As soon as your node has "controller" role configured (in addition to
"broker" or alone) it acts as a "voter" and then being a leader or
"follower" based on the election result.

Thanks
Paolo

On Mon, 25 Mar 2024, 14:25 Sanaa Syed, 
wrote:

> Hi Paolo,
>
> If I'm understanding correctly, a kafka broker (aside from the controller
> quorum) can be the observer? Do we need to set the process role to anything
> in this case to indicate that we want a broker to be an observer?
>
> On Sun, Mar 24, 2024 at 3:01 PM Paolo Patierno 
> wrote:
>
> > Hi Sanaa,
> > in KRaft mode there is the role of "observer" which is typically taken by
> > brokers as part of the data plane. The broker/observer can discover which
> > node is the leader/active controller and fetches the metadata from it (as
> > the "followers" in the KRaft quorum), but cannot vote and cannot become
> > leader.
> > AFAIK it's allowing to change its role at some point if needed to be
> also a
> > controller or anyway taking part at the quorum by also having the
> metadata
> > topic already replicated.
> >
> > Thanks
> > Paolo
> >
> > On Sun, 24 Mar 2024, 19:05 Sanaa Syed, 
> > wrote:
> >
> > > Hello,
> > >
> > > Currently in Zookeeper, there is the concept of an observer/dummy node
> > that
> > > can be used when scaling up a cluster (
> > > https://zookeeper.apache.org/doc/r3.4.13/zookeeperObservers.html).
> > >
> > > Is there an equivalent of an observer node in KRaft mode? Asking
> because
> > we
> > > use the observer node for our stretched Kafka Clusters (6 zookeeper
> nodes
> > > across 3 clusters) and would like to find a direct replacement or come
> up
> > > with an alternative.
> > >
> > > Thank you,
> > > Sanaa
> > >
> >
>
>
> --
> Sanaa Syed
>

Re: KRaft Observer Nodes

[Paolo Patierno]

Hi Sanaa,
in KRaft mode there is the role of "observer" which is typically taken by
brokers as part of the data plane. The broker/observer can discover which
node is the leader/active controller and fetches the metadata from it (as
the "followers" in the KRaft quorum), but cannot vote and cannot become
leader.
AFAIK it's allowing to change its role at some point if needed to be also a
controller or anyway taking part at the quorum by also having the metadata
topic already replicated.

Thanks
Paolo

On Sun, 24 Mar 2024, 19:05 Sanaa Syed, 
wrote:

> Hello,
>
> Currently in Zookeeper, there is the concept of an observer/dummy node that
> can be used when scaling up a cluster (
> https://zookeeper.apache.org/doc/r3.4.13/zookeeperObservers.html).
>
> Is there an equivalent of an observer node in KRaft mode? Asking because we
> use the observer node for our stretched Kafka Clusters (6 zookeeper nodes
> across 3 clusters) and would like to find a direct replacement or come up
> with an alternative.
>
> Thank you,
> Sanaa
>

Re: [VOTE] 3.7.0 RC4

[Paolo Patierno]

+1 (non-binding). I used the staged binaries with Scala 2.13 and mostly
focused on the ZooKeeper to KRaft migration with multiple tests. Everything
works fine.

Thanks
Paolo

On Mon, 12 Feb 2024, 22:06 Jakub Scholz,  wrote:

> +1 (non-binding). I used the staged binaries with Scala 2.13 and the staged
> Maven artifacts to run my tests. All seems to work fine. Thanks.
>
> Jakub
>
> On Fri, Feb 9, 2024 at 4:20 PM Stanislav Kozlovski
>  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate we are considering for release of Apache
> Kafka
> > 3.7.0.
> >
> > Major changes include:
> > - Early Access to KIP-848 - the next generation of the consumer rebalance
> > protocol
> > - Early Access to KIP-858: Adding JBOD support to KRaft
> > - KIP-714: Observability into Client metrics via a standardized interface
> >
> > Release notes for the 3.7.0 release:
> >
> >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc4/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Thursday, February 15th, 9AM PST
> ***
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc4/
> >
> > * Docker release artifact to be voted upon:
> > apache/kafka:3.7.0-rc4
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc4/javadoc/
> >
> > * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> > https://github.com/apache/kafka/releases/tag/3.7.0-rc4
> >
> > * Documentation:
> > https://kafka.apache.org/37/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/37/protocol.html
> >
> > * Successful Jenkins builds for the 3.7 branch:
> >
> > Unit/integration tests: I am in the process of running and analyzing
> these.
> > System tests: I am in the process of running these.
> >
> > Expect a follow-up over the weekend
> >
> > * Successful Docker Image Github Actions Pipeline for 3.7 branch:
> > Docker Build Test Pipeline:
> > https://github.com/apache/kafka/actions/runs/7845614846
> >
> > /**
> >
> > Best,
> > Stanislav
> >
>

Re: [VOTE] 3.7.0 RC2

[Paolo Patierno]

m
> >>> > > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >>> > > > > > - Checked signatures and checksums
> >>> > > > > > - Ran the quickstart using both:
> >>> > > > > > - - The kafka_2.13-3.7.0.tgz artifact from
> >>> > > > > > https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >>> with
> >>> > > Java
> >>> > > > > 11
> >>> > > > > > and Scala 13 in KRaft mode
> >>> > > > > > - - Our shiny new broker Docker image, apache/kafka:3.7.0-rc2
> >>> > > > > > - Ran all unit tests
> >>> > > > > > - Ran all integration tests for Connect and MM2
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > I found two minor areas for concern:
> >>> > > > > >
> >>> > > > > > 1. (Possibly a blocker)
> >>> > > > > > When running the quickstart, I noticed this ERROR-level log
> >>> message
> >>> > > > being
> >>> > > > > > emitted frequently (not not every time) when I killed my
> console
> >>> > > > consumer
> >>> > > > > > via ctrl-C:
> >>> > > > > >
> >>> > > > > > > [2024-01-12 11:00:31,088] ERROR [Consumer
> >>> > > clientId=console-consumer,
> >>> > > > > > groupId=console-consumer-74388] Unable to find
> >>> FetchSessionHandler
> >>> > > for
> >>> > > > > node
> >>> > > > > > 1. Ignoring fetch response
> >>> > > > > > (org.apache.kafka.clients.consumer.internals.AbstractFetch)
> >>> > > > > >
> >>> > > > > > I see that this error message is already reported in
> >>> > > > > > https://issues.apache.org/jira/browse/KAFKA-16029. I think
> we
> >>> > should
> >>> > > > > > prioritize fixing it for this release. I know it's probably
> >>> benign
> >>> > > but
> >>> > > > > it's
> >>> > > > > > really not a good look for us when basic operations log error
> >>> > > messages,
> >>> > > > > and
> >>> > > > > > it may give new users some headaches.
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > 2. (Probably not a blocker)
> >>> > > > > > The following unit tests failed the first time around, and
> all of
> >>> > > them
> >>> > > > > > passed the second time I ran them:
> >>> > > > > >
> >>> > > > > > - (clients)
> >>> > > > >
> ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
> >>> > > > > > - (clients) SelectorTest.testConnectionsByClientMetric()
> >>> > > > > > - (clients) Tls13SelectorTest.testConnectionsByClientMetric()
> >>> > > > > > - (connect)
> >>> > > TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound
> >>> > > > (I
> >>> > > > > > thought I fixed this one! 郎郎)
> >>> > > > > > - (core)
> ProducerIdManagerTest.testUnrecoverableErrors(Errors)[2]
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > Thanks again for your work on this release, and
> congratulations
> >>> to
> >>> > > > Kafka
> >>> > > > > > Streams for having zero flaky unit tests during my
> >>> > > highly-experimental
> >>> > > > > > single laptop run!
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > Cheers,
> >>> > > > > >
> >>> > > > > > Chris
> >>> > > > > >
> >>> > > > > > On Thu, Jan 11, 2024 at 1:33 PM Stanislav Kozlovski
> >>> > > > > >  wrote:
> >>> > > > > >
> >>> > > > > > > Hello Kafka users, developers, and client-developers,
> >>> > > > > > >
> >>> > > > > > > This is the first candidate for release of Apache Kafka
> 3.7.0.
> >>> > > > > > >
> >>> > > > > > > Note it's named "RC2" because I had a few "failed" RCs
> that I
> >>> had
> >>> > > > > > > cut/uploaded but ultimately had to scrap prior to
> announcing
> >>> due
> >>> > to
> >>> > > > new
> >>> > > > > > > blockers arriving before I could even announce them.
> >>> > > > > > >
> >>> > > > > > > Further - I haven't yet been able to set up the system
> tests
> >>> > > > > successfully.
> >>> > > > > > > And the integration/unit tests do have a few failures that
> I
> >>> have
> >>> > > to
> >>> > > > > spend
> >>> > > > > > > time triaging. I would appreciate any help in case anyone
> >>> notices
> >>> > > any
> >>> > > > > tests
> >>> > > > > > > failing that they're subject matters experts in. Expect me
> to
> >>> > > follow
> >>> > > > > up in
> >>> > > > > > > a day or two with more detailed analysis.
> >>> > > > > > >
> >>> > > > > > > Major changes include:
> >>> > > > > > > - Early Access to KIP-848 - the next generation of the
> consumer
> >>> > > > > rebalance
> >>> > > > > > > protocol
> >>> > > > > > > - KIP-858: Adding JBOD support to KRaft
> >>> > > > > > > - KIP-714: Observability into Client metrics via a
> standardized
> >>> > > > > interface
> >>> > > > > > >
> >>> > > > > > > Check more information in the WIP blog post:
> >>> > > > > > > https://github.com/apache/kafka-site/pull/578
> >>> > > > > > >
> >>> > > > > > > Release notes for the 3.7.0 release:
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/RELEASE_NOTES.html
> >>> > > > > > >
> >>> > > > > > > *** Please download, test and vote by Thursday, January
> 18, 9am
> >>> > PT
> >>> > > > ***
> >>> > > > > > >
> >>> > > > > > > Usually these deadlines tend to be 2-3 days, but due to
> this
> >>> > being
> >>> > > > the
> >>> > > > > > > first RC and the tests not having ran yet, I am giving it
> a bit
> >>> > > more
> >>> > > > > time.
> >>> > > > > > >
> >>> > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> >>> release:
> >>> > > > > > > https://kafka.apache.org/KEYS
> >>> > > > > > >
> >>> > > > > > > * Release artifacts to be voted upon (source and binary):
> >>> > > > > > >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/
> >>> > > > > > >
> >>> > > > > > > * Docker release artifact to be voted upon:
> >>> > > > > > > apache/kafka:3.7.0-rc2
> >>> > > > > > >
> >>> > > > > > > * Maven artifacts to be voted upon:
> >>> > > > > > >
> >>> > > >
> >>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >>> > > > > > >
> >>> > > > > > > * Javadoc:
> >>> > > > > > >
> >>> > >
> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/javadoc/
> >>> > > > > > >
> >>> > > > > > > * Tag to be voted upon (off 3.7 branch) is the 3.7.0 tag:
> >>> > > > > > > https://github.com/apache/kafka/releases/tag/3.7.0-rc2
> >>> > > > > > >
> >>> > > > > > > * Documentation:
> >>> > > > > > > https://kafka.apache.org/37/documentation.html
> >>> > > > > > >
> >>> > > > > > > * Protocol:
> >>> > > > > > > https://kafka.apache.org/37/protocol.html
> >>> > > > > > >
> >>> > > > > > > * Successful Jenkins builds for the 3.7 branch:
> >>> > > > > > > Unit/integration tests:
> >>> > > > > > >
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/58/
> >>> > > > > > > There are failing tests here. I have to follow up with
> triaging
> >>> > > some
> >>> > > > of
> >>> > > > > > > the failures and figuring out if they're actual problems or
> >>> > simply
> >>> > > > > flakes.
> >>> > > > > > >
> >>> > > > > > > System tests:
> >>> > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.7/
> >>> > > > > > >
> >>> > > > > > > No successful system test runs yet. I am working on
> getting the
> >>> > job
> >>> > > > to
> >>> > > > > run.
> >>> > > > > > >
> >>> > > > > > > * Successful Docker Image Github Actions Pipeline for 3.7
> >>> branch:
> >>> > > > > > > Attached are the scan_report and report_jvm output files
> from
> >>> the
> >>> > > > > Docker
> >>> > > > > > > Build run:
> >>> > > > > > >
> >>> > > > >
> >>> > >
> >>>
> https://github.com/apache/kafka/actions/runs/7486094960/job/20375761673
> >>> > > > > > >
> >>> > > > > > > And the final docker image build job - Docker Build Test
> >>> > Pipeline:
> >>> > > > > > > https://github.com/apache/kafka/actions/runs/7486178277
> >>> > > > > > >
> >>> > > > > > > The image is apache/kafka:3.7.0-rc2 -
> >>> > > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> https://hub.docker.com/layers/apache/kafka/3.7.0-rc2/images/sha256-5b4707c08170d39549fbb6e2a3dbb83936a50f987c0c097f23cb26b4c210c226?context=explore
> >>> > > > > > >
> >>> > > > > > > /**
> >>> > > > > > >
> >>> > > > > > > Thanks,
> >>> > > > > > > Stanislav Kozlovski
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>>
> >>> --
> >>> Best,
> >>> Stanislav
> >>>
>


-- 
Paolo Patierno

*Senior Principal Software Engineer @ Red Hat**Microsoft MVP on **Azure*

Twitter : @ppatierno <http://twitter.com/ppatierno>
Linkedin : paolopatierno <http://it.linkedin.com/in/paolopatierno>
GitHub : ppatierno <https://github.com/ppatierno>

[jira] [Created] (KAFKA-16101) Kafka cluster unavailable during KRaft migration rollback procedure

[Paolo Patierno (Jira)]

Paolo Patierno created KAFKA-16101:
--

 Summary: Kafka cluster unavailable during KRaft migration rollback 
procedure
 Key: KAFKA-16101
 URL: https://issues.apache.org/jira/browse/KAFKA-16101
 Project: Kafka
  Issue Type: Bug
  Components: kraft
Affects Versions: 3.6.1
Reporter: Paolo Patierno


Hello,

I was trying the KRaft migration rollback procedure locally and I came across a 
potential bug or anyway a situation where the cluster is not usable/available 
for a certain amount of time.

In order to test the procedure, I start with a one broker and one zookeeper 
node cluster. Then I start the migration with a one KRaft controller node. The 
migration runs fine and it reaches the point of "dual write" state.

>From this point, I try to run the rollback procedure as described in the 
>documentation.

As first step, this involves ...
 * stopping the broker
 * removing the __cluster_metadata folder
 * removing ZooKeeper migration flag and controller(s) related configuration 
from the broker
 * restarting the broker

With the above steps done, the broker starts in ZooKeeper mode (no migration, 
no KRaft controllers knowledge) and it keeps logging the following messages in 
DEBUG:
{code:java}
[2024-01-08 11:51:20,608] DEBUG 
[zk-broker-0-to-controller-forwarding-channel-manager]: Controller isn't 
cached, looking for local metadata changes 
(kafka.server.BrokerToControllerRequestThread)
[2024-01-08 11:51:20,608] DEBUG 
[zk-broker-0-to-controller-forwarding-channel-manager]: No controller provided, 
retrying after backoff (kafka.server.BrokerToControllerRequestThread)
[2024-01-08 11:51:20,629] DEBUG 
[zk-broker-0-to-controller-alter-partition-channel-manager]: Controller isn't 
cached, looking for local metadata changes 
(kafka.server.BrokerToControllerRequestThread)
[2024-01-08 11:51:20,629] DEBUG 
[zk-broker-0-to-controller-alter-partition-channel-manager]: No controller 
provided, retrying after backoff (kafka.server.BrokerToControllerRequestThread) 
{code}
What's happening should be clear.

The /controller znode in ZooKeeper still reports the KRaft controller 
(broker.id = 1) as controller. The broker get it from the znode but doesn't 
know how to reach it.

The issue is that until the procedure is complete with the next steps (shutting 
down KRaft controller, deleting /controller znode), the cluster is unusable. 
Any admin or client operation against the broker doesn't work, just hangs, the 
broker doesn't reply.

Imagining this scenario to a more complex one with 10-20-50 brokers and 
partitions' replicas spread across them, when the brokers are rolled one by one 
(in ZK mode) reporting the above error, the topics will become not available 
one after the other, until all brokers are in such a state and nothing can 
work. This is because from a KRaft controller perspective (still running), the 
brokers are not available anymore and the partitions' replicas are out of sync.

Of course, as soon as you complete the rollback procedure, after deleting the 
/controller znode, the brokers are able to elect a new controller among them 
and everything recovers to work.

My first question ... isn't the cluster supposed to work during rollback and 
being always available during the rollback when the procedure is not completed 
yet? Or having the cluster not available is an assumption during the rollback, 
until it's complete?

This "unavailability" time window could be reduced by deleting the /controller 
znode before shutting down the KRaft controllers to allow the brokers electing 
a new controller among them, but in this case, could be a race condition where 
KRaft controllers still running could steal leadership again?

Or is there anything missing in the documentation maybe which is driving to 
this problem?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16005) ZooKeeper to KRaft migration rollback missing disabling controller and migration configuration on brokers

[Paolo Patierno (Jira)]

Paolo Patierno created KAFKA-16005:
--

 Summary: ZooKeeper to KRaft migration rollback missing disabling 
controller and migration configuration on brokers
 Key: KAFKA-16005
 URL: https://issues.apache.org/jira/browse/KAFKA-16005
 Project: Kafka
  Issue Type: Bug
  Components: documentation
Affects Versions: 3.6.1
Reporter: Paolo Patierno


I was following the latest documentation additions to try the rollback process 
of a ZK cluster migrating to KRaft, while it's still in dual-write mode: 
[https://github.com/apache/kafka/pull/14160/files#diff-e4e8d893dc2a4e999c96713dd5b5857203e0756860df0e70fb0cb041aa4d347bR3786]

The first point is just about stopping broker, deleting __cluster_metadata 
folder and restarting broker.

I think it's missing at least the following steps:
 * removing/disabling the ZooKeeper migration flag
 * removing all properties related to controllers configuration (i.e. 
controller.quorum.voters, controller.listener.names, ...)

Without those steps, when the broker restarts, we have got broker re-creating 
the __cluster_metadata folder (because it syncs with controllers while they are 
still running).

Also, when controllers stops, the broker starts to raise exceptions like this:
{code:java}
[2023-12-13 15:22:28,437] DEBUG [BrokerToControllerChannelManager id=0 
name=quorum] Connection with localhost/127.0.0.1 (channelId=1) disconnected 
(org.apache.kafka.common.network.Selector)java.net.ConnectException: Connection 
refusedat java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native 
Method)at 
java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
at 
org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
at 
org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:224)
at 
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:526)   
 at org.apache.kafka.common.network.Selector.poll(Selector.java:481)at 
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:571)at 
org.apache.kafka.server.util.InterBrokerSendThread.pollOnce(InterBrokerSendThread.java:109)
at 
kafka.server.BrokerToControllerRequestThread.doWork(BrokerToControllerChannelManager.scala:421)
at 
org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:130)[2023-12-13
 15:22:28,438] INFO [BrokerToControllerChannelManager id=0 name=quorum] Node 1 
disconnected. (org.apache.kafka.clients.NetworkClient)[2023-12-13 15:22:28,438] 
WARN [BrokerToControllerChannelManager id=0 name=quorum] Connection to node 1 
(localhost/127.0.0.1:9093) could not be established. Broker may not be 
available. (org.apache.kafka.clients.NetworkClient) {code}
(where I have controller locally on port 9093)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16003) The znode /config/topics is not updated during KRaft migration in "dual-write" mode

[Paolo Patierno (Jira)]

Paolo Patierno created KAFKA-16003:
--

 Summary: The znode /config/topics is not updated during KRaft 
migration in "dual-write" mode
 Key: KAFKA-16003
 URL: https://issues.apache.org/jira/browse/KAFKA-16003
 Project: Kafka
  Issue Type: Bug
  Components: controller
Affects Versions: 3.6.1
Reporter: Paolo Patierno


I tried the following scenario ...

I have a ZooKeeper-based cluster and create a my-topic-1 topic (without 
specifying any specific configuration for it). The correct znodes are created 
under /config/topics and /brokers/topics.

I start a migration to KRaft but not moving forward from "dual write" mode. 
While in this mode, I create a new my-topic-2 topic (still without any specific 
config). I see that a new znode is created under /brokers/topics but NOT under 
/config/topics. It seems that the KRaft controller is not updating this 
information in ZooKeeper during the dual-write. The controller log shows that 
the write to ZooKeeper was done, but not everything I would say:
{code:java}
2023-12-13 10:23:26,229 TRACE [KRaftMigrationDriver id=3] Create Topic 
my-topic-2, ID Macbp8BvQUKpzmq2vG_8dA. Transitioned migration state from 
ZkMigrationLeadershipState{kraftControllerId=3, kraftControllerEpoch=7, 
kraftMetadataOffset=445, kraftMetadataEpoch=7, lastUpdatedTimeMs=1702462785587, 
migrationZkVersion=236, controllerZkEpoch=3, controllerZkVersion=3} to 
ZkMigrationLeadershipState{kraftControllerId=3, kraftControllerEpoch=7, 
kraftMetadataOffset=445, kraftMetadataEpoch=7, lastUpdatedTimeMs=1702462785587, 
migrationZkVersion=237, controllerZkEpoch=3, controllerZkVersion=3} 
(org.apache.kafka.metadata.migration.KRaftMigrationDriver) 
[controller-3-migration-driver-event-handler]
2023-12-13 10:23:26,229 DEBUG [KRaftMigrationDriver id=3] Made the following ZK 
writes when handling KRaft delta: {CreateTopic=1} 
(org.apache.kafka.metadata.migration.KRaftMigrationDriver) 
[controller-3-migration-driver-event-handler] {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15754) The kafka-storage tool can generate UUID starting with "-"

[Paolo Patierno (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paolo Patierno resolved KAFKA-15754.

Resolution: Not A Problem

> The kafka-storage tool can generate UUID starting with "-"
> --
>
> Key: KAFKA-15754
> URL: https://issues.apache.org/jira/browse/KAFKA-15754
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.6.0
>Reporter: Paolo Patierno
>Assignee: Paolo Patierno
>Priority: Major
>
> Using the kafka-storage.sh tool, it seems that it can still generate a UUID 
> starting with a dash "-", which then breaks how the argparse4j library works. 
> With such an UUID (i.e. -rmdB0m4T4–Y4thlNXk4Q in my case) the tool exits with 
> the following error:
> kafka-storage: error: argument --cluster-id/-t: expected one argument
> Said that, it seems that this problem was already addressed in the 
> Uuid.randomUuid method which keeps generating a new UUID until it doesn't 
> start with "-". This is the commit addressing it 
> [https://github.com/apache/kafka/commit/5c1dd493d6f608b566fdad5ab3a896cb13622bce]
> The problem is that when the toString is called on the Uuid instance, it's 
> going to do a Base64 encoding on the generated UUID this way:
> {code:java}
> Base64.getUrlEncoder().withoutPadding().encodeToString(getBytesFromUuid()); 
> {code}
> Not sure why, but the code is using an URL (safe) encoder which, taking a 
> look at the Base64 class in Java, is using a RFC4648_URLSAFE encoder using 
> the following alphabet:
>  
> {code:java}
> private static final char[] toBase64URL = new char[]{'A', 'B', 'C', 'D', 'E', 
> 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 
> 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 
> 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 
> 'y', 'z', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '-', '_'}; {code}
> which as you can see includes the "-" character.
> So despite the current Uuid.randomUuid is avoiding the generation of a UUID 
> containing a dash, the Base64 encoding operation can return a final UUID 
> starting with the dash instead.
>  
> I was wondering if there is any good reason for using a Base64 URL encoder 
> and not just the RFC4648 (not URL safe) which uses the common Base64 alphabet 
> not containing the "-".



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-15754) The kafka-storage tool can generate UUID starting with "-"

[Paolo Patierno (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paolo Patierno reopened KAFKA-15754:


> The kafka-storage tool can generate UUID starting with "-"
> --
>
> Key: KAFKA-15754
> URL: https://issues.apache.org/jira/browse/KAFKA-15754
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.6.0
>Reporter: Paolo Patierno
>Assignee: Paolo Patierno
>Priority: Major
>
> Using the kafka-storage.sh tool, it seems that it can still generate a UUID 
> starting with a dash "-", which then breaks how the argparse4j library works. 
> With such an UUID (i.e. -rmdB0m4T4–Y4thlNXk4Q in my case) the tool exits with 
> the following error:
> kafka-storage: error: argument --cluster-id/-t: expected one argument
> Said that, it seems that this problem was already addressed in the 
> Uuid.randomUuid method which keeps generating a new UUID until it doesn't 
> start with "-". This is the commit addressing it 
> [https://github.com/apache/kafka/commit/5c1dd493d6f608b566fdad5ab3a896cb13622bce]
> The problem is that when the toString is called on the Uuid instance, it's 
> going to do a Base64 encoding on the generated UUID this way:
> {code:java}
> Base64.getUrlEncoder().withoutPadding().encodeToString(getBytesFromUuid()); 
> {code}
> Not sure why, but the code is using an URL (safe) encoder which, taking a 
> look at the Base64 class in Java, is using a RFC4648_URLSAFE encoder using 
> the following alphabet:
>  
> {code:java}
> private static final char[] toBase64URL = new char[]{'A', 'B', 'C', 'D', 'E', 
> 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 
> 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 
> 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 
> 'y', 'z', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '-', '_'}; {code}
> which as you can see includes the "-" character.
> So despite the current Uuid.randomUuid is avoiding the generation of a UUID 
> containing a dash, the Base64 encoding operation can return a final UUID 
> starting with the dash instead.
>  
> I was wondering if there is any good reason for using a Base64 URL encoder 
> and not just the RFC4648 (not URL safe) which uses the common Base64 alphabet 
> not containing the "-".



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15754) The kafka-storage tool can generate UUID starting with "-"

[Paolo Patierno (Jira)]

Paolo Patierno created KAFKA-15754:
--

 Summary: The kafka-storage tool can generate UUID starting with "-"
 Key: KAFKA-15754
 URL: https://issues.apache.org/jira/browse/KAFKA-15754
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.6.0
Reporter: Paolo Patierno


Using the kafka-storage.sh tool, it seems that it can still generate a UUID 
starting with a dash "-", which then breaks how the argparse4j library works. 
With such an UUID (i.e. -rmdB0m4T4–Y4thlNXk4Q in my case) the tool exits with 
the following error:
kafka-storage: error: argument --cluster-id/-t: expected one argument
Said that, it seems that this problem was already addressed in the 
Uuid.randomUuid method which keeps generating a new UUID until it doesn't start 
with "-". This is the commit addressing it 
[https://github.com/apache/kafka/commit/5c1dd493d6f608b566fdad5ab3a896cb13622bce]

The problem is that when the toString is called on the Uuid instance, it's 
going to do a Base64 encoding on the generated UUID this way:
{code:java}
Base64.getUrlEncoder().withoutPadding().encodeToString(getBytesFromUuid()); 
{code}
Not sure why, but the code is using an URL (safe) encoder which, taking a look 
at the Base64 class in Java, is using a RFC4648_URLSAFE encoder using the 
following alphabet:
 
{code:java}
private static final char[] toBase64URL = new char[]{'A', 'B', 'C', 'D', 'E', 
'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 
'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 
'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '0', 
'1', '2', '3', '4', '5', '6', '7', '8', '9', '-', '_'}; {code}
which as you can see includes the "-" character.
So despite the current Uuid.randomUuid is avoiding the generation of a UUID 
containing a "-", the Base64 encoded result can contain a "-" instead 
eventually.
 
I was wondering if there is any good reason for using a Base64 URL encoder and 
not just the RFC4648 (not URL safe) which uses the common Base64 alphabet not 
containing the "-".



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15689) KRaftMigrationDriver not logging the skipped event when expected state is wrong

[Paolo Patierno (Jira)]

Paolo Patierno created KAFKA-15689:
--

 Summary: KRaftMigrationDriver not logging the skipped event when 
expected state is wrong
 Key: KAFKA-15689
 URL: https://issues.apache.org/jira/browse/KAFKA-15689
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.6.0
Reporter: Paolo Patierno
Assignee: Paolo Patierno


The KRaftMigrationDriver.checkDriverState is used in multiple implementations 
of the 
MigrationEvent base class but when it comes to log that an event was skipped 
because the expected state is wrong, it always log "KRafrMigrationDriver" 
instead of the skipped event.
This is because its code has something like this:
 
{code:java}
log.info("Expected driver state {} but found {}. Not running this event {}.",
expectedState, migrationState, this.getClass().getSimpleName()); {code}
Of course, the "this" is referring to the KRafrMigrationDriver class.
It should print the specific skipped event instead.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14883) Broker state should be "observer" in KRaft quorum

[Paolo Patierno (Jira)]

Paolo Patierno created KAFKA-14883:
--

 Summary: Broker state should be "observer" in KRaft quorum
 Key: KAFKA-14883
 URL: https://issues.apache.org/jira/browse/KAFKA-14883
 Project: Kafka
  Issue Type: Improvement
  Components: kraft, metrics
Affects Versions: 3.4.0
Reporter: Paolo Patierno
Assignee: Paolo Patierno


Currently, the `current-state` KRaft related metric reports `follower` state 
for a broker while technically it should be reported as an `observer`  as the 
`kafka-metadata-quorum` tool does.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14411) Logging warning when partitions don't exist on assign request

[Paolo Patierno (Jira)]

Paolo Patierno created KAFKA-14411:
--

 Summary: Logging warning when partitions don't exist on assign 
request
 Key: KAFKA-14411
 URL: https://issues.apache.org/jira/browse/KAFKA-14411
 Project: Kafka
  Issue Type: Improvement
  Components: consumer
Reporter: Paolo Patierno


{{When using the assign method on a consumer providing a non existing topic 
(and the Kafka cluster has no auto-creation enabled), the log shows messages 
like.}}
{code:java}
Subscribed to partition(s): not-existing-topic-1
Error while fetching metadata with correlation id 3 : 
{not-existing-topic=UNKNOWN_TOPIC_OR_PARTITION}{code}
{{which could make sense if at some point the user create the topic and the 
consumer will be subscribed to it.}}

{{Different is when the topic exists but not the partition requested by the 
consumer.}}
{code:java}
Subscribed to partition(s): existing-topic-1 {code}
{{The above message shows that the consumer is subscribed but it will start to 
get messages only when the partition will be created as well. Anyway, the log 
could be misleading not printing at least a WARNING that the requested 
partition doesn't exist.}}

{{So, as we have an error on fetching metadata logged when topic not exist (no 
auto-creation enabled), it could be useful to have WARNING messages in the log 
about not existing requested partitions.}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New committer: Tom Bentley

[Paolo Patierno]

Congratulations Tom!

Get Outlook for Android


From: Guozhang Wang 
Sent: Monday, March 15, 2021 8:02:44 PM
To: dev 
Subject: Re: [ANNOUNCE] New committer: Tom Bentley

Congratulations Tom!

Guozhang

On Mon, Mar 15, 2021 at 11:25 AM Bill Bejeck 
wrote:

> Congratulations, Tom!
>
> -Bill
>
> On Mon, Mar 15, 2021 at 2:08 PM Bruno Cadonna 
> wrote:
>
> > Congrats, Tom!
> >
> > Best,
> > Bruno
> >
> > On 15.03.21 18:59, Mickael Maison wrote:
> > > Hi all,
> > >
> > > The PMC for Apache Kafka has invited Tom Bentley as a committer, and
> > > we are excited to announce that he accepted!
> > >
> > > Tom first contributed to Apache Kafka in June 2017 and has been
> > > actively contributing since February 2020.
> > > He has accumulated 52 commits and worked on a number of KIPs. Here are
> > > some of the most significant ones:
> > > KIP-183: Change PreferredReplicaLeaderElectionCommand to use
> > AdminClient
> > > KIP-195: AdminClient.createPartitions
> > > KIP-585: Filter and Conditional SMTs
> > > KIP-621: Deprecate and replace DescribeLogDirsResult.all() and
> > .values()
> > > KIP-707: The future of KafkaFuture (still in discussion)
> > >
> > > In addition, he is very active on the mailing list and has helped
> > > review many KIPs.
> > >
> > > Congratulations Tom and thanks for all the contributions!
> > >
> >
>


--
-- Guozhang

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

[Paolo Patierno]

Congratulations Dong!

Paolo Patierno
Principal Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


From: Dongjin Lee 
Sent: Monday, August 20, 2018 1:00 PM
To: dev@kafka.apache.org
Subject: Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

Congratulations!!

On Mon, Aug 20, 2018 at 9:22 PM Mickael Maison 
wrote:

> Congratulations Dong!
> On Mon, Aug 20, 2018 at 12:46 PM Manikumar 
> wrote:
> >
> > Congrats,  Dong Lin!
> >
> > On Mon, Aug 20, 2018 at 4:25 PM Ismael Juma  wrote:
> >
> > > Hi everyone,
> > >
> > > Dong Lin became a committer in March 2018. Since then, he has remained
> > > active in the community and contributed a number of patches, reviewed
> > > several pull requests and participated in numerous KIP discussions. I
> am
> > > happy to announce that Dong is now a member of the
> > > Apache Kafka PMC.
> > >
> > > Congratulation Dong! Looking forward to your future contributions.
> > >
> > > Ismael, on behalf of the Apache Kafka PMC
> > >
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
> *github:  <http://goog_969573159/>github.com/dongjinleekr
> <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
> <http://kr.linkedin.com/in/dongjinleekr>slideshare: 
> www.slideshare.net/dongjinleekr<http://www.slideshare.net/dongjinleekr>
> <http://www.slideshare.net/dongjinleekr>*
>

KIP-176 and related PR : today deadline for merging in 2.0.0

[Paolo Patierno]

Hi,

because today, as far as I understood, should be the deadline for having the 
PRs related to KIPs merged for the 2.0.0, maybe any committers can take a look 
(review and maybe merge) the PR :

https://github.com/apache/kafka/pull/5097

related to the KIP-176 :

https://cwiki.apache.org/confluence/display/KAFKA/KIP-176%3A+Remove+deprecated+new-consumer+option+for+tools

Thanks


Paolo Patierno
Principal Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

Re: [VOTE] KIP-176: Remove deprecated new-consumer option for tools

[Paolo Patierno]

Thanks for voting!

As far as I understood, today is the last day for having a PR for this one.
The KIP-176 has 3 (bindings) + 2 (non binding) votes so it is accepted.
I'll provide a related PR by the EOD.

Thanks,

Paolo Patierno
Principal Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


From: Ismael Juma 
Sent: Friday, May 25, 2018 9:42 PM
To: dev
Subject: Re: [VOTE] KIP-176: Remove deprecated new-consumer option for tools

+1 (binding).

Ismael

On Wed, 23 May 2018, 09:04 Paolo Patierno,  wrote:

> Sorry ... I hope it's not too late but I created the KIP-176 on September
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-176%3A+Remove+deprecated+new-consumer+option+for+tools
>
> but due to be a breaking change, I needed to wait for a major release ...
> and the right time is now.
> Can you vote for that adding to the release plan, please ?
>
> Thanks,
>
> Paolo Patierno
> Principal Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>

[VOTE] KIP-176: Remove deprecated new-consumer option for tools

[Paolo Patierno]

Sorry ... I hope it's not too late but I created the KIP-176 on September

https://cwiki.apache.org/confluence/display/KAFKA/KIP-176%3A+Remove+deprecated+new-consumer+option+for+tools

but due to be a breaking change, I needed to wait for a major release ... and 
the right time is now.
Can you vote for that adding to the release plan, please ?

Thanks,

Paolo Patierno
Principal Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

Re: [ANNOUNCE] New Committer: Dong Lin

[Paolo Patierno]

Congrats Dong !!

From: Becket Qin 
Sent: Wednesday, March 28, 2018 7:58:07 PM
To: dev; us...@kafka.apache.org
Subject: [ANNOUNCE] New Committer: Dong Lin

Hello everyone,

The PMC of Apache Kafka is pleased to announce that Dong Lin has accepted
our invitation to be a new Kafka committer.

Dong started working on Kafka about four years ago, since which he has
contributed numerous features and patches. His work on Kafka core has been
consistent and important. Among his contributions, most noticeably, Dong
developed JBOD (KIP-112, KIP-113) to handle disk failures and to reduce
overall cost, added deleteDataBefore() API (KIP-107) to allow users
actively remove old messages. Dong has also been active in the community,
participating in KIP discussions and doing code reviews.

Congratulations and looking forward to your future contribution, Dong!

Jiangjie (Becket) Qin, on behalf of Apache Kafka PMC

Re: 1.1 release progress

[Paolo Patierno]

Hi Guozhang and Damian,

I have a couple of issues marked for 1.1.0 but the PR isn't reviewed since a 
long time and both of them should be ready.

They are :

KAFKA-5919
KAFKA-5532

Maybe you can take a look at them ?

Thanks,
Paolo.

From: Guozhang Wang 
Sent: Saturday, January 27, 2018 4:15 AM
To: dev@kafka.apache.org
Subject: Re: 1.1 release progress

Thanks Damian,

I made a pass over the in-progress / open ones and moved some of them out
of 1.1.


Guozhang

On Fri, Jan 26, 2018 at 11:32 AM, Damian Guy  wrote:

> Hi,
>
> Just a quick updated on the 1.1 release. Feature freeze is on the 30th of
> January, so please ensure any major features have been merged and that any
> minor features have PRs by this time.
>
> We currently have 57 issues in progress and 117 todo:
> https://issues.apache.org/jira/projects/KAFKA/versions/12339769
> If people can start moving stuff out of 1.1 that would be great, otherwise
> anything that is todo will be moved to the next release.
>
> Thanks,
> Damian
>



--
-- Guozhang

Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API

[Paolo Patierno]

Hi Matthias,

just a question : what will be the timestamp "type" in the new message on the 
wire ?

Thanks,
Paolo.

From: Matthias J. Sax 
Sent: Wednesday, January 31, 2018 2:06 AM
To: dev@kafka.apache.org
Subject: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API

Hi,

I want to propose a new KIP for Kafka Streams that allows timestamp
manipulation at Processor API level.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-251%3A+Allow+timestamp+manipulation+in+Processor+API

Looking forward to your feedback.


-Matthias

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

[Paolo Patierno]

Congratulations Rajini !


From: Sriram Subramanian 
Sent: Wednesday, January 17, 2018 8:01 PM
To: dev@kafka.apache.org
Subject: Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

Congratulations Rajini!

On Wed, Jan 17, 2018 at 11:40 AM, Edoardo Comar  wrote:

> Congratulations Rajini! Bravissima!
> --
>
> Edoardo Comar
>
> IBM Message Hub
>
> IBM UK Ltd, Hursley Park, SO21 2JN
>
>
>
> From:   Guozhang Wang 
> To: dev@kafka.apache.org
> Date:   17/01/2018 19:29
> Subject:Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram
>
>
>
> Congratulations Rajini ！
>
> Guozhang
>
> On Wed, Jan 17, 2018 at 11:08 AM, Bill Bejeck  wrote:
>
> > Congrats Rajini!
> >
> > -Bill
> >
> > On Wed, Jan 17, 2018 at 2:02 PM, Matthias J. Sax 
> > wrote:
> >
> > > Congrats Rajini!!
> > >
> > > -Matthias
> > >
> > > On 1/17/18 10:54 AM, Vahid S Hashemian wrote:
> > > > Great news! Congratulations Rajini!
> > > >
> > > > --Vahid
> > > >
> > > >
> > > >
> > > > From:   Gwen Shapira 
> > > > To: "dev@kafka.apache.org" , Users
> > > > 
> > > > Date:   01/17/2018 10:49 AM
> > > > Subject:[ANNOUNCE] New Kafka PMC Member: Rajini Sivaram
> > > >
> > > >
> > > >
> > > > Dear Kafka Developers, Users and Fans,
> > > >
> > > > Rajini Sivaram became a committer in April 2017.  Since then, she
> > > remained
> > > > active in the community and contributed major patches, reviews and
> KIP
> > > > discussions. I am glad to announce that Rajini is now a member of
> the
> > > > Apache Kafka PMC.
> > > >
> > > > Congratulations, Rajini and looking forward to your future
> > contributions.
> > > >
> > > > Gwen, on behalf of Apache Kafka PMC
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
>
>
>
> --
> -- Guozhang
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>

Re: [ANNOUNCE] New committer: Matthias J. Sax

[Paolo Patierno]

Congratulations Matthias ! Very well deserved !

From: Guozhang Wang 
Sent: Friday, January 12, 2018 11:59:21 PM
To: dev@kafka.apache.org; us...@kafka.apache.org
Subject: [ANNOUNCE] New committer: Matthias J. Sax

Hello everyone,

The PMC of Apache Kafka is pleased to announce Matthias J. Sax as our
newest Kafka committer.

Matthias has made tremendous contributions to Kafka Streams API since early
2016. His footprint has been all over the places in Streams: in the past
two years he has been the main driver on improving the join semantics
inside Streams DSL, summarizing all their shortcomings and bridging the
gaps; he has also been largely working on the exactly-once semantics of
Streams by leveraging on the transaction messaging feature in 0.11.0. In
addition, Matthias have been very active in community activity that goes
beyond mailing list: he's getting the close to 1000 up votes and 100
helpful flags on SO for answering almost all questions about Kafka Streams.

Thank you for your contribution and welcome to Apache Kafka, Matthias!



Guozhang, on behalf of the Apache Kafka PMC

Re: Stream Processing Meetup@LinkedIn (Dec 4th)

[Paolo Patierno]

Hi Becket,
I watched some of these meetups on the related YouTube channel in the past.
Will be it available in streaming or just recorded for watching it later ?

Thanks
Paolo

From: Becket Qin 
Sent: Friday, November 17, 2017 8:33:04 PM
To: dev@kafka.apache.org; us...@kafka.apache.org
Subject: Stream Processing Meetup@LinkedIn (Dec 4th)

Hi Kafka users and developers,

We are going to host our quarterly Stream Processing Meetup@LinkedIn on Dec
4. There will be three speakers from Slack, Uber and LinkedIn. Please check
the details below if you are interested.

Thanks,

Jiangjie (Becket) Qin

*Stream Processing with Apache Kafka & Apache Samza*

   - Meetup Link: here
   
   - When: Dec 4th 2017 @ 6:00pm
   - Where:  LinkedIn Building F , 605 West Maude Avenue, Sunnyvale, CA (edit
   map
   
   )


*Abstract*

   1. Stream processing using Samza-SQL @ LinkedIn

*Speaker: Srinivasulu Punuru, LinkedIn*
Imagine if you can develop and run a stream processing job in few minutes
and Imagine if a vast majority of your organization (business analysts,
Product manager, Data scientists) can do this on their own without a need
for a development team.
Need for real time insights into the big data is increasing at a rapid
pace. The traditional Java based development model of developing, deploying
and managing the stream processing application is becoming a huge
constraint.
With Samza SQL we can simplify application development by enabling users to
create stream processing applications and get real time insights into their
business using SQL statements.

In this talk we try to answer the following questions

   1. How SQL language can be used to perform stream processing?
   2. How is Samza SQL implemented - Architecture?
   3. How can you deploy Samza SQL in your company?


2.   Streaming data pipeline @ Slack
*Speaker:- Ananth Packkildurai, Slack*
*Abstract:  *Slack is a communication and collaboration platform for teams.
Our millions of users spend 10+ hrs connected to the service on a typical
working day. They expect reliability, low latency, and extraordinarily rich
client experiences across a wide variety of devices and network conditions.
It is crucial for the developers to get the realtime insights on Slack
operational metrics.
In this talk, I will talk about how our data platform evolves from the
batch system to near realtime. I will also touch base on how Samza helps us
to build low latency data pipelines & Experimentation framework.

3.   Improving Kafka at-least-once performance
*Speaker: Ying Zheng, Uber*
*Abstract:*
Abstract:
At Uber, we are seeing an increased demand for Kafka at-least-once
delivery. So far, we are running a dedicated at-least-once Kafka cluster
with special settings. With a very low workload, the dedicated
at-least-once cluster has been working well for more than a year. Now, when
we want to turn on at-least-once producing on all the Kafka clusters, the
at-least-once producing performance is one of the concerns. I have worked a
couple of months to investigate the Kafka performance issues. With Kafka
code changes and Kafka / Java configuration changes, I have reduced
at-least-once producing latency by about 60% to 70%. Some of those
improvements can also improve the general Kafka throughput or reducing
end-to-end Kafka latency, when ack = 0 or ack = 1.

Re: [VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Ismael +1 ... I'm going to update the name


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma 
<ism...@juma.me.uk>
Sent: Monday, November 13, 2017 11:44 PM
To: dev@kafka.apache.org
Cc: Guozhang Wang
Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new 
Admin Client API

Looks good to me, one minor comment: I thought DeleteRecords should be
DeletedRecords. That makes it clearer that it's the result of deleting in
my opinion.

Ismael

On Mon, Nov 13, 2017 at 11:16 PM, Paolo Patierno <ppatie...@live.com> wrote:

> Guozhang thanks :-)
>
>
> It's getting late in my timezone, so maybe it's better for me don't take a
> look at email anymore ;)
>
>
> So finally, the KIP-204 was accepted. Waiting for more comments (if
> needed) on the PR for getting it merged.
>
>
> Thanks,
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> ____
> From: Guozhang Wang <wangg...@gmail.com>
> Sent: Monday, November 13, 2017 11:11 PM
> To: Paolo Patierno
> Cc: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new
> Admin Client API
>
> Paolo,
>
> I have indeed voted +1 on the KIP itself, but I thought you only have two
> binding +1s (Jason and myself); overlooked the vote from Damian and I was
> expecting Ismael to vote.
>
> So I think you are all good now :)
>
>
> Guozhang
>
>
> On Mon, Nov 13, 2017 at 3:08 PM, Paolo Patierno <ppatie...@live.com
> <mailto:ppatie...@live.com>> wrote:
> Sorry Guozhang ... reviewing the emails thread I have misunderstood your
> +1 which was not a vote but just about the wiki design.
>
> So the vote is still opened with 2 binding votes and 5 non binding votes.
> 
> From: Guozhang Wang <wangg...@gmail.com<mailto:wangg...@gmail.com>>
> Sent: Monday, November 13, 2017 9:21:07 PM
> To: Paolo Patierno
> Cc: dev@kafka.apache.org<mailto:dev@kafka.apache.org>
>
> Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new
> Admin Client API
>
> I'm not sure if Ismael's reply on the mailing list is a casted vote for
> this KIP.
>
> @Ismael, could you review the KIP again and cast your vote if possible?
>
>
> Guozhang
>
>
> On Mon, Nov 13, 2017 at 6:10 AM, Paolo Patierno <ppatie...@live.com
> <mailto:ppatie...@live.com>> wrote:
>
> Hi all,
>
> I'm going to close this vote because this KIP was accepted with :
>
>
> 3 binding votes
>
> 5 non-binding votes
>
>
> Thanks everyone for comments and for voting.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Paolo Patierno <ppatie...@live.com<mailto:ppatie...@live.com>>
> Sent: Saturday, November 11, 2017 9:38 AM
> To: Guozhang Wang; dev@kafka.apache.org<mailto:dev@kafka.apache.org>
> Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new
> Admin Client API
>
> So I have updated the KIP-204<https://cwiki.apache.
> org/confluence/display/KAFKA/KIP-204+%3A+Adding+records+
> deletion+operation+to+the+new+Admin+Client+API> using a DeleteRecords
> class as a wrapper of the low watermark for now.
>
> I have updated the related PR<https://github.com/apache/kafka/pull/4132>
> as well.
>
>
> Thanks for your comments !
>
>
> Paolo.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> Fro

Re: [VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Guozhang thanks :-)


It's getting late in my timezone, so maybe it's better for me don't take a look 
at email anymore ;)


So finally, the KIP-204 was accepted. Waiting for more comments (if needed) on 
the PR for getting it merged.


Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Guozhang Wang <wangg...@gmail.com>
Sent: Monday, November 13, 2017 11:11 PM
To: Paolo Patierno
Cc: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new 
Admin Client API

Paolo,

I have indeed voted +1 on the KIP itself, but I thought you only have two 
binding +1s (Jason and myself); overlooked the vote from Damian and I was 
expecting Ismael to vote.

So I think you are all good now :)


Guozhang


On Mon, Nov 13, 2017 at 3:08 PM, Paolo Patierno 
<ppatie...@live.com<mailto:ppatie...@live.com>> wrote:
Sorry Guozhang ... reviewing the emails thread I have misunderstood your +1 
which was not a vote but just about the wiki design.

So the vote is still opened with 2 binding votes and 5 non binding votes.

From: Guozhang Wang <wangg...@gmail.com<mailto:wangg...@gmail.com>>
Sent: Monday, November 13, 2017 9:21:07 PM
To: Paolo Patierno
Cc: dev@kafka.apache.org<mailto:dev@kafka.apache.org>

Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new 
Admin Client API

I'm not sure if Ismael's reply on the mailing list is a casted vote for this 
KIP.

@Ismael, could you review the KIP again and cast your vote if possible?


Guozhang


On Mon, Nov 13, 2017 at 6:10 AM, Paolo Patierno 
<ppatie...@live.com<mailto:ppatie...@live.com>> wrote:

Hi all,

I'm going to close this vote because this KIP was accepted with :


3 binding votes

5 non-binding votes


Thanks everyone for comments and for voting.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Paolo Patierno <ppatie...@live.com<mailto:ppatie...@live.com>>
Sent: Saturday, November 11, 2017 9:38 AM
To: Guozhang Wang; dev@kafka.apache.org<mailto:dev@kafka.apache.org>
Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new 
Admin Client API

So I have updated the 
KIP-204<https://cwiki.apache.org/confluence/display/KAFKA/KIP-204+%3A+Adding+records+deletion+operation+to+the+new+Admin+Client+API>
 using a DeleteRecords class as a wrapper of the low watermark for now.

I have updated the related PR<https://github.com/apache/kafka/pull/4132> as 
well.


Thanks for your comments !


Paolo.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Colin McCabe <co...@cmccabe.xyz<mailto:co...@cmccabe.xyz>>
Sent: Friday, November 10, 2017 8:08 PM
To: Guozhang Wang; dev@kafka.apache.org<mailto:dev@kafka.apache.org>
Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new 
Admin Client API

+1 for returning a named object rather than Long.

C.


On Fri, Nov 10, 2017, at 10:07, Guozhang Wang wrote:
> Sounds good to me for returning an object than a Long type.
>
>
> Guozhang
>
> On Fri, Nov 10, 2017 at 9:26 AM, Paolo Patierno 
> <ppatie...@live.com<mailto:ppatie...@live.com>>
> wrote:
>
> > Ismael,
> >
> >
> > yes it makes sense. Taking a look to the other methods in the Admin> > 
> > Client, there is no use case returning a "simple" type : most of
> > them are> > Void or complex result represented by classes.
> >
> > In order to support future extensibility I like your idea.
> >
> > Let's see what's the others opinions otherwise I'll start to
> > implement in> > such way.
> >
> >
> > I have updated the KIP and the PR using "recordsToDelete"
> > parameter as> > well.
> >
> >
> > Thanks
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Azure & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<

Re: [VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Sorry Guozhang ... reviewing the emails thread I have misunderstood your +1 
which was not a vote but just about the wiki design.

So the vote is still opened with 2 binding votes and 5 non binding votes.

From: Guozhang Wang <wangg...@gmail.com>
Sent: Monday, November 13, 2017 9:21:07 PM
To: Paolo Patierno
Cc: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new 
Admin Client API

I'm not sure if Ismael's reply on the mailing list is a casted vote for this 
KIP.

@Ismael, could you review the KIP again and cast your vote if possible?


Guozhang


On Mon, Nov 13, 2017 at 6:10 AM, Paolo Patierno 
<ppatie...@live.com<mailto:ppatie...@live.com>> wrote:

Hi all,

I'm going to close this vote because this KIP was accepted with :


3 binding votes

5 non-binding votes


Thanks everyone for comments and for voting.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


____
From: Paolo Patierno <ppatie...@live.com<mailto:ppatie...@live.com>>
Sent: Saturday, November 11, 2017 9:38 AM
To: Guozhang Wang; dev@kafka.apache.org<mailto:dev@kafka.apache.org>
Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new 
Admin Client API

So I have updated the 
KIP-204<https://cwiki.apache.org/confluence/display/KAFKA/KIP-204+%3A+Adding+records+deletion+operation+to+the+new+Admin+Client+API>
 using a DeleteRecords class as a wrapper of the low watermark for now.

I have updated the related PR<https://github.com/apache/kafka/pull/4132> as 
well.


Thanks for your comments !


Paolo.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Colin McCabe <co...@cmccabe.xyz<mailto:co...@cmccabe.xyz>>
Sent: Friday, November 10, 2017 8:08 PM
To: Guozhang Wang; dev@kafka.apache.org<mailto:dev@kafka.apache.org>
Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new 
Admin Client API

+1 for returning a named object rather than Long.

C.


On Fri, Nov 10, 2017, at 10:07, Guozhang Wang wrote:
> Sounds good to me for returning an object than a Long type.
>
>
> Guozhang
>
> On Fri, Nov 10, 2017 at 9:26 AM, Paolo Patierno 
> <ppatie...@live.com<mailto:ppatie...@live.com>>
> wrote:
>
> > Ismael,
> >
> >
> > yes it makes sense. Taking a look to the other methods in the Admin> > 
> > Client, there is no use case returning a "simple" type : most of
> > them are> > Void or complex result represented by classes.
> >
> > In order to support future extensibility I like your idea.
> >
> > Let's see what's the others opinions otherwise I'll start to
> > implement in> > such way.
> >
> >
> > I have updated the KIP and the PR using "recordsToDelete"
> > parameter as> > well.
> >
> >
> > Thanks
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Azure & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
> >
> > 
> > From: isma...@gmail.com<mailto:isma...@gmail.com> 
> > <isma...@gmail.com<mailto:isma...@gmail.com>> on behalf of Ismael
> > Juma <> > ism...@juma.me.uk<mailto:ism...@juma.me.uk>>
> > Sent: Friday, November 10, 2017 1:15 PM
> > To: dev@kafka.apache.org<mailto:dev@kafka.apache.org>
> > Subject: Re: [VOTE] KIP-204 : adding records deletion operation to
> > the new> > Admin Client API
> >
> > Paolo,
> >
> > You might return the timestamp if the user did a delete by
> > timestamp for> > example. But let's maybe hear what others think before we 
> > change
> > the KIP.> >
> > Ismael
> >
> > On Fri, Nov 10, 2017 at 12:48 PM, Paolo Patierno
> > <ppatie...@live.com<mailto:ppatie...@live.com>>> > wrote:
> >
> > > Hi Ismael,
> > >
> > >
> > >   1.  yes

Re: [VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Hi all,

I'm going to close this vote because this KIP was accepted with :


3 binding votes

5 non-binding votes


Thanks everyone for comments and for voting.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


____
From: Paolo Patierno <ppatie...@live.com>
Sent: Saturday, November 11, 2017 9:38 AM
To: Guozhang Wang; dev@kafka.apache.org
Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new 
Admin Client API

So I have updated the 
KIP-204<https://cwiki.apache.org/confluence/display/KAFKA/KIP-204+%3A+Adding+records+deletion+operation+to+the+new+Admin+Client+API>
 using a DeleteRecords class as a wrapper of the low watermark for now.

I have updated the related PR<https://github.com/apache/kafka/pull/4132> as 
well.


Thanks for your comments !


Paolo.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Colin McCabe <co...@cmccabe.xyz>
Sent: Friday, November 10, 2017 8:08 PM
To: Guozhang Wang; dev@kafka.apache.org
Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new 
Admin Client API

+1 for returning a named object rather than Long.

C.


On Fri, Nov 10, 2017, at 10:07, Guozhang Wang wrote:
> Sounds good to me for returning an object than a Long type.
>
>
> Guozhang
>
> On Fri, Nov 10, 2017 at 9:26 AM, Paolo Patierno <ppatie...@live.com>
> wrote:
>
> > Ismael,
> >
> >
> > yes it makes sense. Taking a look to the other methods in the Admin> > 
> > Client, there is no use case returning a "simple" type : most of
> > them are> > Void or complex result represented by classes.
> >
> > In order to support future extensibility I like your idea.
> >
> > Let's see what's the others opinions otherwise I'll start to
> > implement in> > such way.
> >
> >
> > I have updated the KIP and the PR using "recordsToDelete"
> > parameter as> > well.
> >
> >
> > Thanks
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Azure & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
> >
> > 
> > From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael
> > Juma <> > ism...@juma.me.uk>
> > Sent: Friday, November 10, 2017 1:15 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-204 : adding records deletion operation to
> > the new> > Admin Client API
> >
> > Paolo,
> >
> > You might return the timestamp if the user did a delete by
> > timestamp for> > example. But let's maybe hear what others think before we 
> > change
> > the KIP.> >
> > Ismael
> >
> > On Fri, Nov 10, 2017 at 12:48 PM, Paolo Patierno
> > <ppatie...@live.com>> > wrote:
> >
> > > Hi Ismael,
> > >
> > >
> > >   1.  yes it sounds better. Agree.
> > >   2.  We are using a wrapper class like RecordsToDelete in order
> > >   to allow> > > future operations that won't be based only on 
> > > specifying an
> > > offset. At> > same
> > > time with this wrapper class and static methods (i.e.
> > > beforeOffset) the> > API
> > > is really clear to understand. About moving to use a class for
> > > wrapping> > the
> > > lowWatermark and not just a Long, do you think that in the
> > > future we> > could
> > > have some other information to bring as part of the delete
> > > operation ? or> > > just for being clearer in terms of API (so not just 
> > > with a
> > > comment) ?> > >
> > > Thanks,
> > > Paolo.
> > >
> > >
> > > Paolo Patierno
> > > Senior Software Engineer (IoT) @ Red Hat
> > > Microsoft MVP on Azure & IoT
> > > Microsoft Azure Advis

Re: [VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

So I have updated the 
KIP-204<https://cwiki.apache.org/confluence/display/KAFKA/KIP-204+%3A+Adding+records+deletion+operation+to+the+new+Admin+Client+API>
 using a DeleteRecords class as a wrapper of the low watermark for now.

I have updated the related PR<https://github.com/apache/kafka/pull/4132> as 
well.


Thanks for your comments !


Paolo.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Colin McCabe <co...@cmccabe.xyz>
Sent: Friday, November 10, 2017 8:08 PM
To: Guozhang Wang; dev@kafka.apache.org
Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new 
Admin Client API

+1 for returning a named object rather than Long.

C.


On Fri, Nov 10, 2017, at 10:07, Guozhang Wang wrote:
> Sounds good to me for returning an object than a Long type.
>
>
> Guozhang
>
> On Fri, Nov 10, 2017 at 9:26 AM, Paolo Patierno <ppatie...@live.com>
> wrote:
>
> > Ismael,
> >
> >
> > yes it makes sense. Taking a look to the other methods in the Admin> > 
> > Client, there is no use case returning a "simple" type : most of
> > them are> > Void or complex result represented by classes.
> >
> > In order to support future extensibility I like your idea.
> >
> > Let's see what's the others opinions otherwise I'll start to
> > implement in> > such way.
> >
> >
> > I have updated the KIP and the PR using "recordsToDelete"
> > parameter as> > well.
> >
> >
> > Thanks
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Azure & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
> >
> > 
> > From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael
> > Juma <> > ism...@juma.me.uk>
> > Sent: Friday, November 10, 2017 1:15 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-204 : adding records deletion operation to
> > the new> > Admin Client API
> >
> > Paolo,
> >
> > You might return the timestamp if the user did a delete by
> > timestamp for> > example. But let's maybe hear what others think before we 
> > change
> > the KIP.> >
> > Ismael
> >
> > On Fri, Nov 10, 2017 at 12:48 PM, Paolo Patierno
> > <ppatie...@live.com>> > wrote:
> >
> > > Hi Ismael,
> > >
> > >
> > >   1.  yes it sounds better. Agree.
> > >   2.  We are using a wrapper class like RecordsToDelete in order
> > >   to allow> > > future operations that won't be based only on 
> > > specifying an
> > > offset. At> > same
> > > time with this wrapper class and static methods (i.e.
> > > beforeOffset) the> > API
> > > is really clear to understand. About moving to use a class for
> > > wrapping> > the
> > > lowWatermark and not just a Long, do you think that in the
> > > future we> > could
> > > have some other information to bring as part of the delete
> > > operation ? or> > > just for being clearer in terms of API (so not just 
> > > with a
> > > comment) ?> > >
> > > Thanks,
> > > Paolo.
> > >
> > >
> > > Paolo Patierno
> > > Senior Software Engineer (IoT) @ Red Hat
> > > Microsoft MVP on Azure & IoT
> > > Microsoft Azure Advisor
> > >
> > > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>> > > 
> > > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> > >
> > >
> > > ____
> > > From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael
> > > Juma <> > > ism...@juma.me.uk>
> > > Sent: Friday, November 10, 2017 12:33 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-204 : adding records deletion operation
> > > to the> > new
> > > Admin Client API
> >

Re: [VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Ismael,


yes it makes sense. Taking a look to the other methods in the Admin Client, 
there is no use case returning a "simple" type : most of them are Void or 
complex result represented by classes.

In order to support future extensibility I like your idea.

Let's see what's the others opinions otherwise I'll start to implement in such 
way.


I have updated the KIP and the PR using "recordsToDelete" parameter as well.


Thanks


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma 
<ism...@juma.me.uk>
Sent: Friday, November 10, 2017 1:15 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new 
Admin Client API

Paolo,

You might return the timestamp if the user did a delete by timestamp for
example. But let's maybe hear what others think before we change the KIP.

Ismael

On Fri, Nov 10, 2017 at 12:48 PM, Paolo Patierno <ppatie...@live.com> wrote:

> Hi Ismael,
>
>
>   1.  yes it sounds better. Agree.
>   2.  We are using a wrapper class like RecordsToDelete in order to allow
> future operations that won't be based only on specifying an offset. At same
> time with this wrapper class and static methods (i.e. beforeOffset) the API
> is really clear to understand. About moving to use a class for wrapping the
> lowWatermark and not just a Long, do you think that in the future we could
> have some other information to bring as part of the delete operation ? or
> just for being clearer in terms of API (so not just with a comment) ?
>
> Thanks,
> Paolo.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma <
> ism...@juma.me.uk>
> Sent: Friday, November 10, 2017 12:33 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new
> Admin Client API
>
> Hi Paolo,
>
> The KIP looks good, I have a couple of comments:
>
> 1. partitionsAndOffsets could perhaps be `recordsToDelete`.
> 2. It seems a bit inconsistent that the argument is `RecordsToDelete`, but
> the result is just a `Long`. Should the result be `DeleteRecords` or
> something like that? It could then have a field logStartOffset or
> lowWatermark instead of having to document it via a comment only.
>
> Ismael
>
> On Tue, Oct 3, 2017 at 6:51 AM, Paolo Patierno <ppatie...@live.com> wrote:
>
> > Hi all,
> >
> > I didn't see any further discussion around this KIP, so I'd like to start
> > the vote for it.
> >
> > Just for reference : https://cwiki.apache.org/confl
> > uence/display/KAFKA/KIP-204+%3A+adding+records+deletion+
> > operation+to+the+new+Admin+Client+API
> >
> >
> > Thanks,
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Azure & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
>

Re: [VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Hi Ismael,


  1.  yes it sounds better. Agree.
  2.  We are using a wrapper class like RecordsToDelete in order to allow 
future operations that won't be based only on specifying an offset. At same 
time with this wrapper class and static methods (i.e. beforeOffset) the API is 
really clear to understand. About moving to use a class for wrapping the 
lowWatermark and not just a Long, do you think that in the future we could have 
some other information to bring as part of the delete operation ? or just for 
being clearer in terms of API (so not just with a comment) ?

Thanks,
Paolo.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma 
<ism...@juma.me.uk>
Sent: Friday, November 10, 2017 12:33 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new 
Admin Client API

Hi Paolo,

The KIP looks good, I have a couple of comments:

1. partitionsAndOffsets could perhaps be `recordsToDelete`.
2. It seems a bit inconsistent that the argument is `RecordsToDelete`, but
the result is just a `Long`. Should the result be `DeleteRecords` or
something like that? It could then have a field logStartOffset or
lowWatermark instead of having to document it via a comment only.

Ismael

On Tue, Oct 3, 2017 at 6:51 AM, Paolo Patierno <ppatie...@live.com> wrote:

> Hi all,
>
> I didn't see any further discussion around this KIP, so I'd like to start
> the vote for it.
>
> Just for reference : https://cwiki.apache.org/confl
> uence/display/KAFKA/KIP-204+%3A+adding+records+deletion+
> operation+to+the+new+Admin+Client+API
>
>
> Thanks,
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>

Re: [VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Because Guozhang and Colin are doing a great job on reviewing the related PR 
and we are really close to have it in a decent/final shape, what do the other 
committers think about this KIP ?

We are stuck at 2 binding votes (and something like 5 non binding). We need 
last binding vote for having the KIP got accepted and we can start to think 
about merging the PR for a future 1.1.0 release.


Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Kamal <kamal.chandraprak...@gmail.com>
Sent: Monday, November 6, 2017 1:26 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new 
Admin Client API

+1 (non-binding)

On Thu, Nov 2, 2017 at 2:13 PM, Paolo Patierno <ppatie...@live.com> wrote:

> Thanks Jason !
>
>
> I have just updated the KIP with DeleteRecordsOptions definition.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Jason Gustafson <ja...@confluent.io>
> Sent: Wednesday, November 1, 2017 10:09 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new
> Admin Client API
>
> +1 (binding). Just one nit: the KIP doesn't define the DeleteRecordsOptions
> object. I see it's empty in the PR, but we may as well document the full
> API in the KIP.
>
> -Jason
>
>
> On Wed, Nov 1, 2017 at 2:54 PM, Guozhang Wang <wangg...@gmail.com> wrote:
>
> > Made a pass over the PR and left some comments. I'm +1 on the wiki design
> > page as well.
> >
> > On Tue, Oct 31, 2017 at 7:13 AM, Bill Bejeck <bbej...@gmail.com> wrote:
> >
> > > +1
> > >
> > > Thanks,
> > > Bill
> > >
> > > On Tue, Oct 31, 2017 at 4:36 AM, Paolo Patierno <ppatie...@live.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > >
> > > > because I don't see any further discussion around KIP-204 (
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 204+%3A+adding+records+deletion+operation+to+the+new+
> Admin+Client+API)
> > > > and I have already opened a PR with the implementation, can we
> re-cover
> > > the
> > > > vote started on October 18 ?
> > > >
> > > > There are only "non binding" votes up to now.
> > > >
> > > > Thanks,
> > > >
> > > >
> > > > Paolo Patierno
> > > > Senior Software Engineer (IoT) @ Red Hat
> > > > Microsoft MVP on Azure & IoT
> > > > Microsoft Azure Advisor
> > > >
> > > > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > > > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > > > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> > > >
> > > >
> > > > 
> > > > From: Viktor Somogyi <viktorsomo...@gmail.com>
> > > > Sent: Wednesday, October 18, 2017 10:49 AM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [VOTE] KIP-204 : adding records deletion operation to
> the
> > > new
> > > > Admin Client API
> > > >
> > > > +1 (non-binding)
> > > >
> > > > On Wed, Oct 18, 2017 at 8:23 AM, Manikumar <
> manikumar.re...@gmail.com>
> > > > wrote:
> > > >
> > > > > + (non-binding)
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Manikumar
> > > > >
> > > > > On Tue, Oct 17, 2017 at 7:42 AM, Dong Lin <lindon...@gmail.com>
> > wrote:
> > > > >
> > > > > > Thanks for the KIP. +1 (non-binding)
> > > > > >
> > > > > > On Wed, Oct 11, 2017 at 2:27 AM, Ted Yu <yuzhih...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > +1
> &g

Re: [ANNOUNCE] New committer: Onur Karaman

[Paolo Patierno]

Congrats Onur ! Well deserved !


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Viktor Somogyi <viktorsomo...@gmail.com>
Sent: Tuesday, November 7, 2017 11:34 AM
To: dev@kafka.apache.org
Subject: Re: [ANNOUNCE] New committer: Onur Karaman

Congrats Onur!

On Tue, Nov 7, 2017 at 9:17 AM, Tom Bentley <t.j.bent...@gmail.com> wrote:

> Well done Onur.
>
> On 7 November 2017 at 06:52, Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
> > Congratulations Onur!!
> > On Tue, 7 Nov 2017 at 06:30, Jaikiran Pai <jai.forums2...@gmail.com>
> > wrote:
> >
> > > Congratulations Onur!
> > >
> > > -Jaikiran
> > >
> > >
> > > On 06/11/17 10:54 PM, Jun Rao wrote:
> > > > Hi, everyone,
> > > >
> > > > The PMC of Apache Kafka is pleased to announce a new Kafka committer
> > Onur
> > > > Karaman.
> > > >
> > > > Onur's most significant work is the improvement of Kafka controller,
> > > which
> > > > is the brain of a Kafka cluster. Over time, we have accumulated
> quite a
> > > few
> > > > correctness and performance issues in the controller. There have been
> > > > attempts to fix controller issues in isolation, which would make the
> > code
> > > > base more complicated without a clear path of solving all problems.
> > Onur
> > > is
> > > > the one who took a holistic approach, by first documenting all known
> > > > issues, writing down a new design, coming up with a plan to deliver
> the
> > > > changes in phases and executing on it. At this point, Onur has
> > completed
> > > > the two most important phases: making the controller single threaded
> > and
> > > > changing the controller to use the async ZK api. The former fixed
> > > multiple
> > > > deadlocks and race conditions. The latter significantly improved the
> > > > performance when there are many partitions. Experimental results show
> > > that
> > > > Onur's work reduced the controlled shutdown time by a factor of 100
> > times
> > > > and the controller failover time by a factor of 3 times.
> > > >
> > > > Congratulations, Onur!
> > > >
> > > > Thanks,
> > > >
> > > > Jun (on behalf of the Apache Kafka PMC)
> > > >
> > >
> > >
> >
>

Re: [ANNOUNCE] Apache Kafka 1.0.0 Released

[Paolo Patierno]

Congratulations for this milestone !


Thanks to Gouzhang for running the release !


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Jaikiran Pai <jai.forums2...@gmail.com>
Sent: Thursday, November 2, 2017 2:59 AM
To: dev@kafka.apache.org
Cc: Users
Subject: Re: [ANNOUNCE] Apache Kafka 1.0.0 Released

Congratulations Kafka team on the release. Happy to see Kafka reach this
milestone. It has been a pleasure using Kafka and also interacting with
the Kafka team.

-Jaikiran


On 01/11/17 7:57 PM, Guozhang Wang wrote:
> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 1.0.0.
>
> This is a major release of the Kafka project, and is no mere bump of the
> version number. The Apache Kafka Project Management Committee has packed a
> number of valuable enhancements into the release. Let me summarize a few of
> them:
>
> ** Since its introduction in version 0.10, the Streams API has become
> hugely popular among Kafka users, including the likes of Pinterest,
> Rabobank, Zalando, and The New York Times. In 1.0, the the API continues to
> evolve at a healthy pace. To begin with, the builder API has been improved
> (KIP-120). A new API has been added to expose the state of active tasks at
> runtime (KIP-130). Debuggability gets easier with enhancements to the
> print() and writeAsText() methods (KIP-160). And if that’s not enough,
> check out KIP-138 and KIP-161 too. For more on streams, check out the
> Apache Kafka Streams documentation (https://kafka.apache.org/docu
> mentation/streams/), including some helpful new tutorial videos.
>
> ** Operating Kafka at scale requires that the system remain observable, and
> to make that easier, we’ve made a number of improvements to metrics. These
> are too many to summarize without becoming tedious, but Connect metrics
> have been significantly improved (KIP-196), a litany of new health check
> metrics are now exposed (KIP-188), and we now have a global topic and
> partition count (KIP-168). Check out KIP-164 and KIP-187 for even more.
>
> ** We now support Java 9, leading, among other things, to significantly
> faster TLS and CRC32C implementations. Over-the-wire encryption will be
> faster now, which will keep Kafka fast and compute costs low when
> encryption is enabled.
>
> ** In keeping with the security theme, KIP-152 cleans up the error handling
> on Simple Authentication Security Layer (SASL) authentication attempts.
> Previously, some authentication error conditions were indistinguishable
> from broker failures and were not logged in a clear way. This is cleaner
> now.
>
> ** Kafka can now tolerate disk failures better. Historically, JBOD storage
> configurations have not been recommended, but the architecture has
> nevertheless been tempting: after all, why not rely on Kafka’s own
> replication mechanism to protect against storage failure rather than using
> RAID? With KIP-112, Kafka now handles disk failure more gracefully. A
> single disk failure in a JBOD broker will not bring the entire broker down;
> rather, the broker will continue serving any log files that remain on
> functioning disks.
>
> ** Since release 0.11.0, the idempotent producer (which is the producer
> used in the presence of a transaction, which of course is the producer we
> use for exactly-once processing) required 
> max.in.flight.requests.per.connection
> to be equal to one. As anyone who has written or tested a wire protocol can
> attest, this put an upper bound on throughput. Thanks to KAFKA-5949, this
> can now be as large as five, relaxing the throughput constraint quite a bit.
>
>
> All of the changes in this release can be found in the release notes:
>
> https://dist.apache.org/repos/dist/release/kafka/1.0.0/RELEASE_NOTES.html
>
>
> You can download the source release from:
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/1.0.0/kafka-1.0.0-src.tgz
>
> and binary releases from:
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/1.0.0/kafka_2.11-1.0.0.tgz
> (Scala
> 2.11)
> https://www.apache.org/dyn/closer.cgi?path=/kafka/1.0.0/kafka_2.12-1.0.0.tgz
> (Scala
> 2.12)
>
>
> 
> ---
>
> Apache Kafka is a distributed streaming platform with four four core APIs:
>
> ** The Producer API allows an application to publish a stream records to one
> or more Kafka topics.
>
> ** The Consumer API allows an application to

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Hi Colin,


I have just updated the KIP mentioning that this new method should replace the 
"legacy" Scala API used for deleting records today.


Thanks,

Paolo.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


________
From: Paolo Patierno <ppatie...@live.com>
Sent: Tuesday, October 31, 2017 8:56 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

Hi Colin,
thanks !

This morning (Italy time zone) I started the vote for this KIP. Up to now there 
are 5 non-binding votes.

In any case, I'll update the section you mentioned. I totally agree with you on 
giving more info to developers who are using the Scala API.

Thanks
Paolo

From: Colin McCabe <cmcc...@apache.org>
Sent: Tuesday, October 31, 2017 9:49:59 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

Hi Paolo,

This looks like a good proposal.  I think it's probably ready to take it
to a vote soon?

Also, in the "Compatibility, Deprecation, and Migration Plan" section,
you might want to mention the internal Scala interface for doing this
which was added in KIP-107.  We should expect users to migrate from that
interface to the new one over time.

best,
Colin


On Wed, Oct 25, 2017, at 03:47, Paolo Patierno wrote:
> Thanks for all your feedback guys. I have updated my current code as
> well.
>
> I know that the vote for this KIP is not started yet (actually I opened
> it due to no feedback on this KIP after a while but then the discussion
> started and it was really useful !) but I have already opened a PR for
> that.
>
> Maybe feedback could be useful on that as well :
>
>
> https://github.com/apache/kafka/pull/4132
>
>
> Thanks
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Colin McCabe <cmcc...@apache.org>
> Sent: Monday, October 23, 2017 4:34 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> new Admin Client API
>
> On Mon, Oct 23, 2017, at 01:37, Tom Bentley wrote:
> > At the risk of muddying the waters further, have you considered
> > "RecordsToDelete" as the name of the class? It's both shorter and more
> > descriptive imho.
>
> +1 for RecordsToDelete
>
> >
> > Also "deleteBefore()" as the factory method name isn't very future proof
> > if
> > we came to support time-based deletion. Something like "beforeOffset()"
> > would be clearer, imho.
>
> Great idea.
>
> best,
> Colin
>
> >
> > Putting these together: RecordsToDelete.beforeOffset() seems much clearer
> > to me than DeleteRecordsTarget.deleteBefore()
> >
> >
> > On 23 October 2017 at 08:45, Paolo Patierno <ppatie...@live.com> wrote:
> >
> > > About the name I just started to have a doubt about DeletetionTarget
> > > because it could be bounded to any deletion operation (i.e. delete topic,
> > > ...) and not just what we want now, so records deletion.
> > >
> > > I have updated the KIP-204 using DeleteRecordsTarget so it's clear that
> > > it's related to the delete records operation and what it means, so the
> > > target for such operation.
> > >
> > >
> > > Paolo Patierno
> > > Senior Software Engineer (IoT) @ Red Hat
> > > Microsoft MVP on Azure & IoT
> > > Microsoft Azure Advisor
> > >
> > > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> > >
> > >
> > > 
> > > From: Paolo Patierno <ppatie...@live.com>
> > > Sent: Monday, October 23, 2017 7:38 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> > > new Admin Client API
> > >
> > > Hi Colin,
> > >
> >

Re: [VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Thanks Jason !


I have just updated the KIP with DeleteRecordsOptions definition.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Jason Gustafson <ja...@confluent.io>
Sent: Wednesday, November 1, 2017 10:09 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new 
Admin Client API

+1 (binding). Just one nit: the KIP doesn't define the DeleteRecordsOptions
object. I see it's empty in the PR, but we may as well document the full
API in the KIP.

-Jason


On Wed, Nov 1, 2017 at 2:54 PM, Guozhang Wang <wangg...@gmail.com> wrote:

> Made a pass over the PR and left some comments. I'm +1 on the wiki design
> page as well.
>
> On Tue, Oct 31, 2017 at 7:13 AM, Bill Bejeck <bbej...@gmail.com> wrote:
>
> > +1
> >
> > Thanks,
> > Bill
> >
> > On Tue, Oct 31, 2017 at 4:36 AM, Paolo Patierno <ppatie...@live.com>
> > wrote:
> >
> > > Hi all,
> > >
> > >
> > > because I don't see any further discussion around KIP-204 (
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 204+%3A+adding+records+deletion+operation+to+the+new+Admin+Client+API)
> > > and I have already opened a PR with the implementation, can we re-cover
> > the
> > > vote started on October 18 ?
> > >
> > > There are only "non binding" votes up to now.
> > >
> > > Thanks,
> > >
> > >
> > > Paolo Patierno
> > > Senior Software Engineer (IoT) @ Red Hat
> > > Microsoft MVP on Azure & IoT
> > > Microsoft Azure Advisor
> > >
> > > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> > >
> > >
> > > 
> > > From: Viktor Somogyi <viktorsomo...@gmail.com>
> > > Sent: Wednesday, October 18, 2017 10:49 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the
> > new
> > > Admin Client API
> > >
> > > +1 (non-binding)
> > >
> > > On Wed, Oct 18, 2017 at 8:23 AM, Manikumar <manikumar.re...@gmail.com>
> > > wrote:
> > >
> > > > + (non-binding)
> > > >
> > > >
> > > > Thanks,
> > > > Manikumar
> > > >
> > > > On Tue, Oct 17, 2017 at 7:42 AM, Dong Lin <lindon...@gmail.com>
> wrote:
> > > >
> > > > > Thanks for the KIP. +1 (non-binding)
> > > > >
> > > > > On Wed, Oct 11, 2017 at 2:27 AM, Ted Yu <yuzhih...@gmail.com>
> wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > On Mon, Oct 2, 2017 at 10:51 PM, Paolo Patierno <
> > ppatie...@live.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I didn't see any further discussion around this KIP, so I'd
> like
> > to
> > > > > start
> > > > > > > the vote for it.
> > > > > > >
> > > > > > > Just for reference : https://cwiki.apache.org/
> > > > > > > confluence/display/KAFKA/KIP-204+%3A+adding+records+
> > > > > > > deletion+operation+to+the+new+Admin+Client+API
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Paolo Patierno
> > > > > > > Senior Software Engineer (IoT) @ Red Hat
> > > > > > > Microsoft MVP on Azure & IoT
> > > > > > > Microsoft Azure Advisor
> > > > > > >
> > > > > > > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > > > > > > Linkedin : paolopatierno<http://it.
> linkedin.com/in/paolopatierno
> > >
> > > > > > > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: Metadata class doesn't "expose" topics with errors

[Paolo Patierno]

Hi Guozhang,

thanks ! Really appreciated !
Yes I think that at this point, having an implementation proposal, it makes 
more sense to comment on the PR directly.

Thanks
Paolo

From: Guozhang Wang <wangg...@gmail.com>
Sent: Tuesday, October 31, 2017 10:00:22 PM
To: dev@kafka.apache.org
Subject: Re: Metadata class doesn't "expose" topics with errors

Hello Paolo,

I'm looking at your PR for KIP-204 now. Will reply on the discussion thread
/ PR diff file directly if I find anything.


Guozhang

On Tue, Oct 24, 2017 at 5:45 AM, Paolo Patierno <ppatie...@live.com> wrote:

> Hi Guozhang,
>
> thanks for replying !
>
>
> I see your point about the Metadata class which doesn't need to expose
> errors because transient.
>
>
> Regarding the KIP-204, the delete operations in the "legacy" client
> doesn't have any retry logic but it just returns the error to the user
> which should retry himself (on topics where the operation failed).
>
> If I should add a retry logic in the "new" admin client, considering a
> delete records operation on more topics partitions at same time, I should
> retry if at least one of the topics partitions will come with a
> LEADER_NOT_AVAILABLE (after metadata request), without going on with other
> topic partitions which have leaders.
>
> Maybe it's better to continue with the operations on such topics and come
> back to the user with a LEADER_NOT_AVAILABLE for the others (it's the
> current behaviour with "legacy" admin client).
>
>
> For now the current implementation I have (I'll push a PR soon), use the
> Call class for sending a MetadataRequest and then its handleResponse for
> using another Call instance for sending the DeleteRecordsRequest.
>
>
> Thanks
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Guozhang Wang <wangg...@gmail.com>
> Sent: Tuesday, October 24, 2017 12:52 AM
> To: dev@kafka.apache.org
> Subject: Re: Metadata class doesn't "expose" topics with errors
>
> Hello Paolo,
>
> The reason we filtered the errors in the topics in the generated Cluster is
> that Metadata and its "fetch()" returned Cluster is a common class that is
> used among all clients (producer, consumer, connect, streams, admin), and
> is treated as a high-level representation of the current snapshot of the
> hosted topic information of the cluster, and hence we intentionally exclude
> any transient errors in the representation to abstract such issues away
> from its users.
>
> As for your implementation on KIP-204, I think just wait-and-retry for the
> updated metadata.fetch() Cluster contain the leader information for the
> topic is fine: since if a LEADER_NOT_AVAILABLE is returned you'll need to
> backoff and retry anyways, right?
>
>
> Guozhang
>
>
>
> On Mon, Oct 23, 2017 at 2:36 AM, Paolo Patierno <ppatie...@live.com>
> wrote:
>
> > Finally another plan could be to use nesting of runnable calls.
> >
> > The first one for asking metadata (using the MetadataRequest which
> > provides us all the errors) and then sending the delete records requests
> in
> > the handleResponse() of such metadata request.
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Azure & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
> >
> > 
> > From: Paolo Patierno <ppatie...@live.com>
> > Sent: Monday, October 23, 2017 9:06 AM
> > To: dev@kafka.apache.org
> > Subject: Metadata class doesn't "expose" topics with errors
> >
> > Hi devs,
> >
> > while developing the KIP-204 (having delete records operation in the
> "new"
> > Admin Client) I'm facing with the following doubt (or maybe a lack of
> info)
> > ...
> >
> >
> > As described by KIP-107 (which implements this feature at protocol level
> > and in the "legacy" Admin Client), the request needs to be sent to the
> > leader.
> >
> >
> > For both KIPs, the operation has a Map<TopicPartition, offset> (

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Hi Colin,
thanks !

This morning (Italy time zone) I started the vote for this KIP. Up to now there 
are 5 non-binding votes.

In any case, I'll update the section you mentioned. I totally agree with you on 
giving more info to developers who are using the Scala API.

Thanks
Paolo

From: Colin McCabe <cmcc...@apache.org>
Sent: Tuesday, October 31, 2017 9:49:59 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

Hi Paolo,

This looks like a good proposal.  I think it's probably ready to take it
to a vote soon?

Also, in the "Compatibility, Deprecation, and Migration Plan" section,
you might want to mention the internal Scala interface for doing this
which was added in KIP-107.  We should expect users to migrate from that
interface to the new one over time.

best,
Colin


On Wed, Oct 25, 2017, at 03:47, Paolo Patierno wrote:
> Thanks for all your feedback guys. I have updated my current code as
> well.
>
> I know that the vote for this KIP is not started yet (actually I opened
> it due to no feedback on this KIP after a while but then the discussion
> started and it was really useful !) but I have already opened a PR for
> that.
>
> Maybe feedback could be useful on that as well :
>
>
> https://github.com/apache/kafka/pull/4132
>
>
> Thanks
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Colin McCabe <cmcc...@apache.org>
> Sent: Monday, October 23, 2017 4:34 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> new Admin Client API
>
> On Mon, Oct 23, 2017, at 01:37, Tom Bentley wrote:
> > At the risk of muddying the waters further, have you considered
> > "RecordsToDelete" as the name of the class? It's both shorter and more
> > descriptive imho.
>
> +1 for RecordsToDelete
>
> >
> > Also "deleteBefore()" as the factory method name isn't very future proof
> > if
> > we came to support time-based deletion. Something like "beforeOffset()"
> > would be clearer, imho.
>
> Great idea.
>
> best,
> Colin
>
> >
> > Putting these together: RecordsToDelete.beforeOffset() seems much clearer
> > to me than DeleteRecordsTarget.deleteBefore()
> >
> >
> > On 23 October 2017 at 08:45, Paolo Patierno <ppatie...@live.com> wrote:
> >
> > > About the name I just started to have a doubt about DeletetionTarget
> > > because it could be bounded to any deletion operation (i.e. delete topic,
> > > ...) and not just what we want now, so records deletion.
> > >
> > > I have updated the KIP-204 using DeleteRecordsTarget so it's clear that
> > > it's related to the delete records operation and what it means, so the
> > > target for such operation.
> > >
> > >
> > > Paolo Patierno
> > > Senior Software Engineer (IoT) @ Red Hat
> > > Microsoft MVP on Azure & IoT
> > > Microsoft Azure Advisor
> > >
> > > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> > >
> > >
> > > 
> > > From: Paolo Patierno <ppatie...@live.com>
> > > Sent: Monday, October 23, 2017 7:38 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> > > new Admin Client API
> > >
> > > Hi Colin,
> > >
> > > I was using the long primitive in the code but not updated the KIP yet,
> > > sorry ... now it's updated !
> > >
> > > At same time I agree on using DeletionTarget ... KIP updated !
> > >
> > >
> > > Regarding the deleteBefore factory method, it's a pattern already used
> > > witn NewPartitions.increaseTo which I think it's really clear and give us
> > > more possibility to evolve this DeletionTarget class if we'll add 
> > > different
> > > ways to specify such target not only offset based.
> > >
> > >
> > > Thanks,
> > >
> > >
> > > Paolo Patierno
> > > Sen

Re: [VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Hi all,


because I don't see any further discussion around KIP-204 
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-204+%3A+adding+records+deletion+operation+to+the+new+Admin+Client+API)
 and I have already opened a PR with the implementation, can we re-cover the 
vote started on October 18 ?

There are only "non binding" votes up to now.

Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Viktor Somogyi <viktorsomo...@gmail.com>
Sent: Wednesday, October 18, 2017 10:49 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-204 : adding records deletion operation to the new 
Admin Client API

+1 (non-binding)

On Wed, Oct 18, 2017 at 8:23 AM, Manikumar <manikumar.re...@gmail.com>
wrote:

> + (non-binding)
>
>
> Thanks,
> Manikumar
>
> On Tue, Oct 17, 2017 at 7:42 AM, Dong Lin <lindon...@gmail.com> wrote:
>
> > Thanks for the KIP. +1 (non-binding)
> >
> > On Wed, Oct 11, 2017 at 2:27 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> >
> > > +1
> > >
> > > On Mon, Oct 2, 2017 at 10:51 PM, Paolo Patierno <ppatie...@live.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I didn't see any further discussion around this KIP, so I'd like to
> > start
> > > > the vote for it.
> > > >
> > > > Just for reference : https://cwiki.apache.org/
> > > > confluence/display/KAFKA/KIP-204+%3A+adding+records+
> > > > deletion+operation+to+the+new+Admin+Client+API
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Paolo Patierno
> > > > Senior Software Engineer (IoT) @ Red Hat
> > > > Microsoft MVP on Azure & IoT
> > > > Microsoft Azure Advisor
> > > >
> > > > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > > > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > > > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Thanks for all your feedback guys. I have updated my current code as well.

I know that the vote for this KIP is not started yet (actually I opened it due 
to no feedback on this KIP after a while but then the discussion started and it 
was really useful !) but I have already opened a PR for that.

Maybe feedback could be useful on that as well :


https://github.com/apache/kafka/pull/4132


Thanks


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Colin McCabe <cmcc...@apache.org>
Sent: Monday, October 23, 2017 4:34 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

On Mon, Oct 23, 2017, at 01:37, Tom Bentley wrote:
> At the risk of muddying the waters further, have you considered
> "RecordsToDelete" as the name of the class? It's both shorter and more
> descriptive imho.

+1 for RecordsToDelete

>
> Also "deleteBefore()" as the factory method name isn't very future proof
> if
> we came to support time-based deletion. Something like "beforeOffset()"
> would be clearer, imho.

Great idea.

best,
Colin

>
> Putting these together: RecordsToDelete.beforeOffset() seems much clearer
> to me than DeleteRecordsTarget.deleteBefore()
>
>
> On 23 October 2017 at 08:45, Paolo Patierno <ppatie...@live.com> wrote:
>
> > About the name I just started to have a doubt about DeletetionTarget
> > because it could be bounded to any deletion operation (i.e. delete topic,
> > ...) and not just what we want now, so records deletion.
> >
> > I have updated the KIP-204 using DeleteRecordsTarget so it's clear that
> > it's related to the delete records operation and what it means, so the
> > target for such operation.
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Azure & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
> >
> > 
> > From: Paolo Patierno <ppatie...@live.com>
> > Sent: Monday, October 23, 2017 7:38 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> > new Admin Client API
> >
> > Hi Colin,
> >
> > I was using the long primitive in the code but not updated the KIP yet,
> > sorry ... now it's updated !
> >
> > At same time I agree on using DeletionTarget ... KIP updated !
> >
> >
> > Regarding the deleteBefore factory method, it's a pattern already used
> > witn NewPartitions.increaseTo which I think it's really clear and give us
> > more possibility to evolve this DeletionTarget class if we'll add different
> > ways to specify such target not only offset based.
> >
> >
> > Thanks,
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Azure & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
> >
> > 
> > From: Colin McCabe <cmcc...@apache.org>
> > Sent: Friday, October 20, 2017 8:18 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> > new Admin Client API
> >
> > > /** Describe records to delete */
> >  > public class DeleteRecords {
> >  > private Long offset;
> >
> > "DeleteRecords" doesn't really explain what the class is, though.  How
> > about "DeletionTarget"?  Also, why do we need a Long object rather than
> > a long primitive?
> >
> >  >
> >  > /**
> >  > * Delete all the records before the given {@code offset}
> >  > */
> >  > public static DeleteRecords deleteBefore(Long offset) { ... }
> >
> > This seems confusing to me.  What's wrong with a regular constructor for
> > DeletionTarget?
> >
> > best,
> > Colin
> >
> >
> > On Fri, Oct 20, 201

Re: Metadata class doesn't "expose" topics with errors

[Paolo Patierno]

Hi Guozhang,

thanks for replying !


I see your point about the Metadata class which doesn't need to expose errors 
because transient.


Regarding the KIP-204, the delete operations in the "legacy" client doesn't 
have any retry logic but it just returns the error to the user which should 
retry himself (on topics where the operation failed).

If I should add a retry logic in the "new" admin client, considering a delete 
records operation on more topics partitions at same time, I should retry if at 
least one of the topics partitions will come with a LEADER_NOT_AVAILABLE (after 
metadata request), without going on with other topic partitions which have 
leaders.

Maybe it's better to continue with the operations on such topics and come back 
to the user with a LEADER_NOT_AVAILABLE for the others (it's the current 
behaviour with "legacy" admin client).


For now the current implementation I have (I'll push a PR soon), use the Call 
class for sending a MetadataRequest and then its handleResponse for using 
another Call instance for sending the DeleteRecordsRequest.


Thanks


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Guozhang Wang <wangg...@gmail.com>
Sent: Tuesday, October 24, 2017 12:52 AM
To: dev@kafka.apache.org
Subject: Re: Metadata class doesn't "expose" topics with errors

Hello Paolo,

The reason we filtered the errors in the topics in the generated Cluster is
that Metadata and its "fetch()" returned Cluster is a common class that is
used among all clients (producer, consumer, connect, streams, admin), and
is treated as a high-level representation of the current snapshot of the
hosted topic information of the cluster, and hence we intentionally exclude
any transient errors in the representation to abstract such issues away
from its users.

As for your implementation on KIP-204, I think just wait-and-retry for the
updated metadata.fetch() Cluster contain the leader information for the
topic is fine: since if a LEADER_NOT_AVAILABLE is returned you'll need to
backoff and retry anyways, right?


Guozhang



On Mon, Oct 23, 2017 at 2:36 AM, Paolo Patierno <ppatie...@live.com> wrote:

> Finally another plan could be to use nesting of runnable calls.
>
> The first one for asking metadata (using the MetadataRequest which
> provides us all the errors) and then sending the delete records requests in
> the handleResponse() of such metadata request.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Paolo Patierno <ppatie...@live.com>
> Sent: Monday, October 23, 2017 9:06 AM
> To: dev@kafka.apache.org
> Subject: Metadata class doesn't "expose" topics with errors
>
> Hi devs,
>
> while developing the KIP-204 (having delete records operation in the "new"
> Admin Client) I'm facing with the following doubt (or maybe a lack of info)
> ...
>
>
> As described by KIP-107 (which implements this feature at protocol level
> and in the "legacy" Admin Client), the request needs to be sent to the
> leader.
>
>
> For both KIPs, the operation has a Map<TopicPartition, offset> (offset is
> a long in the "legacy" API but it's becoming to be a class in the "new"
> API) and in order to reduce the number of requests to different leaders, my
> code groups partitions having same leader so having a Map<Node,
> Map<TopicPartition, offset>>.
>
>
> In order to know the leaders I need to request metadata and there are two
> ways for doing that :
>
>
>   *   using something like the producer does with Metadata class, putting
> the topics, request update and waiting for it
>   *   using the low level MetadataRequest and handling the related
> response (which is what the "legacy" API does today)
>
> I noticed that building the Cluster object from the MetadataResponse, the
> topics with errors are skipped and it means that in the final "high level"
> Metadata class (fetching the Cluster object) there is no information about
> them. So with first solution we have no info about topics with errors
> (maybe the only errors I'm able to handle is the "LEADER_NOT_AVAILABLE", if
> leaderFor

Re: Metadata class doesn't "expose" topics with errors

[Paolo Patierno]

Finally another plan could be to use nesting of runnable calls.

The first one for asking metadata (using the MetadataRequest which provides us 
all the errors) and then sending the delete records requests in the 
handleResponse() of such metadata request.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


____
From: Paolo Patierno <ppatie...@live.com>
Sent: Monday, October 23, 2017 9:06 AM
To: dev@kafka.apache.org
Subject: Metadata class doesn't "expose" topics with errors

Hi devs,

while developing the KIP-204 (having delete records operation in the "new" 
Admin Client) I'm facing with the following doubt (or maybe a lack of info) ...


As described by KIP-107 (which implements this feature at protocol level and in 
the "legacy" Admin Client), the request needs to be sent to the leader.


For both KIPs, the operation has a Map<TopicPartition, offset> (offset is a 
long in the "legacy" API but it's becoming to be a class in the "new" API) and 
in order to reduce the number of requests to different leaders, my code groups 
partitions having same leader so having a Map<Node, Map<TopicPartition, 
offset>>.


In order to know the leaders I need to request metadata and there are two ways 
for doing that :


  *   using something like the producer does with Metadata class, putting the 
topics, request update and waiting for it
  *   using the low level MetadataRequest and handling the related response 
(which is what the "legacy" API does today)

I noticed that building the Cluster object from the MetadataResponse, the 
topics with errors are skipped and it means that in the final "high level" 
Metadata class (fetching the Cluster object) there is no information about 
them. So with first solution we have no info about topics with errors (maybe 
the only errors I'm able to handle is the "LEADER_NOT_AVAILABLE", if 
leaderFor() on the Cluster returns a null Node).

Is there any specific reason why "topics with errors" are not exposed in the 
Metadata instance ?
Is the preferred pattern using the low level protocol stuff in such case ?

Thanks


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

Re: Metadata class doesn't "expose" topics with errors

[Paolo Patierno]

I'd like just to add that in my specific case I could resolve creating a new 
NodeProvider implementation like LeaderForProvider in order to send the request 
to the right leader for each topic.

The drawback of this solution is not grouping topics for the same leader so not 
having only one requests but different of them (even if for the same leader).


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


____
From: Paolo Patierno <ppatie...@live.com>
Sent: Monday, October 23, 2017 9:06 AM
To: dev@kafka.apache.org
Subject: Metadata class doesn't "expose" topics with errors

Hi devs,

while developing the KIP-204 (having delete records operation in the "new" 
Admin Client) I'm facing with the following doubt (or maybe a lack of info) ...


As described by KIP-107 (which implements this feature at protocol level and in 
the "legacy" Admin Client), the request needs to be sent to the leader.


For both KIPs, the operation has a Map<TopicPartition, offset> (offset is a 
long in the "legacy" API but it's becoming to be a class in the "new" API) and 
in order to reduce the number of requests to different leaders, my code groups 
partitions having same leader so having a Map<Node, Map<TopicPartition, 
offset>>.


In order to know the leaders I need to request metadata and there are two ways 
for doing that :


  *   using something like the producer does with Metadata class, putting the 
topics, request update and waiting for it
  *   using the low level MetadataRequest and handling the related response 
(which is what the "legacy" API does today)

I noticed that building the Cluster object from the MetadataResponse, the 
topics with errors are skipped and it means that in the final "high level" 
Metadata class (fetching the Cluster object) there is no information about 
them. So with first solution we have no info about topics with errors (maybe 
the only errors I'm able to handle is the "LEADER_NOT_AVAILABLE", if 
leaderFor() on the Cluster returns a null Node).

Is there any specific reason why "topics with errors" are not exposed in the 
Metadata instance ?
Is the preferred pattern using the low level protocol stuff in such case ?

Thanks


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

Metadata class doesn't "expose" topics with errors

[Paolo Patierno]

Hi devs,

while developing the KIP-204 (having delete records operation in the "new" 
Admin Client) I'm facing with the following doubt (or maybe a lack of info) ...


As described by KIP-107 (which implements this feature at protocol level and in 
the "legacy" Admin Client), the request needs to be sent to the leader.


For both KIPs, the operation has a Map<TopicPartition, offset> (offset is a 
long in the "legacy" API but it's becoming to be a class in the "new" API) and 
in order to reduce the number of requests to different leaders, my code groups 
partitions having same leader so having a Map<Node, Map<TopicPartition, 
offset>>.


In order to know the leaders I need to request metadata and there are two ways 
for doing that :


  *   using something like the producer does with Metadata class, putting the 
topics, request update and waiting for it
  *   using the low level MetadataRequest and handling the related response 
(which is what the "legacy" API does today)

I noticed that building the Cluster object from the MetadataResponse, the 
topics with errors are skipped and it means that in the final "high level" 
Metadata class (fetching the Cluster object) there is no information about 
them. So with first solution we have no info about topics with errors (maybe 
the only errors I'm able to handle is the "LEADER_NOT_AVAILABLE", if 
leaderFor() on the Cluster returns a null Node).

Is there any specific reason why "topics with errors" are not exposed in the 
Metadata instance ?
Is the preferred pattern using the low level protocol stuff in such case ?

Thanks


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

About the name I just started to have a doubt about DeletetionTarget because it 
could be bounded to any deletion operation (i.e. delete topic, ...) and not 
just what we want now, so records deletion.

I have updated the KIP-204 using DeleteRecordsTarget so it's clear that it's 
related to the delete records operation and what it means, so the target for 
such operation.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


____
From: Paolo Patierno <ppatie...@live.com>
Sent: Monday, October 23, 2017 7:38 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

Hi Colin,

I was using the long primitive in the code but not updated the KIP yet, sorry 
... now it's updated !

At same time I agree on using DeletionTarget ... KIP updated !


Regarding the deleteBefore factory method, it's a pattern already used witn 
NewPartitions.increaseTo which I think it's really clear and give us more 
possibility to evolve this DeletionTarget class if we'll add different ways to 
specify such target not only offset based.


Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Colin McCabe <cmcc...@apache.org>
Sent: Friday, October 20, 2017 8:18 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

> /** Describe records to delete */
 > public class DeleteRecords {
 > private Long offset;

"DeleteRecords" doesn't really explain what the class is, though.  How
about "DeletionTarget"?  Also, why do we need a Long object rather than
a long primitive?

 >
 > /**
 > * Delete all the records before the given {@code offset}
 > */
 > public static DeleteRecords deleteBefore(Long offset) { ... }

This seems confusing to me.  What's wrong with a regular constructor for
DeletionTarget?

best,
Colin


On Fri, Oct 20, 2017, at 01:28, Paolo Patierno wrote:
> Hi all,
>
>
> I have just updated the KIP with your suggestion.
>
> I'm going to continue implementation and tests with these changes,
> waiting for further discussion.
>
>
> Thanks,
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Paolo Patierno <ppatie...@live.com>
> Sent: Thursday, October 19, 2017 1:37 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> new Admin Client API
>
> @Colin Yes you are right I'll update the KIP-204 mentioning the related
> ACL permission DELETE on TOPIC
>
>
> @Dong @Ismael
>
>
> Considering future improvements on this, it makes sense to me using a
> class instead of a Long.
>
> Maybe the name could be just "DeleteRecords" (as "NewPartitions") having
> a deleteBefore(Long) factory method for a simple creation when you need
> to delete before the specified offset.
>
>
> Thanks,
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Colin McCabe <cmcc...@apache.org>
> Sent: Wednesday, October 18, 2017 3:58 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> new Admin Client API
>
> Having a class there makes sense to me.  It also helps clarify what the
> Long represents (a record offset).
>
> regards,
> Colin
>
>
> On Wed, Oct 18, 2017, at 06:19, Dong Lin wrote:
> > Sure. This makes sense. I agree it is better to replace Long with a new
> > class.
> >
> > On Wed, Oct 18, 2017 at 6:16 AM, Ismael Juma <ism...@juma.me.uk> wrote:
> >
> &

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Hi Colin,

I was using the long primitive in the code but not updated the KIP yet, sorry 
... now it's updated !

At same time I agree on using DeletionTarget ... KIP updated !


Regarding the deleteBefore factory method, it's a pattern already used witn 
NewPartitions.increaseTo which I think it's really clear and give us more 
possibility to evolve this DeletionTarget class if we'll add different ways to 
specify such target not only offset based.


Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Colin McCabe <cmcc...@apache.org>
Sent: Friday, October 20, 2017 8:18 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

> /** Describe records to delete */
 > public class DeleteRecords {
 > private Long offset;

"DeleteRecords" doesn't really explain what the class is, though.  How
about "DeletionTarget"?  Also, why do we need a Long object rather than
a long primitive?

 >
 > /**
 > * Delete all the records before the given {@code offset}
 > */
 > public static DeleteRecords deleteBefore(Long offset) { ... }

This seems confusing to me.  What's wrong with a regular constructor for
DeletionTarget?

best,
Colin


On Fri, Oct 20, 2017, at 01:28, Paolo Patierno wrote:
> Hi all,
>
>
> I have just updated the KIP with your suggestion.
>
> I'm going to continue implementation and tests with these changes,
> waiting for further discussion.
>
>
> Thanks,
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Paolo Patierno <ppatie...@live.com>
> Sent: Thursday, October 19, 2017 1:37 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> new Admin Client API
>
> @Colin Yes you are right I'll update the KIP-204 mentioning the related
> ACL permission DELETE on TOPIC
>
>
> @Dong @Ismael
>
>
> Considering future improvements on this, it makes sense to me using a
> class instead of a Long.
>
> Maybe the name could be just "DeleteRecords" (as "NewPartitions") having
> a deleteBefore(Long) factory method for a simple creation when you need
> to delete before the specified offset.
>
>
> Thanks,
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Colin McCabe <cmcc...@apache.org>
> Sent: Wednesday, October 18, 2017 3:58 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> new Admin Client API
>
> Having a class there makes sense to me.  It also helps clarify what the
> Long represents (a record offset).
>
> regards,
> Colin
>
>
> On Wed, Oct 18, 2017, at 06:19, Dong Lin wrote:
> > Sure. This makes sense. I agree it is better to replace Long with a new
> > class.
> >
> > On Wed, Oct 18, 2017 at 6:16 AM, Ismael Juma <ism...@juma.me.uk> wrote:
> >
> > > Hi Dong,
> > >
> > > Yes, I mean replacing the `Long` with a class in the map. The class would
> > > have static factory methods for the various use cases. To use the
> > > `createPartitions` example, there is a `NewPartitions.increaseTo` method.
> > >
> > > Not sure why you think it's too complicated. It provides better type
> > > safety, it's more informative and makes it easier to evolve. Thankfully
> > > Java has lambdas for a while now and mapping a collection from one type to
> > > another is reasonably simple.
> > >
> > > Your suggestion doesn't work because both methods would have the same
> > > "erased" signature. You can't have two overloaded methods that have the
> > > same signature apart from generic parameters. Also, we'd end up with 2
> > > methods in AdminClient.

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Hi all,


I have just updated the KIP with your suggestion.

I'm going to continue implementation and tests with these changes, waiting for 
further discussion.


Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


____
From: Paolo Patierno <ppatie...@live.com>
Sent: Thursday, October 19, 2017 1:37 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

@Colin Yes you are right I'll update the KIP-204 mentioning the related ACL 
permission DELETE on TOPIC


@Dong @Ismael


Considering future improvements on this, it makes sense to me using a class 
instead of a Long.

Maybe the name could be just "DeleteRecords" (as "NewPartitions") having a 
deleteBefore(Long) factory method for a simple creation when you need to delete 
before the specified offset.


Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Colin McCabe <cmcc...@apache.org>
Sent: Wednesday, October 18, 2017 3:58 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

Having a class there makes sense to me.  It also helps clarify what the
Long represents (a record offset).

regards,
Colin


On Wed, Oct 18, 2017, at 06:19, Dong Lin wrote:
> Sure. This makes sense. I agree it is better to replace Long with a new
> class.
>
> On Wed, Oct 18, 2017 at 6:16 AM, Ismael Juma <ism...@juma.me.uk> wrote:
>
> > Hi Dong,
> >
> > Yes, I mean replacing the `Long` with a class in the map. The class would
> > have static factory methods for the various use cases. To use the
> > `createPartitions` example, there is a `NewPartitions.increaseTo` method.
> >
> > Not sure why you think it's too complicated. It provides better type
> > safety, it's more informative and makes it easier to evolve. Thankfully
> > Java has lambdas for a while now and mapping a collection from one type to
> > another is reasonably simple.
> >
> > Your suggestion doesn't work because both methods would have the same
> > "erased" signature. You can't have two overloaded methods that have the
> > same signature apart from generic parameters. Also, we'd end up with 2
> > methods in AdminClient.
> >
> > Ismael
> >
> >
> > On Wed, Oct 18, 2017 at 1:42 PM, Dong Lin <lindon...@gmail.com> wrote:
> >
> > > Hey Ismael,
> > >
> > > To clarify, I think you are saying that we should replace
> > > "deleteRecords(Map<TopicPartition, Long> partitionsAndOffsets)" with
> > > "deleteRecords(Map<TopicPartition, DeleteRecordsParameter>
> > > partitionsAndOffsets)", where DeleteRecordsParameter should be include a
> > > "Long value", and probably "Boolean isBeforeOrAfter" and "Boolean
> > > isOffsetOrTime" in the future.
> > >
> > > I get the point that we want to only include additional parameter
> > > in DeleteRecordsOptions. I just feel it is a bit overkill to have a new
> > > class DeleteRecordsParameter which will only support offset in the near
> > > future. This method signature seems a bit too complicated.
> > >
> > > How about we use deleteRecords(Map<TopicPartition, Long>
> > > partitionsAndOffsets) for now, and add deleteRecords(Map<TopicPartition,
> > > DeleteRecordsParameter> partitionsAndOffsets) when we need to support
> > core
> > > parameter in the future? By doing this we can make user's experience
> > better
> > > (i.e. not having to instantiate DeleteRecordsParameter) until it is
> > > necessary to be more complicated.
> > >
> > > Dong
> > >
> > > On Wed, Oct 18, 2017 at 4:46 AM, Ismael Juma <ism...@juma.me.uk> wrote:
> > >
> > > > Hi Dong,
> > > >
> > > > I think it makes sense to use the parameters to define the specifics of
> > > the
> > > > request. However, we would probably want to replace the `Long` with a
> > > class
> > > > (similar to `createPartitions`) instead of relying on
&

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

@Colin Yes you are right I'll update the KIP-204 mentioning the related ACL 
permission DELETE on TOPIC


@Dong @Ismael


Considering future improvements on this, it makes sense to me using a class 
instead of a Long.

Maybe the name could be just "DeleteRecords" (as "NewPartitions") having a 
deleteBefore(Long) factory method for a simple creation when you need to delete 
before the specified offset.


Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Colin McCabe <cmcc...@apache.org>
Sent: Wednesday, October 18, 2017 3:58 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

Having a class there makes sense to me.  It also helps clarify what the
Long represents (a record offset).

regards,
Colin


On Wed, Oct 18, 2017, at 06:19, Dong Lin wrote:
> Sure. This makes sense. I agree it is better to replace Long with a new
> class.
>
> On Wed, Oct 18, 2017 at 6:16 AM, Ismael Juma <ism...@juma.me.uk> wrote:
>
> > Hi Dong,
> >
> > Yes, I mean replacing the `Long` with a class in the map. The class would
> > have static factory methods for the various use cases. To use the
> > `createPartitions` example, there is a `NewPartitions.increaseTo` method.
> >
> > Not sure why you think it's too complicated. It provides better type
> > safety, it's more informative and makes it easier to evolve. Thankfully
> > Java has lambdas for a while now and mapping a collection from one type to
> > another is reasonably simple.
> >
> > Your suggestion doesn't work because both methods would have the same
> > "erased" signature. You can't have two overloaded methods that have the
> > same signature apart from generic parameters. Also, we'd end up with 2
> > methods in AdminClient.
> >
> > Ismael
> >
> >
> > On Wed, Oct 18, 2017 at 1:42 PM, Dong Lin <lindon...@gmail.com> wrote:
> >
> > > Hey Ismael,
> > >
> > > To clarify, I think you are saying that we should replace
> > > "deleteRecords(Map<TopicPartition, Long> partitionsAndOffsets)" with
> > > "deleteRecords(Map<TopicPartition, DeleteRecordsParameter>
> > > partitionsAndOffsets)", where DeleteRecordsParameter should be include a
> > > "Long value", and probably "Boolean isBeforeOrAfter" and "Boolean
> > > isOffsetOrTime" in the future.
> > >
> > > I get the point that we want to only include additional parameter
> > > in DeleteRecordsOptions. I just feel it is a bit overkill to have a new
> > > class DeleteRecordsParameter which will only support offset in the near
> > > future. This method signature seems a bit too complicated.
> > >
> > > How about we use deleteRecords(Map<TopicPartition, Long>
> > > partitionsAndOffsets) for now, and add deleteRecords(Map<TopicPartition,
> > > DeleteRecordsParameter> partitionsAndOffsets) when we need to support
> > core
> > > parameter in the future? By doing this we can make user's experience
> > better
> > > (i.e. not having to instantiate DeleteRecordsParameter) until it is
> > > necessary to be more complicated.
> > >
> > > Dong
> > >
> > > On Wed, Oct 18, 2017 at 4:46 AM, Ismael Juma <ism...@juma.me.uk> wrote:
> > >
> > > > Hi Dong,
> > > >
> > > > I think it makes sense to use the parameters to define the specifics of
> > > the
> > > > request. However, we would probably want to replace the `Long` with a
> > > class
> > > > (similar to `createPartitions`) instead of relying on
> > > > `DeleteRecordsOptions`. The latter is typically used for defining
> > > > additional options, not for defining the core behaviour.
> > > >
> > > > Ismael
> > > >
> > > > On Wed, Oct 18, 2017 at 12:27 AM, Dong Lin <lindon...@gmail.com>
> > wrote:
> > > >
> > > > > Hey Colin,
> > > > >
> > > > > I have also thought about deleteRecordsBeforeOffset so that we can
> > keep
> > > > the
> > > > > name consistent with the existing API in the Scala AdminClient. But
> > > then
> > > > I
> > > > > think it m

Re: [VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Hi all,


since I started voting KIP-204 on October 3rd I haven't seen any votes on that. 
I know you are busy on 1.0.0 release, just as reminder 


Thanks.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


____
From: Paolo Patierno <ppatie...@live.com>
Sent: Tuesday, October 3, 2017 5:51 AM
To: dev@kafka.apache.org
Subject: [VOTE] KIP-204 : adding records deletion operation to the new Admin 
Client API

Hi all,

I didn't see any further discussion around this KIP, so I'd like to start the 
vote for it.

Just for reference : 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-204+%3A+adding+records+deletion+operation+to+the+new+Admin+Client+API


Thanks,

Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

[Paolo Patierno]

Just as reminder for other committers in order to have other binding votes, for 
now we have ...


binding

+1 Ismael Juma

+1 Guozhang Wang


non binding

+1 Mickael Maison

+1 Vahid Hashemian


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma 
<ism...@juma.me.uk>
Sent: Tuesday, September 26, 2017 7:46 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

Removals can only happen in major releases.

Ismael

On Tue, Sep 26, 2017 at 8:37 AM, Paolo Patierno <ppatie...@live.com> wrote:

> Hi devs,
>
> I know that we are already voting for this (+1 bindings from Ismael Juma
> and Guozhang Wang, +1 non binding from Mickael Maison) but I'd like to ask
> a question about the possible cycle release for this change.
>
> We are really closed to the 1.0.0 release which will have the
> --new-consumer deprecation. The current KIP-176 proposes to remove it in
> the 2.0.0 release but I'm starting to think that it could happen even in
> one year or so while we could have more releases in the middle (1.x.y)
> every 4 months.
>
> Maybe we could have this KIP-176 included in the 1.1.0 release ? Wdyt ?
>
>
> Thanks.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Mickael Maison <mickael.mai...@gmail.com>
> Sent: Thursday, September 7, 2017 9:53 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for
> tools
>
> +1 (non binding)
> Thanks
>
> On Thu, Sep 7, 2017 at 9:09 AM, Paolo Patierno <ppatie...@live.com> wrote:
> > KIP updated to clarify it will be removed in the 2.0.0 version.
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
> >
> > 
> > From: Vahid S Hashemian <vahidhashem...@us.ibm.com>
> > Sent: Wednesday, September 6, 2017 11:45 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for
> tools
> >
> > +1. Thanks for the KIP.
> >
> > --Vahid
> >
> >
> >
> > From:   Guozhang Wang <wangg...@gmail.com>
> > To: "dev@kafka.apache.org" <dev@kafka.apache.org>
> > Date:   09/06/2017 03:41 PM
> > Subject:Re: [VOTE] KIP-176 : Remove deprecated new-consumer
> option
> > for tools
> >
> >
> >
> > +1. Thanks.
> >
> > On Wed, Sep 6, 2017 at 7:57 AM, Ismael Juma <ism...@juma.me.uk> wrote:
> >
> >> Thanks for the KIP. +1 (binding). Please make it clear in the KIP that
> >> removal will happen in 2.0.0.
> >>
> >> Ismael
> >>
> >> On Tue, Aug 8, 2017 at 11:53 AM, Paolo Patierno <ppatie...@live.com>
> >> wrote:
> >>
> >> > Hi devs,
> >> >
> >> >
> >> > I didn't see any more comments about this KIP. The JIRAs related to
> > the
> >> > first step (so making --new-consumer as deprecated with warning
> > messages)
> >> > are merged.
> >> >
> >> > I'd like to start a vote for this KIP.
> >> >
> >> >
> >> > Thanks,
> >> >
> >> >
> >> > Paolo Patierno
> >> > Senior Software Engineer (IoT) @ Red Hat
> >> > Microsoft MVP on Windows Embedded & IoT
> >> > Microsoft Azure Advisor
> >> >
> >> > Twitter : @ppatierno<
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.
> com_ppatierno=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_
> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=JR2f85JOCHQwbXDKcQkgD6ay-
> ECdCkrh3HaWqtSjF5w=YUFjd7tCwfVz6mZjZ8KbxeH_yQLhzwBXTmm8Wwf_4Wk=
> >>
> >> > Linkedin : paolopatierno<
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__it.
> linkedin.com_in_paolopatierno=DwIBaQ=jf_iaSHvJObTbx-
> siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=
> JR2f85JOCHQwbXDKcQkgD6ay-ECdCkrh3HaWqtSjF5w=-
> q3fI0mMvYa2M1PyPrZxDOFWZoyt66zllRYNw00vIIk=
> >>
> >> > Blog : DevExperience<
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__
> paolopatierno.wordpress.com_=DwIBaQ=jf_iaSHvJObTbx-
> siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=
> JR2f85JOCHQwbXDKcQkgD6ay-ECdCkrh3HaWqtSjF5w=M0NFIAt5g9yjQXa4D-DHFs4-
> UQ3F0KWHfVFPLIaLAyg=
> >>
> >> >
> >>
> >
> >
> >
> > --
> > -- Guozhang
> >
> >
> >
> >
>

[VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Hi all,

I didn't see any further discussion around this KIP, so I'd like to start the 
vote for it.

Just for reference : 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-204+%3A+adding+records+deletion+operation+to+the+new+Admin+Client+API


Thanks,

Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Hi,


maybe we want to start without the delete records policy for now waiting for a 
real needs. So I'm removing it from the KIP.

I hope for more comments on this KIP-204 so that we can start a vote on Monday.


Thanks.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


____
From: Paolo Patierno <ppatie...@live.com>
Sent: Thursday, September 28, 2017 5:56 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

Hi,


I have just updated the KIP-204 description with the new TopicDeletionPolicy 
suggested by the KIP-201.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


____
From: Paolo Patierno <ppatie...@live.com>
Sent: Tuesday, September 26, 2017 4:57 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

Hi Tom,

as I said in the KIP-201 discussion I'm ok with having a unique 
DeleteTopicPolicy but then it could be useful having more information then just 
the topic name; partitions and offset for messages deletion could be useful for 
a fine grained use cases.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Tom Bentley <t.j.bent...@gmail.com>
Sent: Tuesday, September 26, 2017 4:32 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

Hi Paolo,

I guess a RecordDeletionPolicy should work at the partition level, whereas
the TopicDeletionPolicy should work at the topic level. But then we run
into a similar situation as described in the motivation for KIP-201, where
the administrator might have to write+configure two policies in order to
express their intended rules. For example, it's no good preventing people
from deleting topics if they can delete all the messages in those topics,
or vice versa.

On that reasoning, perhaps there should be a single policy interface
covering topic deletion and message deletion. Alternatively, the topic
deletion API could also invoke the record deletion policy (before the topic
deletion policy I mean). But the former would be more consistent with
what's proposed in KIP-201.

Wdyt? I can add this to KIP-201 if you want.

Cheers,

Tom





On 26 September 2017 at 17:01, Paolo Patierno <ppatie...@live.com> wrote:

> Hi Tom,
>
> I think that we could live with the current authorizer based on delete
> topic (for both, deleting messages and topic as a whole) but then the
> RecordsDeletePolicy could be even more fine grained giving the possibility
> to avoid deleting messages for specific partitions (inside the topic) and
> starting from a specific offset.
>
> I could think on some users solutions where they know exactly what the
> partitions means inside of a specific topic (because they are using a
> custom partitioner on the producer side) so they know what kind of messages
> are inside a partition allowing to delete them but not the other.
>
> In such a policy a user could also check the timestamp related to the
> offset for allowing or not deletion on time base.
>
>
> Wdyt ?
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Tom Bentley <t.j.bent...@gmail.com>
> Sent: Tuesday, September 26, 2017 2:55 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> new Admin Client API
>
> Hi Edoardo and Paolo,
>
>
> On 26 September 2017 at 14:10, Paolo Patierno <ppatie...@live.com> wrote:
>
> > What could be useful use cases for having a RecordsDeletePolicy ? Records
> > can't be deleted for a topic name ? Starting from a specific offset ?
> >
>
> I can imagine some

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Hi,


I have just updated the KIP-204 description with the new TopicDeletionPolicy 
suggested by the KIP-201.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


____
From: Paolo Patierno <ppatie...@live.com>
Sent: Tuesday, September 26, 2017 4:57 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

Hi Tom,

as I said in the KIP-201 discussion I'm ok with having a unique 
DeleteTopicPolicy but then it could be useful having more information then just 
the topic name; partitions and offset for messages deletion could be useful for 
a fine grained use cases.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Tom Bentley <t.j.bent...@gmail.com>
Sent: Tuesday, September 26, 2017 4:32 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

Hi Paolo,

I guess a RecordDeletionPolicy should work at the partition level, whereas
the TopicDeletionPolicy should work at the topic level. But then we run
into a similar situation as described in the motivation for KIP-201, where
the administrator might have to write+configure two policies in order to
express their intended rules. For example, it's no good preventing people
from deleting topics if they can delete all the messages in those topics,
or vice versa.

On that reasoning, perhaps there should be a single policy interface
covering topic deletion and message deletion. Alternatively, the topic
deletion API could also invoke the record deletion policy (before the topic
deletion policy I mean). But the former would be more consistent with
what's proposed in KIP-201.

Wdyt? I can add this to KIP-201 if you want.

Cheers,

Tom





On 26 September 2017 at 17:01, Paolo Patierno <ppatie...@live.com> wrote:

> Hi Tom,
>
> I think that we could live with the current authorizer based on delete
> topic (for both, deleting messages and topic as a whole) but then the
> RecordsDeletePolicy could be even more fine grained giving the possibility
> to avoid deleting messages for specific partitions (inside the topic) and
> starting from a specific offset.
>
> I could think on some users solutions where they know exactly what the
> partitions means inside of a specific topic (because they are using a
> custom partitioner on the producer side) so they know what kind of messages
> are inside a partition allowing to delete them but not the other.
>
> In such a policy a user could also check the timestamp related to the
> offset for allowing or not deletion on time base.
>
>
> Wdyt ?
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Tom Bentley <t.j.bent...@gmail.com>
> Sent: Tuesday, September 26, 2017 2:55 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> new Admin Client API
>
> Hi Edoardo and Paolo,
>
>
> On 26 September 2017 at 14:10, Paolo Patierno <ppatie...@live.com> wrote:
>
> > What could be useful use cases for having a RecordsDeletePolicy ? Records
> > can't be deleted for a topic name ? Starting from a specific offset ?
> >
>
> I can imagine some users wanting to prohibit using this API completely.
> Maybe others divide up the topic namespace according to some scheme, and so
> it would be allowed for some topics, but not for others based on the name.
> Both these could be done using authz, but would be much simpler to express
> using a policy.
>
> Since both deleting messages and deleting topics are authorized using
> delete operation on the topic, using policies it would be possible to allow
> deleting messages from a topic, but not deleting the topic itself.
>
>
> On 26 September 2017 at 15:27, Edoardo Comar <eco...@uk.ibm.com> wrote:
>
> >
> > Our KIP-170 did indeed suggest a TopicDeletePolicy - but, as said, for
> our
> > intent a

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

[Paolo Patierno]

Hi Ismael,


  1.  I don't have a real requirement now but "deleting" is an operation that 
could be really dangerous so it's always better having a way for having more 
control on that. I know that we have the authorizer used for that (delete on 
topic) but fine grained control could be better (even already happens for topic 
deletion).
  2.  I know about the problem of restarting broker due to changes on policies 
but what do you mean by doing that on the clients ?


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma 
<ism...@juma.me.uk>
Sent: Wednesday, September 27, 2017 10:30 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

A couple of questions:

1. Is this a concrete requirement from a user or is it hypothetical?
2. You sure you would want to do this in the broker instead of the clients?
It's worth remembering that updating broker policies involves a rolling
restart of the cluster, so it's not the right place for things that change
frequently.

Ismael

On Wed, Sep 27, 2017 at 11:26 AM, Paolo Patierno <ppatie...@live.com> wrote:

> Hi Ismael,
>
> regarding motivations for delete records, as I said during the discussion
> on KIP-204, it gives the possibility to avoid deleting messages for
> specific partitions (inside the topic) and starting from a specific offset.
> I could think on some users solutions where they know exactly what the
> partitions means in a specific topic (because they are using a custom
> partitioner on the producer side) so they know what kind of messages are
> inside a partition allowing to delete them but not the others.  In such a
> policy a user could also check the timestamp related to the offset for
> allowing or not deletion on time base.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma <
> ism...@juma.me.uk>
> Sent: Wednesday, September 27, 2017 10:18 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-201: Rationalising Policy interfaces
>
> A couple more comments:
>
> 1. "If this KIP is accepted for Kafka 1.1.0 this removal could happen in
> Kafka 1.2.0 or a later release." -> we only remove code in major releases.
> So, if it's deprecated in 1.1.0, it would be removed in 2.0.0.
>
> 2. Deleting all messages in a topic is not really the same as deleting a
> topic. The latter will cause consumers and producers to error out while the
> former will not. It would be good to motivate the need for the delete
> records policy more.
>
> Ismael
>
> On Wed, Sep 27, 2017 at 11:12 AM, Ismael Juma <ism...@juma.me.uk> wrote:
>
> > Another quick comment: the KIP states that having multiple interfaces
> > imply that the logic must be in 2 places. That is not true because the
> same
> > class can implement multiple interfaces (this aspect was considered when
> we
> > decided to introduce policies incrementally).
> >
> > The main reason why I think the original approach doesn't work well is
> > that there is no direct mapping between an operation and the policy. That
> > is, we initially thought we would have create/alter/delete topics, but
> that
> > didn't work out as the alter case is better served by multiple request
> > types. Given that, it's a bit awkward to maintain the original approach
> and
> > a policy for topic management seemed easier to understand. On that note,
> > would `TopicManagementPolicy` be a better name?
> >
> > Looking at the updated KIP, I notice that we actually have a
> > TopicDeletionPolicy with a separate config. That seems to be a halfway
> > house. Not sure about that.
> >
> > Ismael
> >
> > On Wed, Sep 27, 2017 at 10:47 AM, Tom Bentley <t.j.bent...@gmail.com>
> > wrote:
> >
> >> I have updated the KIP to add a common policy interface for topic and
> >> message deletion. This included pulling ClusterState and TopicState
> >> interfaces up to the top level so that they can be shared between the
> two
>

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

[Paolo Patierno]

Hi Tom,


I guess that at


"On topic deletion will be applied on topic and message deletion."


you meant something like "The TopicDeletionPolicy will be applied on topic and 
messages deletion".


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Tom Bentley <t.j.bent...@gmail.com>
Sent: Wednesday, September 27, 2017 9:47 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

I have updated the KIP to add a common policy interface for topic and
message deletion. This included pulling ClusterState and TopicState
interfaces up to the top level so that they can be shared between the two
policies.

Cheers,

Tom

On 26 September 2017 at 18:09, Edoardo Comar <eco...@uk.ibm.com> wrote:

> Thanks Tom,
> In my original KIP-170 I mentioned that the method
>
> public Map<String, Integer> topicsPartitionCount();
>
> was just a starting point for a general purpose ClusterState
> as it happened to be exactly the info we needed for our policy
> implementation :-)
> it definitely doesn't feel general purpose enough.
>
> what about
>
> interface ClusterState {
> public TopicState topicState(String topicName);
> public Set topics();
> }
>
> I think that the implementation of ClusterState that the server will pass
> to the policy.validate method
> would just lazily tap into MetadataCache. No need for big upfront
> allocations.
>
> ciao,
> Edo
> --
>
> Edoardo Comar
>
> IBM Message Hub
>
> IBM UK Ltd, Hursley Park, SO21 2JN
>
>
>
> From:   Tom Bentley <t.j.bent...@gmail.com>
> To: dev@kafka.apache.org
> Date:   26/09/2017 17:39
> Subject:Re: [DISCUSS] KIP-201: Rationalising Policy interfaces
>
>
>
> Hi Edoardo,
>
>
> what about a single method in ClusterState
> >
> > interface ClusterState {
> > public Map<String,TopicState> topicsState();
> >
> > }
> >
> > which could return a read-only snapshot of the cluster metadata ?
> >
>
> Sure that would work too. A concern with that is that we end up allocating
> a potentially rather large amount for the Map and the collections present
> in the TopicStates in order to provide the snapshot. The caller might only
> be interested in one item from the TopicState for one topic in the map.
> Accessing this information via methods means the caller only pays for what
> they use.
>
> Cheers,
>
> Tom
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

[Paolo Patierno]

Hi Ismael,

regarding motivations for delete records, as I said during the discussion on 
KIP-204, it gives the possibility to avoid deleting messages for specific 
partitions (inside the topic) and starting from a specific offset. I could 
think on some users solutions where they know exactly what the partitions means 
in a specific topic (because they are using a custom partitioner on the 
producer side) so they know what kind of messages are inside a partition 
allowing to delete them but not the others.  In such a policy a user could also 
check the timestamp related to the offset for allowing or not deletion on time 
base.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma 
<ism...@juma.me.uk>
Sent: Wednesday, September 27, 2017 10:18 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

A couple more comments:

1. "If this KIP is accepted for Kafka 1.1.0 this removal could happen in
Kafka 1.2.0 or a later release." -> we only remove code in major releases.
So, if it's deprecated in 1.1.0, it would be removed in 2.0.0.

2. Deleting all messages in a topic is not really the same as deleting a
topic. The latter will cause consumers and producers to error out while the
former will not. It would be good to motivate the need for the delete
records policy more.

Ismael

On Wed, Sep 27, 2017 at 11:12 AM, Ismael Juma <ism...@juma.me.uk> wrote:

> Another quick comment: the KIP states that having multiple interfaces
> imply that the logic must be in 2 places. That is not true because the same
> class can implement multiple interfaces (this aspect was considered when we
> decided to introduce policies incrementally).
>
> The main reason why I think the original approach doesn't work well is
> that there is no direct mapping between an operation and the policy. That
> is, we initially thought we would have create/alter/delete topics, but that
> didn't work out as the alter case is better served by multiple request
> types. Given that, it's a bit awkward to maintain the original approach and
> a policy for topic management seemed easier to understand. On that note,
> would `TopicManagementPolicy` be a better name?
>
> Looking at the updated KIP, I notice that we actually have a
> TopicDeletionPolicy with a separate config. That seems to be a halfway
> house. Not sure about that.
>
> Ismael
>
> On Wed, Sep 27, 2017 at 10:47 AM, Tom Bentley <t.j.bent...@gmail.com>
> wrote:
>
>> I have updated the KIP to add a common policy interface for topic and
>> message deletion. This included pulling ClusterState and TopicState
>> interfaces up to the top level so that they can be shared between the two
>> policies.
>>
>> Cheers,
>>
>> Tom
>>
>> On 26 September 2017 at 18:09, Edoardo Comar <eco...@uk.ibm.com> wrote:
>>
>> > Thanks Tom,
>> > In my original KIP-170 I mentioned that the method
>> >
>> > public Map<String, Integer> topicsPartitionCount();
>> >
>> > was just a starting point for a general purpose ClusterState
>> > as it happened to be exactly the info we needed for our policy
>> > implementation :-)
>> > it definitely doesn't feel general purpose enough.
>> >
>> > what about
>> >
>> > interface ClusterState {
>> > public TopicState topicState(String topicName);
>> > public Set topics();
>> > }
>> >
>> > I think that the implementation of ClusterState that the server will
>> pass
>> > to the policy.validate method
>> > would just lazily tap into MetadataCache. No need for big upfront
>> > allocations.
>> >
>> > ciao,
>> > Edo
>> > --
>> >
>> > Edoardo Comar
>> >
>> > IBM Message Hub
>> >
>> > IBM UK Ltd, Hursley Park, SO21 2JN
>> >
>> >
>> >
>> > From:   Tom Bentley <t.j.bent...@gmail.com>
>> > To: dev@kafka.apache.org
>> > Date:   26/09/2017 17:39
>> > Subject:Re: [DISCUSS] KIP-201: Rationalising Policy interfaces
>> >
>> >
>> >
>> > Hi Edoardo,
>> >
>> >
>> > what about a single method in ClusterState
>> > >
>> > > interfa

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Hi Tom,

as I said in the KIP-201 discussion I'm ok with having a unique 
DeleteTopicPolicy but then it could be useful having more information then just 
the topic name; partitions and offset for messages deletion could be useful for 
a fine grained use cases.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Tom Bentley <t.j.bent...@gmail.com>
Sent: Tuesday, September 26, 2017 4:32 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

Hi Paolo,

I guess a RecordDeletionPolicy should work at the partition level, whereas
the TopicDeletionPolicy should work at the topic level. But then we run
into a similar situation as described in the motivation for KIP-201, where
the administrator might have to write+configure two policies in order to
express their intended rules. For example, it's no good preventing people
from deleting topics if they can delete all the messages in those topics,
or vice versa.

On that reasoning, perhaps there should be a single policy interface
covering topic deletion and message deletion. Alternatively, the topic
deletion API could also invoke the record deletion policy (before the topic
deletion policy I mean). But the former would be more consistent with
what's proposed in KIP-201.

Wdyt? I can add this to KIP-201 if you want.

Cheers,

Tom





On 26 September 2017 at 17:01, Paolo Patierno <ppatie...@live.com> wrote:

> Hi Tom,
>
> I think that we could live with the current authorizer based on delete
> topic (for both, deleting messages and topic as a whole) but then the
> RecordsDeletePolicy could be even more fine grained giving the possibility
> to avoid deleting messages for specific partitions (inside the topic) and
> starting from a specific offset.
>
> I could think on some users solutions where they know exactly what the
> partitions means inside of a specific topic (because they are using a
> custom partitioner on the producer side) so they know what kind of messages
> are inside a partition allowing to delete them but not the other.
>
> In such a policy a user could also check the timestamp related to the
> offset for allowing or not deletion on time base.
>
>
> Wdyt ?
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Tom Bentley <t.j.bent...@gmail.com>
> Sent: Tuesday, September 26, 2017 2:55 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> new Admin Client API
>
> Hi Edoardo and Paolo,
>
>
> On 26 September 2017 at 14:10, Paolo Patierno <ppatie...@live.com> wrote:
>
> > What could be useful use cases for having a RecordsDeletePolicy ? Records
> > can't be deleted for a topic name ? Starting from a specific offset ?
> >
>
> I can imagine some users wanting to prohibit using this API completely.
> Maybe others divide up the topic namespace according to some scheme, and so
> it would be allowed for some topics, but not for others based on the name.
> Both these could be done using authz, but would be much simpler to express
> using a policy.
>
> Since both deleting messages and deleting topics are authorized using
> delete operation on the topic, using policies it would be possible to allow
> deleting messages from a topic, but not deleting the topic itself.
>
>
> On 26 September 2017 at 15:27, Edoardo Comar <eco...@uk.ibm.com> wrote:
>
> >
> > Our KIP-170 did indeed suggest a TopicDeletePolicy - but, as said, for
> our
> > intent an Authorizer implementation will be usable instead,
> >
>
> I guess authorization in the most general sense encompass es both the
> ACL-based authorization inherent in Authorizer and the various
> operation-specific *Policies. But they're not the same. The Policies are
> about deciding on *what* is requested, and the Authorizer is about making a
> decision purely on *who* is making the request. It's quite legitimate to
> want to use both, or just one or the other.
>

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

[Paolo Patierno]

If we want to use the same DeleteTopicPolicy for the KIP-170 and for the delete 
records (in the topic) proposed by KIP-204, I think that we need more 
information other than topic name.

As I mentioned in the KIP-204 discussion, the user could be interested in 
having more information because the delete records takes info like partition 
and specific offset.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Edoardo Comar <eco...@uk.ibm.com>
Sent: Tuesday, September 26, 2017 4:18 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Thanks Tom,

what about a single method in ClusterState

interface ClusterState {
public Map<String,TopicState> topicsState();

}

which could return a read-only snapshot of the cluster metadata ?

--

Edoardo Comar

IBM Message Hub

IBM UK Ltd, Hursley Park, SO21 2JN



From:   Tom Bentley <t.j.bent...@gmail.com>
To: dev@kafka.apache.org
Date:   26/09/2017 16:41
Subject:Re: [DISCUSS] KIP-201: Rationalising Policy interfaces



Hi Edoardo and Ismael,

Thanks for the comments. If we're going to have an new ClusterState
parameter via which we can query the cluster state it would make sense to
obtain the existing state  from the ClusterState than from the
RequestMetadata as in the KIP up to now. So I've updated the KIP along
these lines, but there are still a couple of issues:

1 ClusterState.topicsPartitionCount() isn't really necessary, since the
same information is available by repeated calls to
ClusterState.topicState().
We still need a way to get the topics in the cluster, but maybe the method
would allow including/excluding the internal and marked-for-deletion
topics
from the result.
2. Should the TopicState include whether the topic is an internal topic?
3. Do we really need TopicState at all? It would imply creating a new
instance for each topic queried. We could avoid that by pushing the
methods
into ClusterState and RequestMetadata. Since I don't think we need to
abstract over TopicStates, I'm minded to do this.

Cheers,

Tom



On 26 September 2017 at 14:59, Edoardo Comar <eco...@uk.ibm.com> wrote:

> Furthermore, the deletion case could be implemented by a custom
> Authorizer, as discussed in the KIP-204 thread.
> --
>
> Edoardo Comar
>
> IBM Message Hub
>
> IBM UK Ltd, Hursley Park, SO21 2JN
>
>
>
> From:   Edoardo Comar <eco...@uk.ibm.com>
> To: dev@kafka.apache.org
> Date:   26/09/2017 14:01
> Subject:Re: [DISCUSS] KIP-201: Rationalising Policy interfaces
>
>
>
> Hi Tom,
> yes it makes sense to have a single KIP for enhancing these policies.
>
> As Mickael pointed out, we need that the create/alter topic policy are
> able to assess the current cluster metadata.
> KIP-170 suggested a Provider interface with the minimal set of methods
> that we needed.
>
> However Ismael on 22/06/2017 suggested changes to KIP-170 that i didn't
> manage to incorporate yet there.
> instead of a provider interface Ismael suggested to extend the
> RequestMetadata :
>
> > Thanks for the KIP, Edoardo. A few comments:
> >
> > 1. Have you considered extending RequestMetadata with the additional
> >information you need? We could add Cluster to it, which has topic
> >assignment information, for example. This way, there would be no need
for
>
> a
> V2 interface.
>
> >2. Something else that could be useful is passing an instance of
> `Session`
> >so that one can provide custom behaviour depending on the logged in
user.
> >Would this be useful?
>
> > 3. For the delete case, we may consider passing a class instead of
just
> a
> > string to the validate method so that we have options if we need to
> extend
> >it.
>
> > 4. Do we want to enhance the AlterConfigs policy as well?
> >
> > Ismael
>
>
> His change proposal #1 would be fine with me - although I do not see how
> we could check if isTopicMarkedForDeletion.
> as for change #2 having the KafkaPrincipal instead of the session works
> for me
>
> As for the delete policy - our motivation is entirely different : we
> needed a policy essentially to ensure that the topic was not registered
> with dependent services that we did not want to break.  For a delete
> policy to be usable in a friendly way, the Kafka protocol needs to be
> updated as in KIP-170 to

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Hi Tom,

I think that we could live with the current authorizer based on delete topic 
(for both, deleting messages and topic as a whole) but then the 
RecordsDeletePolicy could be even more fine grained giving the possibility to 
avoid deleting messages for specific partitions (inside the topic) and starting 
from a specific offset.

I could think on some users solutions where they know exactly what the 
partitions means inside of a specific topic (because they are using a custom 
partitioner on the producer side) so they know what kind of messages are inside 
a partition allowing to delete them but not the other.

In such a policy a user could also check the timestamp related to the offset 
for allowing or not deletion on time base.


Wdyt ?


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Tom Bentley <t.j.bent...@gmail.com>
Sent: Tuesday, September 26, 2017 2:55 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

Hi Edoardo and Paolo,


On 26 September 2017 at 14:10, Paolo Patierno <ppatie...@live.com> wrote:

> What could be useful use cases for having a RecordsDeletePolicy ? Records
> can't be deleted for a topic name ? Starting from a specific offset ?
>

I can imagine some users wanting to prohibit using this API completely.
Maybe others divide up the topic namespace according to some scheme, and so
it would be allowed for some topics, but not for others based on the name.
Both these could be done using authz, but would be much simpler to express
using a policy.

Since both deleting messages and deleting topics are authorized using
delete operation on the topic, using policies it would be possible to allow
deleting messages from a topic, but not deleting the topic itself.


On 26 September 2017 at 15:27, Edoardo Comar <eco...@uk.ibm.com> wrote:

>
> Our KIP-170 did indeed suggest a TopicDeletePolicy - but, as said, for our
> intent an Authorizer implementation will be usable instead,
>

I guess authorization in the most general sense encompass es both the
ACL-based authorization inherent in Authorizer and the various
operation-specific *Policies. But they're not the same. The Policies are
about deciding on *what* is requested, and the Authorizer is about making a
decision purely on *who* is making the request. It's quite legitimate to
want to use both, or just one or the other.

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Hi Edoardo,


I was referring to the KIP-107 where the delete records operation is coming 
with the authorizer I mentioned. You are referring to KIP-170 ... same digits, 
inverse order ! Sorry for that ;)


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


____
From: Paolo Patierno <ppatie...@live.com>
Sent: Tuesday, September 26, 2017 2:12 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

Hi Edoardo,

reading the KIP-170 at this point :

3) API Authorization

Given the potential damage that can be caused if this API is used by mistake, 
it is important that we limit its usage to only authorized users. For this 
matter, we can take advantage of the existing authorization framework 
implemented in 
KIP-11<https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface>.
 deleteDataBefore() will have the same authorization setting as deleteTopic(). 
Its operation type is be DELETE and its resource type is TOPIC.


it seems to me that the KIP doesn't suggest a policy as you are saying but 
using the authorizer mechanism with operation = DELETE and resource = TOPIC.

Is my understanding right ?


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Edoardo Comar <eco...@uk.ibm.com>
Sent: Tuesday, September 26, 2017 1:59 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

Hi Paolo,

our the motivation for suggesting a delete policy as described in KIP-170
was to avoid deletion of topics that have a specific usage (essential to
other services).
As you say, it would be entirely replaceable by an Authorizer
implementation that performs the equivalent check.

thanks
Edo
--

Edoardo Comar

IBM Message Hub

IBM UK Ltd, Hursley Park, SO21 2JN



From:   Paolo Patierno <ppatie...@live.com>
To: "dev@kafka.apache.org" <dev@kafka.apache.org>
Date:   26/09/2017 14:10
Subject:Re: [DISCUSS] KIP-204 : adding records deletion operation
to the new Admin Client API



Hi Tom,


the KIP-170 doesn't propose to use a TopicDeletePolicy as policy classes
are meant to be.

It's referring to the authorization interface (KIP-11<
https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D11-2B-2D-2BAuthorization-2BInterface=DwIFAw=jf_iaSHvJObTbx-siA1ZOg=EzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ=7f9IdD9OiWfZojnX4FN8OUimAl_0Q72yxvE8Nz_QdpU=NQw-765HFBIbCYIZLpYIJrvCaZ4-Ml0L1e4z4wkx8fg=
>) with operation = DELETE and resource = TOPIC.

You know that when a request comes in there is the "authorizer" first
which is based on KIP-11 and then the operation is called and then you
could have policy.

I.e. for create topics, first the "authorizer" is called in the KafkaApis
(operation = CREATE, resource = TOPIC) and then the CreateTopicPolicy is
applied in the AdminManager.


In any case what KIP-170 is proposing is not good. Maybe we could have
operation = DELETE and a new resource = MESSAGE/RECORD.

What could be useful use cases for having a RecordsDeletePolicy ? Records
can't be deleted for a topic name ? Starting from a specific offset ?


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<
https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_ppatierno=DwIFAw=jf_iaSHvJObTbx-siA1ZOg=EzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ=7f9IdD9OiWfZojnX4FN8OUimAl_0Q72yxvE8Nz_QdpU=XDNIqoF_Lv9mDGsBvFaRn8bazYxBs3a4lWKrekvRxrw=
>
Linkedin : paolopatierno<
https://urldefense.proofpoint.com/v2/url?u=http-3A__it.linkedin.com_in_paolopatierno=DwIFAw=jf_iaSHvJObTbx-siA1ZOg=EzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ=7f9IdD9OiWfZojnX4FN8OUimAl_0Q72yxvE8Nz_QdpU=6dww9uXw2L0IZ3JKqvoxImEA07L5d7ORCtWi3_wxwcw=
>
Blog : DevExperience<
https://urldefense.proofpoint.com/v2/url?u=http-3A__paolopatierno.wordpress.com_=DwIFAw=jf_iaSHvJObTbx-siA1ZOg=EzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ=7f9IdD9OiWfZojnX4FN8OUimAl_0Q72yxvE8Nz_QdpU=RcYQ4n1Fcl2vHgy2Aj_J9wUO3ZUaFgl8vCZ4slCD5ro=
>



From: Tom Bentley <t.j.bent...@gmail.com>
Sent: Tuesday, September 19, 2017 8:55 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 :

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Hi Edoardo,

reading the KIP-170 at this point :

3) API Authorization

Given the potential damage that can be caused if this API is used by mistake, 
it is important that we limit its usage to only authorized users. For this 
matter, we can take advantage of the existing authorization framework 
implemented in 
KIP-11<https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface>.
 deleteDataBefore() will have the same authorization setting as deleteTopic(). 
Its operation type is be DELETE and its resource type is TOPIC.


it seems to me that the KIP doesn't suggest a policy as you are saying but 
using the authorizer mechanism with operation = DELETE and resource = TOPIC.

Is my understanding right ?


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Edoardo Comar <eco...@uk.ibm.com>
Sent: Tuesday, September 26, 2017 1:59 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

Hi Paolo,

our the motivation for suggesting a delete policy as described in KIP-170
was to avoid deletion of topics that have a specific usage (essential to
other services).
As you say, it would be entirely replaceable by an Authorizer
implementation that performs the equivalent check.

thanks
Edo
--

Edoardo Comar

IBM Message Hub

IBM UK Ltd, Hursley Park, SO21 2JN



From:   Paolo Patierno <ppatie...@live.com>
To: "dev@kafka.apache.org" <dev@kafka.apache.org>
Date:   26/09/2017 14:10
Subject:Re: [DISCUSS] KIP-204 : adding records deletion operation
to the new Admin Client API



Hi Tom,


the KIP-170 doesn't propose to use a TopicDeletePolicy as policy classes
are meant to be.

It's referring to the authorization interface (KIP-11<
https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D11-2B-2D-2BAuthorization-2BInterface=DwIFAw=jf_iaSHvJObTbx-siA1ZOg=EzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ=7f9IdD9OiWfZojnX4FN8OUimAl_0Q72yxvE8Nz_QdpU=NQw-765HFBIbCYIZLpYIJrvCaZ4-Ml0L1e4z4wkx8fg=
>) with operation = DELETE and resource = TOPIC.

You know that when a request comes in there is the "authorizer" first
which is based on KIP-11 and then the operation is called and then you
could have policy.

I.e. for create topics, first the "authorizer" is called in the KafkaApis
(operation = CREATE, resource = TOPIC) and then the CreateTopicPolicy is
applied in the AdminManager.


In any case what KIP-170 is proposing is not good. Maybe we could have
operation = DELETE and a new resource = MESSAGE/RECORD.

What could be useful use cases for having a RecordsDeletePolicy ? Records
can't be deleted for a topic name ? Starting from a specific offset ?


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<
https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_ppatierno=DwIFAw=jf_iaSHvJObTbx-siA1ZOg=EzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ=7f9IdD9OiWfZojnX4FN8OUimAl_0Q72yxvE8Nz_QdpU=XDNIqoF_Lv9mDGsBvFaRn8bazYxBs3a4lWKrekvRxrw=
>
Linkedin : paolopatierno<
https://urldefense.proofpoint.com/v2/url?u=http-3A__it.linkedin.com_in_paolopatierno=DwIFAw=jf_iaSHvJObTbx-siA1ZOg=EzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ=7f9IdD9OiWfZojnX4FN8OUimAl_0Q72yxvE8Nz_QdpU=6dww9uXw2L0IZ3JKqvoxImEA07L5d7ORCtWi3_wxwcw=
>
Blog : DevExperience<
https://urldefense.proofpoint.com/v2/url?u=http-3A__paolopatierno.wordpress.com_=DwIFAw=jf_iaSHvJObTbx-siA1ZOg=EzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ=7f9IdD9OiWfZojnX4FN8OUimAl_0Q72yxvE8Nz_QdpU=RcYQ4n1Fcl2vHgy2Aj_J9wUO3ZUaFgl8vCZ4slCD5ro=
>



From: Tom Bentley <t.j.bent...@gmail.com>
Sent: Tuesday, September 19, 2017 8:55 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
new Admin Client API

Hi Paolo,

Thanks for the KIP.

What errors can be anticipated for the API method (See
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_KAFKA-2D5445-29-3F=DwIFAw=jf_iaSHvJObTbx-siA1ZOg=EzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ=7f9IdD9OiWfZojnX4FN8OUimAl_0Q72yxvE8Nz_QdpU=EaM1W6ZCxDBgKxKpVthbGgK2riVb9EESHu3EfeLMvOQ=


It's not mentioned in KIP-107, but maybe now is a good time to consider
whether there should be some kind of DeleteRecordsPolicy, like there is
for
creating topics? Note: KIP-170 proposes a TopicDeletePolicy, but I don't
think you could use that as currently proposed, because it would be unable
to distinguish the deletion of an entire topic f

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Hi Tom,


the KIP-170 doesn't propose to use a TopicDeletePolicy as policy classes are 
meant to be.

It's referring to the authorization interface 
(KIP-11<https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface>)
 with operation = DELETE and resource = TOPIC.

You know that when a request comes in there is the "authorizer" first which is 
based on KIP-11 and then the operation is called and then you could have policy.

I.e. for create topics, first the "authorizer" is called in the KafkaApis 
(operation = CREATE, resource = TOPIC) and then the CreateTopicPolicy is 
applied in the AdminManager.


In any case what KIP-170 is proposing is not good. Maybe we could have 
operation = DELETE and a new resource = MESSAGE/RECORD.

What could be useful use cases for having a RecordsDeletePolicy ? Records can't 
be deleted for a topic name ? Starting from a specific offset ?


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Tom Bentley <t.j.bent...@gmail.com>
Sent: Tuesday, September 19, 2017 8:55 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the new 
Admin Client API

Hi Paolo,

Thanks for the KIP.

What errors can be anticipated for the API method (See
https://issues.apache.org/jira/browse/KAFKA-5445)?

It's not mentioned in KIP-107, but maybe now is a good time to consider
whether there should be some kind of DeleteRecordsPolicy, like there is for
creating topics? Note: KIP-170 proposes a TopicDeletePolicy, but I don't
think you could use that as currently proposed, because it would be unable
to distinguish the deletion of an entire topic from the deletion of some
records within a topic.

Thanks again,

Tom


On 18 September 2017 at 21:06, Dong Lin <lindon...@gmail.com> wrote:

> +1 (non-binding)
>
> On Mon, Sep 18, 2017 at 1:04 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > +1
> >
> > On Mon, Sep 18, 2017 at 9:19 AM, Paolo Patierno <ppatie...@live.com>
> > wrote:
> >
> > > Hi devs,
> > >
> > >
> > > I'd like to start a discussion around adding the delete records
> > operation,
> > > already available at protocol level and in the "legacy" Admin Client in
> > > Scala, to the "new" Admin Client API in Java.
> > >
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 204+%3A+adding+records+deletion+operation+to+the+new+Admin+Client+API
> > >
> > >
> > > Thanks,
> > >
> > >
> > > Paolo Patierno
> > > Senior Software Engineer (IoT) @ Red Hat
> > > Microsoft MVP on Azure & IoT
> > > Microsoft Azure Advisor
> > >
> > > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> > >
> >
>

The way for the tools and the Admin Client API usage

[Paolo Patierno]

Hi devs and committers,

I'd like to raise a point about Kafka tools and Admin Client API in order to 
have a sort of agreement on the path we should follow for having consistency in 
the project.

We all know that the bigger part of tools is in Scala with only few of them in 
Java. At same time more of them are still using Zookeeper and few of them are 
using Admin Client API (even because it's quite new and evolving release by 
release).


My initial understanding was that the community (at least the committers) would 
like to have tools migrated in Java (something like already happened for 
clients) and using the new Admin Client API.

This was the path I was trying to follow with TopicCommand tool then I noticed 
that ...


A new delete records tool was added (the PR was merged) and it was written in 
Scala, adding the new protocol commands for doing that to the "legacy" Admin 
client (for this I opened the 
KIP-204<https://cwiki.apache.org/confluence/display/KAFKA/KIP-204+%3A+Adding+records+deletion+operation+to+the+new+Admin+Client+API>
 for having it in the new Admin Client API). So, to summarize, new tool ... no 
Java ... no new Admin Client API usage.


What I see is that there is a real "mix" : Scala tools using "legacy" Admin 
Client, Scala tools using new Admin Client API, few Java tools just using 
consumer/producer clients, ...


In the past, during this 
discussion<https://www.mail-archive.com/dev@kafka.apache.org/msg78014.html>, 
the final idea was having the bash script layer using different tools (Scala or 
Java) based on Zookeeper parameter usage.


I'd like to know the opinion about the final objective and the path to follow 
around tools. Maybe the final objective should be having all tools in Java 
using Admin Client API but maybe during this path having a smooth migration 
just using new Admin Client API in the current Scala tools first (removing 
Zookeeper calls) could be better. Maybe for committers, in order to review and 
merge PRs, little ones are better ...


What do you think about this ?


Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

[Paolo Patierno]

Hi devs,

I know that we are already voting for this (+1 bindings from Ismael Juma and 
Guozhang Wang, +1 non binding from Mickael Maison) but I'd like to ask a 
question about the possible cycle release for this change.

We are really closed to the 1.0.0 release which will have the --new-consumer 
deprecation. The current KIP-176 proposes to remove it in the 2.0.0 release but 
I'm starting to think that it could happen even in one year or so while we 
could have more releases in the middle (1.x.y) every 4 months.

Maybe we could have this KIP-176 included in the 1.1.0 release ? Wdyt ?


Thanks.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Mickael Maison <mickael.mai...@gmail.com>
Sent: Thursday, September 7, 2017 9:53 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

+1 (non binding)
Thanks

On Thu, Sep 7, 2017 at 9:09 AM, Paolo Patierno <ppatie...@live.com> wrote:
> KIP updated to clarify it will be removed in the 2.0.0 version.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Vahid S Hashemian <vahidhashem...@us.ibm.com>
> Sent: Wednesday, September 6, 2017 11:45 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools
>
> +1. Thanks for the KIP.
>
> --Vahid
>
>
>
> From:   Guozhang Wang <wangg...@gmail.com>
> To: "dev@kafka.apache.org" <dev@kafka.apache.org>
> Date:   09/06/2017 03:41 PM
> Subject:Re: [VOTE] KIP-176 : Remove deprecated new-consumer option
> for tools
>
>
>
> +1. Thanks.
>
> On Wed, Sep 6, 2017 at 7:57 AM, Ismael Juma <ism...@juma.me.uk> wrote:
>
>> Thanks for the KIP. +1 (binding). Please make it clear in the KIP that
>> removal will happen in 2.0.0.
>>
>> Ismael
>>
>> On Tue, Aug 8, 2017 at 11:53 AM, Paolo Patierno <ppatie...@live.com>
>> wrote:
>>
>> > Hi devs,
>> >
>> >
>> > I didn't see any more comments about this KIP. The JIRAs related to
> the
>> > first step (so making --new-consumer as deprecated with warning
> messages)
>> > are merged.
>> >
>> > I'd like to start a vote for this KIP.
>> >
>> >
>> > Thanks,
>> >
>> >
>> > Paolo Patierno
>> > Senior Software Engineer (IoT) @ Red Hat
>> > Microsoft MVP on Windows Embedded & IoT
>> > Microsoft Azure Advisor
>> >
>> > Twitter : @ppatierno<
> https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_ppatierno=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=JR2f85JOCHQwbXDKcQkgD6ay-ECdCkrh3HaWqtSjF5w=YUFjd7tCwfVz6mZjZ8KbxeH_yQLhzwBXTmm8Wwf_4Wk=
>>
>> > Linkedin : paolopatierno<
> https://urldefense.proofpoint.com/v2/url?u=http-3A__it.linkedin.com_in_paolopatierno=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=JR2f85JOCHQwbXDKcQkgD6ay-ECdCkrh3HaWqtSjF5w=-q3fI0mMvYa2M1PyPrZxDOFWZoyt66zllRYNw00vIIk=
>>
>> > Blog : DevExperience<
> https://urldefense.proofpoint.com/v2/url?u=http-3A__paolopatierno.wordpress.com_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=JR2f85JOCHQwbXDKcQkgD6ay-ECdCkrh3HaWqtSjF5w=M0NFIAt5g9yjQXa4D-DHFs4-UQ3F0KWHfVFPLIaLAyg=
>>
>> >
>>
>
>
>
> --
> -- Guozhang
>
>
>
>

[jira] [Reopened] (KAFKA-5588) ConsoleConsumer : uselss --new-consumer option

[Paolo Patierno (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paolo Patierno reopened KAFKA-5588:
---

> ConsoleConsumer : uselss --new-consumer option
> --
>
> Key: KAFKA-5588
> URL: https://issues.apache.org/jira/browse/KAFKA-5588
> Project: Kafka
>  Issue Type: Bug
>    Reporter: Paolo Patierno
>    Assignee: Paolo Patierno
>Priority: Minor
>
> Hi,
> it seems to me that the --new-consumer option on the ConsoleConsumer is 
> useless.
> The useOldConsumer var is related to specify --zookeeper on the command line 
> but then the bootstrap-server option (or the --new-consumer) can't be 
> used.
> If you use --bootstrap-server option then the new consumer is used 
> automatically so no need for --new-consumer.
> It turns out the using the old or new consumer is just related on using 
> --zookeeper or --bootstrap-server option (which can't be used together, so I 
> can't use new consumer connecting to zookeeper).
> It's also clear when you use --zookeeper for the old consumer and the output 
> from help says :
> "Consider using the new consumer by passing [bootstrap-server] instead of 
> [zookeeper]"
> I'm going to remove the --new-consumer option from the tool.
> Thanks,
> Paolo.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

[Paolo Patierno]

Hi devs,


I'd like to start a discussion around adding the delete records operation, 
already available at protocol level and in the "legacy" Admin Client in Scala, 
to the "new" Admin Client API in Java.


https://cwiki.apache.org/confluence/display/KAFKA/KIP-204+%3A+adding+records+deletion+operation+to+the+new+Admin+Client+API


Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

[jira] [Created] (KAFKA-5925) Adding records deletion operation to the new Admin Client API

[Paolo Patierno (JIRA)]

Paolo Patierno created KAFKA-5925:
-

 Summary: Adding records deletion operation to the new Admin Client 
API
 Key: KAFKA-5925
 URL: https://issues.apache.org/jira/browse/KAFKA-5925
 Project: Kafka
  Issue Type: Improvement
  Components: admin
Reporter: Paolo Patierno
Assignee: Paolo Patierno
Priority: Minor


Hi,
The 
[KIP-107|https://cwiki.apache.org/confluence/display/KAFKA/KIP-107%3A+Add+purgeDataBefore%28%29+API+in+AdminClient]
 provides a way to delete messages starting from a specified offset inside a 
topic partition which we don’t want to take anymore so without relying on 
time-based and size-based log retention policies. The already implemented 
protocol request and response messages (DeleteRecords API, key 21) are used 
only by the “legacy” Admin Client in Scala and aren’t provided by the new Admin 
Client API in Java.

The 
[KIP-204|https://cwiki.apache.org/confluence/display/KAFKA/KIP-204+%3A+adding+records+deletion+operation+to+the+new+Admin+Client+API]
 is about addressing this JIRA.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5919) Delete records command "version" parameter ignored

[Paolo Patierno (JIRA)]

Paolo Patierno created KAFKA-5919:
-

 Summary: Delete records command "version" parameter ignored
 Key: KAFKA-5919
 URL: https://issues.apache.org/jira/browse/KAFKA-5919
 Project: Kafka
  Issue Type: Bug
  Components: tools
Reporter: Paolo Patierno
Assignee: Paolo Patierno
Priority: Minor


Hi,
the kafka-delete-records script allows user to pass information about records 
to delete through a JSON file. Such file, as described in the command help, is 
made by a "partitions" array and a "version" field. Reading 
[KIP-107|https://cwiki.apache.org/confluence/display/KAFKA/KIP-107%3A+Add+purgeDataBefore%28%29+API+in+AdminClient]
 and the DeleteRecords API (Key: 21) description it's not clear what such field 
is and even it's not used at all (in the current implementation).
I'm going to remove it from tool help description and it should not need a KIP 
because today it's just ignored and even using a JSON file without "version" 
the tool just works.
[~lindong] you implemented such delete command, are my considerations right ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Admin Client : no way to create topic with default partitions and replication factor

[Paolo Patierno]

Hi devs,


taking a look at the Admin Client API and the related implementation it seems 
that :


  *   the CreateTopics request allows you to have "num_partitions" and 
"replication_factor" with -1 as value (it means unset) which is used when you 
specify the "replica_assignment" instead
  *   the NewTopic available constructor doesn't allow you to use such a 
feature. Even trying to pass null as "replicaAssignments" throws an exception 
because a collection is built there.

I think that it could be useful from an administrative point of view having 
this possibility as it already happens when you have auto creation enabled and 
a topic is created with default partitions and replicas when a 
consumer/producer asks for metadata (and the topic doesn't exist).


Of course this proposal needs a KIP because we are changing the Admin Client 
API (but not breaking). Other than a change in the admin client side it will 
need a change in the broker as well because the current path (when a create 
topic request comes) doesn't handle the -1 values for "num_partitions" and 
"replication_factor", so it needs to set default values in this case.


What do you think about that ?


Thanks.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

Re: [DISCUSS] KIP-195: AdminClient.increasePartitions

[Paolo Patierno]

My 2 cents about naming ...


PartitionCount or NumPartitions sound better to me providing an "absolute" 
value (as today the kafka-topics script work) for an idempotent operation while 
the NumPartitionsIncrease name sounds to me like the "increment" value.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Tom Bentley <t.j.bent...@gmail.com>
Sent: Friday, September 8, 2017 9:39 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-195: AdminClient.increasePartitions

Hi Ismael,

Thanks for the comments.

My bad for not noticing the custom assignment requirement. The current
> proposal has the following example (I updated it a little so that 2
> partitions are added):
>
> increasePartitionCount(4, asList(asList(1, 2), asList(2, 3))
>
> Why not simply provide the assignment? For example, if you want to add 2
> partitions, you'd simply do:
>
> increasePartitionCount(asList(asList(1, 2), asList(2, 3))
>
> Not sure why need the number.


kafka-topics.sh allows to increase the number of partitions without
supplying an assignment, so one reason is simply to be able to support that.

When you don't supply an assignment you're leaving it to the cluster to
decide for you. By requiring an assignment you're forcing the user to
decide. The user might not care much and thus make a worse choice than if
you'd left it to the server.


> The other two questions:
>
> 2. Do we want to allow people to add non-consecutive partition ids? This is
> possible to do with the current AdminUtils, but it's not exactly supported
> (although it apparently works fine in the broker). Still, I wanted to make
> sure we considered it.
>

I admit I had assumed this wasn't possible. How does partitioning work if
there are holes? You would need the list of partition ids in order to
produce a correct partition id.

Suspending my scepticism for a moment, to support something like that we'd
have to change the List<List> assignment to a Map<Integer,
List>, so the request explicitly identified the new topics, rather
than it being implied. That would make it slightly less easy to form a
valid request for the normal case of consecutive partition ids: You'd have
to actually know the current number of partitions, which might necessitate
a describeTopics().

It doesn't sound like there are any known use cases for non-consecutive
partition ids. It also sounds like whatever existing support there is might
be only lightly tested. It sounds like a source of gotchas and subtle bugs
to me (both in Kafka itself and for users). I have to wonder whether it
would be worth supporting this.

If we decide not to support it, we should fix the rest of the AdminClient
so it's not possible to create non-consecutive partition ids.

WDYT?


3. Do we want to provide the target partition count or the number we want
> to increase it by? This is related to the first point as well. Thinking
> about it, one benefit of specifying the target number is that the operation
> is then idempotent. If we state the number we want to increase by, it's
> easier to make a mistake and increase it twice under failure scenarios. Was
> that the motivation for specifying the target partition count?
>
>
Right, if you're just supplying an increment you can't be certain what
you're incrementing it to (which is what you actually care about). And
idempotency is so nice to have if something goes wrong.

Using an increment would make the `NumPartitionIncrease` class a bit more
easily understood, as then the outer list would have to be the same size as
the increment. But for me idempotency is the more valuable feature.

Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new AdminClient

[Paolo Patierno]

So as commented on the KAFKA-5723 JIRA my plan could be :


  *   Using the KAFKA-3268 as umbrella for the others JIRAs related to tools 
refactoring
  *   Regarding common components needed by developers involved on different 
tools, I will create a subtask in this JIRA (i.e. related to the new 
CommandOptions class)
  *   For such subtask I'll open a PR with the work I have already done so a 
simple PR
  *   The committers and the other "tools" developers will review the PR for 
making changes
  *   When the PR will be merged the developers can continue on the different 
JIRAs related to different tools making their PRs


Does this plan make sense ?


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma 
<ism...@juma.me.uk>
Sent: Thursday, September 7, 2017 10:56 AM
To: dev@kafka.apache.org
Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new 
AdminClient

No, I'm suggesting that we think how can merge smaller PRs to trunk. Having
a separate branch doesn't help as it can diverge from trunk and a committer
would be needed to merge rebases, etc.

Ismael

On Thu, Sep 7, 2017 at 11:25 AM, Paolo Patierno <ppatie...@live.com> wrote:

> Hi Ismael,
>
> first of all thanks for your reply.
>
> So as far as I understood having a branch in the Kafka repo could be
> better for you as committer to validate small PRs from us and not a big one
> at the end, right ?
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma <
> ism...@juma.me.uk>
> Sent: Thursday, September 7, 2017 10:19 AM
> To: dev@kafka.apache.org
> Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> AdminClient
>
> I would also add that it would be easier to review if there were smaller
> PRs than one big PR. So, it may be worth thinking how progress could be
> made more incrementally.
>
> Ismael
>
> On Thu, Sep 7, 2017 at 11:17 AM, Tom Bentley <t.j.bent...@gmail.com>
> wrote:
>
> > I can't speak for the committers, but there's nothing to stop you
> > submitting PRs against each others branches. It just needs you to agree
> > which of you will host the integration branch. This would be pretty much
> > exactly the same developer experience as of the branch was in the main
> > Kafak repo AFAICS, except the committers wouldn't have to be involved
> with
> > merging your PRs into your integration branch (which is probably a
> benefit
> > to both you and them).
> >
> > On 7 September 2017 at 10:57, Paolo Patierno <ppatie...@live.com> wrote:
> >
> > > Hi committers,
> > >
> > >
> > > as already asked by Viktor on the JIRA yesterday can you give us a
> > > feedback/advice on how to move on that ? Thanks !
> > >
> > >
> > > Paolo Patierno
> > > Senior Software Engineer (IoT) @ Red Hat
> > > Microsoft MVP on Windows Embedded & IoT
> > > Microsoft Azure Advisor
> > >
> > > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> > >
> > >
> > > 
> > > From: Paolo Patierno <ppatie...@live.com>
> > > Sent: Thursday, August 17, 2017 3:18 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the
> new
> > > AdminClient
> > >
> > > As I started on working to develop the TopicCommand tool in Java (with
> > > this PR<https://github.com/apache/kafka/pull/3514>), Andrey is working
> > on
> > > another one and he started to use some of my classes (mostly related to
> > > handling command line arguments) just with copy/paste/modify so
> > duplicating
> > > the code.
> > >
> > > As mentioned in the JIRA discussion<https://i

Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new AdminClient

[Paolo Patierno]

Hi Ismael,

first of all thanks for your reply.

So as far as I understood having a branch in the Kafka repo could be better for 
you as committer to validate small PRs from us and not a big one at the end, 
right ?


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma 
<ism...@juma.me.uk>
Sent: Thursday, September 7, 2017 10:19 AM
To: dev@kafka.apache.org
Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new 
AdminClient

I would also add that it would be easier to review if there were smaller
PRs than one big PR. So, it may be worth thinking how progress could be
made more incrementally.

Ismael

On Thu, Sep 7, 2017 at 11:17 AM, Tom Bentley <t.j.bent...@gmail.com> wrote:

> I can't speak for the committers, but there's nothing to stop you
> submitting PRs against each others branches. It just needs you to agree
> which of you will host the integration branch. This would be pretty much
> exactly the same developer experience as of the branch was in the main
> Kafak repo AFAICS, except the committers wouldn't have to be involved with
> merging your PRs into your integration branch (which is probably a benefit
> to both you and them).
>
> On 7 September 2017 at 10:57, Paolo Patierno <ppatie...@live.com> wrote:
>
> > Hi committers,
> >
> >
> > as already asked by Viktor on the JIRA yesterday can you give us a
> > feedback/advice on how to move on that ? Thanks !
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
> >
> > 
> > From: Paolo Patierno <ppatie...@live.com>
> > Sent: Thursday, August 17, 2017 3:18 PM
> > To: dev@kafka.apache.org
> > Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> > AdminClient
> >
> > As I started on working to develop the TopicCommand tool in Java (with
> > this PR<https://github.com/apache/kafka/pull/3514>), Andrey is working
> on
> > another one and he started to use some of my classes (mostly related to
> > handling command line arguments) just with copy/paste/modify so
> duplicating
> > the code.
> >
> > As mentioned in the JIRA discussion<https://issues.
> > apache.org/jira/browse/KAFKA-5723>, having a branch in the original repo
> > could help us to have personal forks with such branch, opening PRs
> against
> > this branch and having the common code there (syncing the code with
> forked
> > repos).
> >
> > In conclusion the problem is : having more people working on different
> > pieces of code which have some common code (still under development) not
> > yet available in the original repo.
> >
> > At least it's the idea we had but maybe the committers have a different
> > way to go in such situations. Any feedback is really appreciated :-)
> >
> >
> > Thanks,
> >
> > Paolo
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
> >
> > 
> > From: Andrey Dyachkov <andrey.dyach...@gmail.com>
> > Sent: Thursday, August 17, 2017 3:10 PM
> > To: dev@kafka.apache.org
> > Subject: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> > AdminClient
> >
> > Good day,
> > I am quite new here, my apologies for being kind of pushy.
> > We had discussion here https://issues.apache.org/jira/browse/KAFKA-5723,
> > that having separate branch for implementing the whole change
> > https://issues.apache.org/jira/browse/KAFKA-3268 would simplify dev
> > process. Could some of committers give a hand here?
> >
> > Also could you review pr https://github.com/apache/kafka/pull/3671 and
> add
> > me to contributors list in order to add myself to ticket assignee?
> >
> > Thank you!
> > --
> >
> > With great enthusiasm,
> > Andrey
> >
>

Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new AdminClient

[Paolo Patierno]

Yes Tom that's true, we could chose one of us as the integration branch owner 
without involving committers but at same time it could be useful to involve 
committers for having the PRs evaluated step by step and not only at the end. 
For sure when committers are really busy for merging it's not good for us 
because we can't go forward with development.

Let's see what they say, maybe something like that already happened in the past 
...


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Tom Bentley <t.j.bent...@gmail.com>
Sent: Thursday, September 7, 2017 10:17 AM
To: dev@kafka.apache.org
Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new 
AdminClient

I can't speak for the committers, but there's nothing to stop you
submitting PRs against each others branches. It just needs you to agree
which of you will host the integration branch. This would be pretty much
exactly the same developer experience as of the branch was in the main
Kafak repo AFAICS, except the committers wouldn't have to be involved with
merging your PRs into your integration branch (which is probably a benefit
to both you and them).

On 7 September 2017 at 10:57, Paolo Patierno <ppatie...@live.com> wrote:

> Hi committers,
>
>
> as already asked by Viktor on the JIRA yesterday can you give us a
> feedback/advice on how to move on that ? Thanks !
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Paolo Patierno <ppatie...@live.com>
> Sent: Thursday, August 17, 2017 3:18 PM
> To: dev@kafka.apache.org
> Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> AdminClient
>
> As I started on working to develop the TopicCommand tool in Java (with
> this PR<https://github.com/apache/kafka/pull/3514>), Andrey is working on
> another one and he started to use some of my classes (mostly related to
> handling command line arguments) just with copy/paste/modify so duplicating
> the code.
>
> As mentioned in the JIRA discussion<https://issues.
> apache.org/jira/browse/KAFKA-5723>, having a branch in the original repo
> could help us to have personal forks with such branch, opening PRs against
> this branch and having the common code there (syncing the code with forked
> repos).
>
> In conclusion the problem is : having more people working on different
> pieces of code which have some common code (still under development) not
> yet available in the original repo.
>
> At least it's the idea we had but maybe the committers have a different
> way to go in such situations. Any feedback is really appreciated :-)
>
>
> Thanks,
>
> Paolo
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Andrey Dyachkov <andrey.dyach...@gmail.com>
> Sent: Thursday, August 17, 2017 3:10 PM
> To: dev@kafka.apache.org
> Subject: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new
> AdminClient
>
> Good day,
> I am quite new here, my apologies for being kind of pushy.
> We had discussion here https://issues.apache.org/jira/browse/KAFKA-5723,
> that having separate branch for implementing the whole change
> https://issues.apache.org/jira/browse/KAFKA-3268 would simplify dev
> process. Could some of committers give a hand here?
>
> Also could you review pr https://github.com/apache/kafka/pull/3671 and add
> me to contributors list in order to add myself to ticket assignee?
>
> Thank you!
> --
>
> With great enthusiasm,
> Andrey
>

Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new AdminClient

[Paolo Patierno]

Hi committers,


as already asked by Viktor on the JIRA yesterday can you give us a 
feedback/advice on how to move on that ? Thanks !


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


____
From: Paolo Patierno <ppatie...@live.com>
Sent: Thursday, August 17, 2017 3:18 PM
To: dev@kafka.apache.org
Subject: Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new 
AdminClient

As I started on working to develop the TopicCommand tool in Java (with this 
PR<https://github.com/apache/kafka/pull/3514>), Andrey is working on another 
one and he started to use some of my classes (mostly related to handling 
command line arguments) just with copy/paste/modify so duplicating the code.

As mentioned in the JIRA 
discussion<https://issues.apache.org/jira/browse/KAFKA-5723>, having a branch 
in the original repo could help us to have personal forks with such branch, 
opening PRs against this branch and having the common code there (syncing the 
code with forked repos).

In conclusion the problem is : having more people working on different pieces 
of code which have some common code (still under development) not yet available 
in the original repo.

At least it's the idea we had but maybe the committers have a different way to 
go in such situations. Any feedback is really appreciated :-)


Thanks,

Paolo


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Andrey Dyachkov <andrey.dyach...@gmail.com>
Sent: Thursday, August 17, 2017 3:10 PM
To: dev@kafka.apache.org
Subject: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new 
AdminClient

Good day,
I am quite new here, my apologies for being kind of pushy.
We had discussion here https://issues.apache.org/jira/browse/KAFKA-5723,
that having separate branch for implementing the whole change
https://issues.apache.org/jira/browse/KAFKA-3268 would simplify dev
process. Could some of committers give a hand here?

Also could you review pr https://github.com/apache/kafka/pull/3671 and add
me to contributors list in order to add myself to ticket assignee?

Thank you!
--

With great enthusiasm,
Andrey

Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

[Paolo Patierno]

KIP updated to clarify it will be removed in the 2.0.0 version.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Vahid S Hashemian <vahidhashem...@us.ibm.com>
Sent: Wednesday, September 6, 2017 11:45 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

+1. Thanks for the KIP.

--Vahid



From:   Guozhang Wang <wangg...@gmail.com>
To: "dev@kafka.apache.org" <dev@kafka.apache.org>
Date:   09/06/2017 03:41 PM
Subject:Re: [VOTE] KIP-176 : Remove deprecated new-consumer option
for tools



+1. Thanks.

On Wed, Sep 6, 2017 at 7:57 AM, Ismael Juma <ism...@juma.me.uk> wrote:

> Thanks for the KIP. +1 (binding). Please make it clear in the KIP that
> removal will happen in 2.0.0.
>
> Ismael
>
> On Tue, Aug 8, 2017 at 11:53 AM, Paolo Patierno <ppatie...@live.com>
> wrote:
>
> > Hi devs,
> >
> >
> > I didn't see any more comments about this KIP. The JIRAs related to
the
> > first step (so making --new-consumer as deprecated with warning
messages)
> > are merged.
> >
> > I'd like to start a vote for this KIP.
> >
> >
> > Thanks,
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<
https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_ppatierno=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=JR2f85JOCHQwbXDKcQkgD6ay-ECdCkrh3HaWqtSjF5w=YUFjd7tCwfVz6mZjZ8KbxeH_yQLhzwBXTmm8Wwf_4Wk=
>
> > Linkedin : paolopatierno<
https://urldefense.proofpoint.com/v2/url?u=http-3A__it.linkedin.com_in_paolopatierno=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=JR2f85JOCHQwbXDKcQkgD6ay-ECdCkrh3HaWqtSjF5w=-q3fI0mMvYa2M1PyPrZxDOFWZoyt66zllRYNw00vIIk=
>
> > Blog : DevExperience<
https://urldefense.proofpoint.com/v2/url?u=http-3A__paolopatierno.wordpress.com_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=JR2f85JOCHQwbXDKcQkgD6ay-ECdCkrh3HaWqtSjF5w=M0NFIAt5g9yjQXa4D-DHFs4-UQ3F0KWHfVFPLIaLAyg=
>
> >
>



--
-- Guozhang

Fw: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

[Paolo Patierno]

Hi devs,


I haven't seen any votes for this since last month.

Is there something that should be addressed in the KIP (it didn't have any 
comments anymore and for this reason I started the vote).


Thanks.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


____
From: Paolo Patierno <ppatie...@live.com>
Sent: Tuesday, August 8, 2017 10:53 AM
To: dev@kafka.apache.org
Subject: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools


Hi devs,


I didn't see any more comments about this KIP. The JIRAs related to the first 
step (so making --new-consumer as deprecated with warning messages) are merged.

I'd like to start a vote for this KIP.


Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new AdminClient

[Paolo Patierno]

As I started on working to develop the TopicCommand tool in Java (with this 
PR<https://github.com/apache/kafka/pull/3514>), Andrey is working on another 
one and he started to use some of my classes (mostly related to handling 
command line arguments) just with copy/paste/modify so duplicating the code.

As mentioned in the JIRA 
discussion<https://issues.apache.org/jira/browse/KAFKA-5723>, having a branch 
in the original repo could help us to have personal forks with such branch, 
opening PRs against this branch and having the common code there (syncing the 
code with forked repos).

In conclusion the problem is : having more people working on different pieces 
of code which have some common code (still under development) not yet available 
in the original repo.

At least it's the idea we had but maybe the committers have a different way to 
go in such situations. Any feedback is really appreciated :-)


Thanks,

Paolo


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Andrey Dyachkov <andrey.dyach...@gmail.com>
Sent: Thursday, August 17, 2017 3:10 PM
To: dev@kafka.apache.org
Subject: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new 
AdminClient

Good day,
I am quite new here, my apologies for being kind of pushy.
We had discussion here https://issues.apache.org/jira/browse/KAFKA-5723,
that having separate branch for implementing the whole change
https://issues.apache.org/jira/browse/KAFKA-3268 would simplify dev
process. Could some of committers give a hand here?

Also could you review pr https://github.com/apache/kafka/pull/3671 and add
me to contributors list in order to add myself to ticket assignee?

Thank you!
--

With great enthusiasm,
Andrey

Re: Command tools : from Scala to Java, from Zookeeper utils to Admin Client API

[Paolo Patierno]

Hi Ismael,


after your first review on my PR and the conversation here in the mailing list 
... I have a doubt ...

We are going to remove the --zookeeper option but adding the --broker-list but 
according to this discussion we agree on your solution having the choice at 
shell script level.

Does the Java re-implementation of the TopicCommand tool need a KIP ?


Thanks,

Paolo


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma 
<ism...@juma.me.uk>
Sent: Wednesday, August 2, 2017 8:51 AM
To: dev@kafka.apache.org
Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to Admin 
Client API

Hi Arseniy,

At some point (before I joined), it was decided that clients would be
written in Java for a few reasons:

1. It's easier to maintain binary compatibility
2. Calling Java from Scala is easier than calling Scala from Java (without
extra effort)
3. A single artifact instead of one artifact per Scala version supported
4. There are a larger number of Java users than Scala users

Since the plan is for most tools to be a thin wrapper over the Java
clients, it seems sensible for such tools not to bring unnecessary
dependencies.

Ismael


On Wed, Aug 2, 2017 at 9:28 AM, Arseniy Tashoyan <tasho...@gmail.com> wrote:

> Hi Paolo,
>
> I doubt that rewriting in Java makes value by itself. What is the reason of
> redoing the things that are already done? The switching to new API can be
> done without the switching to another language. Less new code - less new
> bugs.
> In some cases, Scala may be more convenient than Java. For example, one can
> find Scala collections more handy than Java collections. Just select a
> language that gives necessary tools.
>
> Arseniy
>
> 2017-08-01 17:30 GMT+03:00 Paolo Patierno <ppatie...@live.com>:
>
> > Hi Tom,
> >
> >
> > thanks for your reply. I think that you are right and what you have
> > proposed should be the way to go.
> >
> >
> > Until today I have been working on refactoring the TopicCommand tool in
> > Java using the AdminClient getting rid of the Zookeeper usage in only
> "one
> > step" and maybe it's wrong.
> >
> > I'd like to have some input from committers as well to be sure that the
> > way is good about how handling such use cases.
> >
> >
> > Thanks,
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
> >
> > ____
> > From: Tom Bentley <t.j.bent...@gmail.com>
> > Sent: Friday, July 28, 2017 10:36 AM
> > To: dev@kafka.apache.org
> > Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to
> > Admin Client API
> >
> > Hi Paolo,
> >
> > Replies in line...
> >
> > On 28 July 2017 at 11:14, Paolo Patierno <ppatie...@live.com> wrote:
> >
> > > Hi committers,
> > >
> > > in my understanding there is the common idea to move all tools from
> Scala
> > > to Java and then using the new Admin Client API instead of using the
> > > Zookeeper connection.
> > >
> > > Regarding this subject I started to work on refactoring the
> TopicCommand
> > > tool but with two steps at same time :
> > >
> > >
> > >   *   re-writing it in Java
> > >   *   removing --zookeeper option (so no Zookeeper connection) and
> adding
> > > --broker-list option (so using the Admin Client API)
> > >
> > > Of course, such option substitution is a breaking change for the tool
> > (and
> > > the users who are using it).
> > > In such a scenario, the two steps path should be needed : first
> > > deprecation, then removal (for the --zookeeper option)
> > >
> >
> > A change to tools (and their options) requires a KIP. There's no
> > fundamental reason why both couldn't be supported during a transition
> > period. So I doubt a KIP that didn't propose a transition period would
> get
> > passed.
> >
> >
> > It seems that in t

[jira] [Created] (KAFKA-5739) Rewrite KStreamPeekTest at processor level avoiding driver usage

[Paolo Patierno (JIRA)]

Paolo Patierno created KAFKA-5739:
-

 Summary: Rewrite KStreamPeekTest at processor level avoiding 
driver usage
 Key: KAFKA-5739
 URL: https://issues.apache.org/jira/browse/KAFKA-5739
 Project: Kafka
  Issue Type: Test
  Components: streams
Reporter: Paolo Patierno
Assignee: Paolo Patierno
Priority: Minor


Hi,
as already done for the {{KStreamPrintTest}} we could remove the usage of 
{{KStreamTestDriver}} even in the {{KStreamPeekTest}} and testing it at 
processor level not at stream level.
My proposal is to :

* create the {{KStreamPeek}} instance providing the action which fill a 
collection as already happens today
* testing for both {{forwardDownStream}} values true and false
* using the {{MockProcessorContext}} class for overriding the {{forward}} 
method filling a streamObserved collection as happens today 
{{forwardDownStream}} is true; checking that the {{forward}} isn't called when 
{{forwardDownStream}} is false (so the test fails)

Thanks,
Paolo 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5684) KStreamPrintProcessor as customized KStreamPeekProcessor

[Paolo Patierno (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paolo Patierno resolved KAFKA-5684.
---
Resolution: Feedback Received

> KStreamPrintProcessor as customized KStreamPeekProcessor
> 
>
> Key: KAFKA-5684
> URL: https://issues.apache.org/jira/browse/KAFKA-5684
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>    Reporter: Paolo Patierno
>    Assignee: Paolo Patierno
>Priority: Minor
>
> Hi,
> the {{KStreamPrintProcessor}} is implemented from scratch (from the 
> {{AbstractProcessor}}) and the same for the related supplier.
> It looks to me that it's just a special {{KStreamPeekProcessor}} with 
> forwardDownStream to false and that allows the possibility to specify Serdes 
> instances used if key/values are bytes.
> At same time used by a {{print()}} method it provides a fast way to print 
> data flowing through the pipeline (while using just {{peek()}} you need to 
> write the code).
> I think that it could be useful to refactoring the {{KStreamPrintProcessor}} 
> as derived from the {{KStreamPeekProcessor}} customizing its behavior.
> Thanks,
> Paolo.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

[Paolo Patierno]

Hi Tom,

good question.


Due to the new version policies, the deprecation will be in the coming 1.0.0 
release but then, due to a breaking change on the API, the removal should be on 
2.0.0 I guess.


Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Tom Bentley <t.j.bent...@gmail.com>
Sent: Tuesday, August 8, 2017 11:00 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

The KIP is here for any one, like me, who hasn't seen it yet:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-176:+Remove+deprecated+new-consumer+option+for+tools

Paolo, the KIP says "On the next release cycle we could totally remove the
option." Exactly which release are you proposing that be?

Thanks,

Tom

On 8 August 2017 at 11:53, Paolo Patierno <ppatie...@live.com> wrote:

> Hi devs,
>
>
> I didn't see any more comments about this KIP. The JIRAs related to the
> first step (so making --new-consumer as deprecated with warning messages)
> are merged.
>
> I'd like to start a vote for this KIP.
>
>
> Thanks,
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>

[VOTE] KIP-176 : Remove deprecated new-consumer option for tools

[Paolo Patierno]

Hi devs,


I didn't see any more comments about this KIP. The JIRAs related to the first 
step (so making --new-consumer as deprecated with warning messages) are merged.

I'd like to start a vote for this KIP.


Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

Jenkins problem with JDK 8 and Scala 2.12 ?

[Paolo Patierno]

Hi devs,

is there any problems with Jenkins for building with JDK 8 and Scala 2.12.

I see the following error in this console log 
(https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6618/consoleFull)


ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
git://github.com/apache/kafka.git


I re-tried twice with same results.

All is good with JDK 7 and Scala 2.11


Thanks,

Paolo


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

Re: KStreamPrintTest : no differences in the unit tests

[Paolo Patierno]

Hi Damian,


I submit the patch as part of this PR


 https://github.com/apache/kafka/pull/3611


adding a couple of other tests as well.

Maybe you can take a look at it :-)


Thanks !


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Damian Guy <damian@gmail.com>
Sent: Wednesday, August 2, 2017 5:29 PM
To: dev@kafka.apache.org
Subject: Re: KStreamPrintTest : no differences in the unit tests

Yes - they are basically the same. Feel free to submit a patch to remove
one of them

On Wed, 2 Aug 2017 at 15:28 Paolo Patierno <ppatie...@live.com> wrote:

> Hi devs,
>
> taking a look at KStreamPrintTest I can't find any substantial difference
> between the two tests :
>
>
> testPrintStreamWithProvidedKeyValueMapper
>
> testPrintKeyValueWithName
>
>
> The only tiny difference is the mapper output "%d, %s" instead of "(%d,
> %s)" but then all the code seems to be the same (some other "final" missing
> ...).
>
> Is there something that my eyes can't see ?
>
>
> Thanks,
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>

KStreamPrintTest : no differences in the unit tests

[Paolo Patierno]

Hi devs,

taking a look at KStreamPrintTest I can't find any substantial difference 
between the two tests :


testPrintStreamWithProvidedKeyValueMapper

testPrintKeyValueWithName


The only tiny difference is the mapper output "%d, %s" instead of "(%d, %s)" 
but then all the code seems to be the same (some other "final" missing ...).

Is there something that my eyes can't see ?


Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

Re: Command tools : from Scala to Java, from Zookeeper utils to Admin Client API

[Paolo Patierno]

Hi Arseniy,


I opened this JIRA about one month ago 
(https://issues.apache.org/jira/browse/KAFKA-5536) asking why we have some 
tools in Java and other in Scala.

As Ismael pointed out the end state should be having tools written in Java (as 
already happened to clients from Scala to Java) living in the external "tools" 
module (where there are already some tools) so outside of the "core" module in 
Kafka. Another key point is to use the Admin Client API provided by the 
"clients" module in all the tools where we don't need Zookeeper connection 
anymore.

My guess is that, even if Kafka is developed in Scala, the committers have the 
idea to have all the external modules written in Java (so clients and tools for 
now).

As you know it's the same as for the other two bigger projects around Kafka 
(core) like Kafka Connect and Kafka Streams ... entirely developed in Java.


Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Arseniy Tashoyan <tasho...@gmail.com>
Sent: Wednesday, August 2, 2017 8:28 AM
To: dev@kafka.apache.org
Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to Admin 
Client API

Hi Paolo,

I doubt that rewriting in Java makes value by itself. What is the reason of
redoing the things that are already done? The switching to new API can be
done without the switching to another language. Less new code - less new
bugs.
In some cases, Scala may be more convenient than Java. For example, one can
find Scala collections more handy than Java collections. Just select a
language that gives necessary tools.

Arseniy

2017-08-01 17:30 GMT+03:00 Paolo Patierno <ppatie...@live.com>:

> Hi Tom,
>
>
> thanks for your reply. I think that you are right and what you have
> proposed should be the way to go.
>
>
> Until today I have been working on refactoring the TopicCommand tool in
> Java using the AdminClient getting rid of the Zookeeper usage in only "one
> step" and maybe it's wrong.
>
> I'd like to have some input from committers as well to be sure that the
> way is good about how handling such use cases.
>
>
> Thanks,
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Tom Bentley <t.j.bent...@gmail.com>
> Sent: Friday, July 28, 2017 10:36 AM
> To: dev@kafka.apache.org
> Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to
> Admin Client API
>
> Hi Paolo,
>
> Replies in line...
>
> On 28 July 2017 at 11:14, Paolo Patierno <ppatie...@live.com> wrote:
>
> > Hi committers,
> >
> > in my understanding there is the common idea to move all tools from Scala
> > to Java and then using the new Admin Client API instead of using the
> > Zookeeper connection.
> >
> > Regarding this subject I started to work on refactoring the TopicCommand
> > tool but with two steps at same time :
> >
> >
> >   *   re-writing it in Java
> >   *   removing --zookeeper option (so no Zookeeper connection) and adding
> > --broker-list option (so using the Admin Client API)
> >
> > Of course, such option substitution is a breaking change for the tool
> (and
> > the users who are using it).
> > In such a scenario, the two steps path should be needed : first
> > deprecation, then removal (for the --zookeeper option)
> >
>
> A change to tools (and their options) requires a KIP. There's no
> fundamental reason why both couldn't be supported during a transition
> period. So I doubt a KIP that didn't propose a transition period would get
> passed.
>
>
> It seems that in the "deprecation" phase we have two possible solutions :
> >
> >
> >   1.  Adding Admin Client API to the Scala tools and so the --broker-list
> > option and a warning on --zookeeper for deprecation
> >   2.  Rewriting the tool in Java using the Admin Client API (so
> > --broker-list) but at same time providing --zookeeper as well (so
> Zookeeper
> > connection)
> >
> > With the solution 2) we have the advantage to have the tool already in
> > Java when the --zookeeper option will be removed in a next s

Re: Command tools : from Scala to Java, from Zookeeper utils to Admin Client API

[Paolo Patierno]

Hi Ismael,


in this case I embrace your way :-)


To be more specific on what I was doing, It means that I can continue to 
develop the TopicCommand tool in Java using AdminClient API and then adding the 
deprecation for --zookeeper and the "tool switch" in the shell script. It 
doesn't need a KIP, does it ?

For the future --zookeeper removal, a KIP would be needed and I'll write that. 
Because it's something common to more tools, does it make sense to have only 
one KIP or writing a KIP for each effected tool ? I guess the second option 
because each KIP will have effect on a specific version (tools migration will 
be done over more releases). Is that right ?


Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma 
<ism...@juma.me.uk>
Sent: Tuesday, August 1, 2017 4:46 PM
To: dev@kafka.apache.org
Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to Admin 
Client API

Hi Paolo,

If the old tool is deprecated, then it's OK to add functionality to the new
tool without adding it to the old tool as long as the old tool still works.
So, there is some maintenance overhead to keep it working (but not to add
new features). In practice, the code used by the old tool is shared with
the broker so the maintenance overhead should be small.

Ismael

On Tue, Aug 1, 2017 at 5:12 PM, Paolo Patierno <ppatie...@live.com> wrote:

> Hi Ismael,
>
>
> your third option is really appealing for me :-)
>
>
> Just one thought about that ...
>
> If we decide to go in this way we could have an 1.0 release with both
> Scala (zookeeper) and Java (bootstrap-server) implementations but then the
> shell script to chose one of them with the --zookeeper deprecation.
>
> The final Scala (zookeeper) removal will happen on version 2.0 (following
> the policies about "major.minor.patch" release) but it could be months and
> months later.
>
> In the meantime, let's suppose that we want to add a new --foo option to
> the tool; we should do that on both implementations having them in sync and
> it could be a problem until on 2.0 release when we get rid of the Scala
> implementation.
>
> I really like your way but it could be better if we agree that new command
> options will be added only to the Java (bootstrap-server) implementation.
> It could mean pushing users to move from old to new tool ... that is good
> for us.
>
>
> What do you think ?
>
>
> Thanks,
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma <
> ism...@juma.me.uk>
> Sent: Tuesday, August 1, 2017 3:00 PM
> To: dev@kafka.apache.org
> Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to
> Admin Client API
>
> Hi Paolo,
>
> Another option is to write the new tool in Java without support for
> `--zookeeper` and include some logic in the shell script to pick the
> implementation based on the presence of `--bootstrap-server` or
> `--zookeeper`. This would mean that we can deprecate the Scala tool while
> still supporting it for existing users.
>
> I think this would involve the least amount of rewriting once we drop
> support for `--zookeeper`.
>
> Thoughts?
>
> Ismael
>
> On Tue, Aug 1, 2017 at 3:30 PM, Paolo Patierno <ppatie...@live.com> wrote:
>
> > Hi Tom,
> >
> >
> > thanks for your reply. I think that you are right and what you have
> > proposed should be the way to go.
> >
> >
> > Until today I have been working on refactoring the TopicCommand tool in
> > Java using the AdminClient getting rid of the Zookeeper usage in only
> "one
> > step" and maybe it's wrong.
> >
> > I'd like to have some input from committers as well to be sure that the
> > way is good about how handling such use cases.
> >
> >
> > Thanks,
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
>

Re: Command tools : from Scala to Java, from Zookeeper utils to Admin Client API

[Paolo Patierno]

Hi Ismael,


your third option is really appealing for me :-)


Just one thought about that ...

If we decide to go in this way we could have an 1.0 release with both Scala 
(zookeeper) and Java (bootstrap-server) implementations but then the shell 
script to chose one of them with the --zookeeper deprecation.

The final Scala (zookeeper) removal will happen on version 2.0 (following the 
policies about "major.minor.patch" release) but it could be months and months 
later.

In the meantime, let's suppose that we want to add a new --foo option to the 
tool; we should do that on both implementations having them in sync and it 
could be a problem until on 2.0 release when we get rid of the Scala 
implementation.

I really like your way but it could be better if we agree that new command 
options will be added only to the Java (bootstrap-server) implementation. It 
could mean pushing users to move from old to new tool ... that is good for us.


What do you think ?


Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma 
<ism...@juma.me.uk>
Sent: Tuesday, August 1, 2017 3:00 PM
To: dev@kafka.apache.org
Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to Admin 
Client API

Hi Paolo,

Another option is to write the new tool in Java without support for
`--zookeeper` and include some logic in the shell script to pick the
implementation based on the presence of `--bootstrap-server` or
`--zookeeper`. This would mean that we can deprecate the Scala tool while
still supporting it for existing users.

I think this would involve the least amount of rewriting once we drop
support for `--zookeeper`.

Thoughts?

Ismael

On Tue, Aug 1, 2017 at 3:30 PM, Paolo Patierno <ppatie...@live.com> wrote:

> Hi Tom,
>
>
> thanks for your reply. I think that you are right and what you have
> proposed should be the way to go.
>
>
> Until today I have been working on refactoring the TopicCommand tool in
> Java using the AdminClient getting rid of the Zookeeper usage in only "one
> step" and maybe it's wrong.
>
> I'd like to have some input from committers as well to be sure that the
> way is good about how handling such use cases.
>
>
> Thanks,
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Tom Bentley <t.j.bent...@gmail.com>
> Sent: Friday, July 28, 2017 10:36 AM
> To: dev@kafka.apache.org
> Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to
> Admin Client API
>
> Hi Paolo,
>
> Replies in line...
>
> On 28 July 2017 at 11:14, Paolo Patierno <ppatie...@live.com> wrote:
>
> > Hi committers,
> >
> > in my understanding there is the common idea to move all tools from Scala
> > to Java and then using the new Admin Client API instead of using the
> > Zookeeper connection.
> >
> > Regarding this subject I started to work on refactoring the TopicCommand
> > tool but with two steps at same time :
> >
> >
> >   *   re-writing it in Java
> >   *   removing --zookeeper option (so no Zookeeper connection) and adding
> > --broker-list option (so using the Admin Client API)
> >
> > Of course, such option substitution is a breaking change for the tool
> (and
> > the users who are using it).
> > In such a scenario, the two steps path should be needed : first
> > deprecation, then removal (for the --zookeeper option)
> >
>
> A change to tools (and their options) requires a KIP. There's no
> fundamental reason why both couldn't be supported during a transition
> period. So I doubt a KIP that didn't propose a transition period would get
> passed.
>
>
> It seems that in the "deprecation" phase we have two possible solutions :
> >
> >
> >   1.  Adding Admin Client API to the Scala tools and so the --broker-list
> > option and a warning on --zookeeper for deprecation
> >   2.  Rewriting the tool in Java using the Admin Client API (so
> > --broker-list) but at same time providing --zookeeper as well (so
> Zookeeper
> > connection)
> >
> > 

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

[Paolo Patierno]

Hi,


Regarding the double API I agree on having only one. Compared to an HTTP REST 
API you could have an "already executed" (fast) operation so the HTTP response 
status code could be just 200 OK while a "long running" operation could have an 
HTTP response status code as 202 ACCEPTED (but the processing has not 
completed).


Regarding adding the possibility to alter the topic config through the 
AlterTopic API, the current TopicCommand implementation provides a warning on 
doing this suggesting to use the ConfigCommand tool. So it would be a step back 
allowing to do the configs change with the alter topic as well.


Last thing on the KIP ... the "timeout" field in the AlterTopicRequest is 
missing in the table with related description.


Thanks


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Tom Bentley <t.j.bent...@gmail.com>
Sent: Tuesday, August 1, 2017 2:22 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use 
AdminClient

Hi Ismael,

Thanks for taking the time to look at this.

Currently the proposal around the ReplicaStatusRequest/Response just allows
you to see the lag for the given replicas. It's not something that's tied
to a prior request to alter the topic at all, though obviously you can use
it to monitor how a reassignment is progressing.

I think having the API independent of AlterTopicsRequest/Response is a good
thing, because people might want to know that independently of a
reassignment, for example. But the fact that you just see the instantaneous
lag makes it less than perfect for monitoring a reassignment. You would
have to make multiple calls to figure out how quickly the lag is
diminishing or growing. But I don't want each partition leader having to
maintain some kind of stateful calculation just in case it is queried for
the state of that partition.

So I think that problem might better be solved in the
kafka-reassign-partitions.sh tool. Let _it_ make multiple requests for
status and figure out how the lag is changing.

>From the PoV of the API presented by the protocol, the proposal has moved
in the direction of making AlterTopicsRequest/Response more and more like
CreateTopicsRequest/Response, and I think it has improved as a result of
this.

Personally I don't think we should split the AlterTopicsRequest/Response
API in two, one which we expect to be short running and another which we
expect to be long running. I think that would make the API less symmetric,
and thus less easily understood. What might make more sense is to use the
return code to indicate the distinction between, "OK, it's done completely"
and "OK, it's been started, but will finish asynchronously" -- In a way
that distinction already exists via the timeout in AlterTopicsOptions. In
other words a configs change might return NONE, but a reassignment (with a
short timeout) might return REQUEST_TIMED_OUT, though I guess the problem
with that is that TimeoutException is retriable, and REQUEST_TIMED_OUT
doesn't suggest that you can use another API for query for completion, so
perhaps a dedicated status code is warranted (ASYNCHRONOUS_COMPLETION?)

To be honest, if we added the ability to update the topic configs, it would
be even more symmetric, and I wouldn't be opposed to that, except that I
dislike having two code paths for achieving the same thing, and we already
have AlterConfigsRequest and Response.

What do other people think?

Cheers,

Tom


On 1 August 2017 at 14:25, Ismael Juma <ism...@juma.me.uk> wrote:

> Hi Tom,
>
> A high-level point for discussion before going into the details. The
> proposed protocol API `alterTopics` has 2 types of operations:
>
> 1. Operations that cause data movement (or deletion): increase/decrease of
> replication factor and partition reassignment. These are currently done by
> `kafka-reassign-partitions.sh` and having a way to check the progress is
> useful.
>
> 2. Operations that don't involve data movement: increase the number of
> partitions and update topic configs (this is not in the proposal, but could
> be). These are currently done by `kafka-topics.sh` and they don't usually
> take very long so progress reporting is less important.
>
> It would be worth thinking whether having 2 separate protocol APIs would be
> better. I can see pros and cons, so I'd be interested in what you and
> others think.
>
> Ismael
>
> On Tue, Aug 1, 2017 at 10:19 AM, Tom Bentley <t.j.bent...@gmail.com>
> wrote:
>
> > I have added the timeout I mentioned before, because it makes the
> &g

Re: Command tools : from Scala to Java, from Zookeeper utils to Admin Client API

[Paolo Patierno]

Hi Tom,


thanks for your reply. I think that you are right and what you have proposed 
should be the way to go.


Until today I have been working on refactoring the TopicCommand tool in Java 
using the AdminClient getting rid of the Zookeeper usage in only "one step" and 
maybe it's wrong.

I'd like to have some input from committers as well to be sure that the way is 
good about how handling such use cases.


Thanks,


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Tom Bentley <t.j.bent...@gmail.com>
Sent: Friday, July 28, 2017 10:36 AM
To: dev@kafka.apache.org
Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to Admin 
Client API

Hi Paolo,

Replies in line...

On 28 July 2017 at 11:14, Paolo Patierno <ppatie...@live.com> wrote:

> Hi committers,
>
> in my understanding there is the common idea to move all tools from Scala
> to Java and then using the new Admin Client API instead of using the
> Zookeeper connection.
>
> Regarding this subject I started to work on refactoring the TopicCommand
> tool but with two steps at same time :
>
>
>   *   re-writing it in Java
>   *   removing --zookeeper option (so no Zookeeper connection) and adding
> --broker-list option (so using the Admin Client API)
>
> Of course, such option substitution is a breaking change for the tool (and
> the users who are using it).
> In such a scenario, the two steps path should be needed : first
> deprecation, then removal (for the --zookeeper option)
>

A change to tools (and their options) requires a KIP. There's no
fundamental reason why both couldn't be supported during a transition
period. So I doubt a KIP that didn't propose a transition period would get
passed.


It seems that in the "deprecation" phase we have two possible solutions :
>
>
>   1.  Adding Admin Client API to the Scala tools and so the --broker-list
> option and a warning on --zookeeper for deprecation
>   2.  Rewriting the tool in Java using the Admin Client API (so
> --broker-list) but at same time providing --zookeeper as well (so Zookeeper
> connection)
>
> With the solution 2) we have the advantage to have the tool already in
> Java when the --zookeeper option will be removed in a next step. In any
> case we have to write the part related to use the Admin Client API so it
> make more sense to me option 2) porting the Zookeeper needed code from
> Scala to Java (then removing it in the next phase).
>
> Is my understanding right on how we have to handle deprecation and removal
> for something that is a breaking change ?
> Or this case is something "special" and we can live with a new Java based
> tool without zookeeper but with Admin Client API at same time ?
>
> Of course having both Admin Client API and Zookeeper utils working at the
> same time in the tools means more complexity in the code but it's something
> that could be factorized.
>

I think the right thing to do would be:

1. deprecate the old option, adding support for the replacement option
(using the AdminClient). Keep the code in scala. All this is in one KIP.
2. Remove the old option (needs a KIP)
3. Rewrite the tool in Java.

Parts 2 and 3 could be done at the same time. I don't believe part 3 needs
a KIP if it were a drop-in replacement.

The reason I think this is the right way to proceed is:

* It gives users a transition period to learn about the new option, and
replace any tooling of their own.
* By keeping the tool in scala you get to release your new AdminClient API
and get to iron out all the creases while users still have the --zookeeper
option as a fallback.
* Then when you know the AdminClient API works in the field you have a
straight porting job to do, and you have less code to port because you
don't have to port the code to support --zookeeper.

But I'm fairly new here, so maybe a committer will chime in an correct me
where I'm wrong!

Cheers,

Tom

Re: Kafka Streams debugging with "no fluent" API choice

[Paolo Patierno]

Hi Damian,


changing the print() method for sure needs a KIP but I guess there is some 
reason we don't know why they decided to not have a fluent API for that.

Regarding my JIRA I don't think a KIP is required, it's just internal stuff ... 
no ?


Thanks


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Damian Guy <damian@gmail.com>
Sent: Tuesday, August 1, 2017 2:11 PM
To: dev@kafka.apache.org
Subject: Re: Kafka Streams debugging with "no fluent" API choice

Hi Paolo,

The change would require a KIP as it is a public API change.  I don't see
any harm in making the change, but I also don't think it is that difficult
to use peek to achieve the same thing.

Thanks,
Damian

On Tue, 1 Aug 2017 at 13:52 Paolo Patierno <ppatie...@live.com> wrote:

> Thanks Damian,
>
>
> I knew about that but you have to write the code for printing by yourself.
> Of course you can do that even with the print() without using the default
> keyvalue mapper but passing a custom one.
>
> At same time if you want to print you should use a Serdes for key and
> value if they are bytes and it's something which happens for free with
> print() passing Serdes instances.
>
>
> Another point is ...
>
> as both foreach() and peek() methods relies on the KStreamPeek node, it
> could be the same for the print() method which uses a KStreamPrint that is
> a special KStreamPeek with forwardDownStream = false and providing the
> usage of Serdes. For this I have opened the following JIRA :
>
>
> https://issues.apache.org/jira/browse/KAFKA-5684
>
>
> What do you think ?
>
>
> Thanks
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Damian Guy <damian@gmail.com>
> Sent: Tuesday, August 1, 2017 12:11 PM
> To: dev@kafka.apache.org
> Subject: Re: Kafka Streams debugging with "no fluent" API choice
>
> I don't know specifically why this is removed, however if you want to get
> the same functionality you can use peek, i.e:
>
> stream.map(...).peek(...).filter(..)
>
> You can log the key values out in the peek call.
>
> On Tue, 1 Aug 2017 at 11:48 Paolo Patierno <ppatie...@live.com> wrote:
>
> > Hi guys,
> >
> >
> > I was thinking about Kafka Streams debug features and why the print()
> > method overloads didn't return a KStream, in order to have a fluent DSL
> > construction even during debugging, but just void.
> >
> > Then I came across this PR :
> >
> >
> > https://github.com/apache/kafka/pull/1187
> >
> >
> > May I ask why the form :
> >
> >
> > stream1 = source.map(...).filter(...);
> > stream1.print();   // this is for debugging, deleted before moving to
> > productiong
> >
> > stream2 = stream1.transform(...);
> > stream2.print();   // this is for debugging, deleted before moving to
> > productiong
> >
> > was considered better then the fluent one :
> >
> >
> > source.map(...).filter(...)
> >   .print()   // this is for debugging, deleted before moving to
> > productiong
> >   .transform(...)
> >   .print();   // this is for debugging, deleted before moving to
> > productiong
> >
> >
> > In this first case the user has to break the topology for printing and
> > after that, when the code works fine, he has to rewrite the code for
> > avoiding these broken processors having a fluent construction.
> >
> > In the second solution, the user has just to remove the print() calls
> > without touching the other parts of the code.
> >
> > I really liked the first solution (it was something I was going to
> propose
> > before coming across the PR :-)).
> >
> >
> > Why this preference on the first one ?
> >
> >
> > Thanks
> >
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
>

[jira] [Created] (KAFKA-5684) KStreamPrintProcessor as customized KStreamPeekProcessor

[Paolo Patierno (JIRA)]

Paolo Patierno created KAFKA-5684:
-

 Summary: KStreamPrintProcessor as customized KStreamPeekProcessor
 Key: KAFKA-5684
 URL: https://issues.apache.org/jira/browse/KAFKA-5684
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Paolo Patierno
Assignee: Paolo Patierno
Priority: Minor


Hi,
the {{KStreamPrintProcessor}} is implemented from scratch (from the 
{{AbstractProcessor}}) and the same for the related supplier.
It looks to me that it's just a special {{KStreamPeekProcessor}} with 
forwardDownStream to false and that allows the possibility to specify Serdes 
instances used if key/values are bytes.
At same time used by a {{print()}} method it provides a fast way to print data 
flowing through the pipeline (while using just {{peek()}} you need to write the 
code).
I think that it could be useful to refactoring the {{KStreamPrintProcessor}} as 
derived from the {{KStreamPeekProcessor}} customizing its behavior.

Thanks,
Paolo.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Kafka Streams debugging with "no fluent" API choice

[Paolo Patierno]

Thanks Damian,


I knew about that but you have to write the code for printing by yourself. Of 
course you can do that even with the print() without using the default keyvalue 
mapper but passing a custom one.

At same time if you want to print you should use a Serdes for key and value if 
they are bytes and it's something which happens for free with print() passing 
Serdes instances.


Another point is ...

as both foreach() and peek() methods relies on the KStreamPeek node, it could 
be the same for the print() method which uses a KStreamPrint that is a special 
KStreamPeek with forwardDownStream = false and providing the usage of Serdes. 
For this I have opened the following JIRA :


https://issues.apache.org/jira/browse/KAFKA-5684


What do you think ?


Thanks


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Damian Guy <damian@gmail.com>
Sent: Tuesday, August 1, 2017 12:11 PM
To: dev@kafka.apache.org
Subject: Re: Kafka Streams debugging with "no fluent" API choice

I don't know specifically why this is removed, however if you want to get
the same functionality you can use peek, i.e:

stream.map(...).peek(...).filter(..)

You can log the key values out in the peek call.

On Tue, 1 Aug 2017 at 11:48 Paolo Patierno <ppatie...@live.com> wrote:

> Hi guys,
>
>
> I was thinking about Kafka Streams debug features and why the print()
> method overloads didn't return a KStream, in order to have a fluent DSL
> construction even during debugging, but just void.
>
> Then I came across this PR :
>
>
> https://github.com/apache/kafka/pull/1187
>
>
> May I ask why the form :
>
>
> stream1 = source.map(...).filter(...);
> stream1.print();   // this is for debugging, deleted before moving to
> productiong
>
> stream2 = stream1.transform(...);
> stream2.print();   // this is for debugging, deleted before moving to
> productiong
>
> was considered better then the fluent one :
>
>
> source.map(...).filter(...)
>   .print()   // this is for debugging, deleted before moving to
> productiong
>   .transform(...)
>   .print();   // this is for debugging, deleted before moving to
> productiong
>
>
> In this first case the user has to break the topology for printing and
> after that, when the code works fine, he has to rewrite the code for
> avoiding these broken processors having a fluent construction.
>
> In the second solution, the user has just to remove the print() calls
> without touching the other parts of the code.
>
> I really liked the first solution (it was something I was going to propose
> before coming across the PR :-)).
>
>
> Why this preference on the first one ?
>
>
> Thanks
>
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>

Kafka Streams debugging with "no fluent" API choice

[Paolo Patierno]

Hi guys,


I was thinking about Kafka Streams debug features and why the print() method 
overloads didn't return a KStream, in order to have a fluent DSL construction 
even during debugging, but just void.

Then I came across this PR :


https://github.com/apache/kafka/pull/1187


May I ask why the form :


stream1 = source.map(...).filter(...);
stream1.print();   // this is for debugging, deleted before moving to 
productiong

stream2 = stream1.transform(...);
stream2.print();   // this is for debugging, deleted before moving to 
productiong

was considered better then the fluent one :


source.map(...).filter(...)
  .print()   // this is for debugging, deleted before moving to 
productiong
  .transform(...)
  .print();   // this is for debugging, deleted before moving to 
productiong


In this first case the user has to break the topology for printing and after 
that, when the code works fine, he has to rewrite the code for avoiding these 
broken processors having a fluent construction.

In the second solution, the user has just to remove the print() calls without 
touching the other parts of the code.

I really liked the first solution (it was something I was going to propose 
before coming across the PR :-)).


Why this preference on the first one ?


Thanks



Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

Re: [DISCUSS] KIP-176: Remove deprecated new-consumer option from tools

[Paolo Patierno]

Hi devs,


I didn't see any new comments around this KIP. Maybe we could start the vote 
for that ?

Before that, could it be worth to check (and eventually merge) the following 
PRs about deprecation for having that in the 1.0.0 release ? (then removing in 
the next release cycle).


https://github.com/apache/kafka/pull/3537
https://github.com/apache/kafka/pull/3555


Thanks


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>


____
From: Paolo Patierno <ppatie...@live.com>
Sent: Thursday, July 20, 2017 12:47 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-176: Remove deprecated new-consumer option from tools

Hi all,

I have updated the KIP-176 
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-176%3A+Remove+deprecated+new-consumer+option+for+tools>
 in order to consider to remove the --new-consumer option not only for 
ConsoleConsumer but even for other tools which use it : ConsumerPerformance and 
ConsumerGroupCommand.

I'm also working on a PR for a new-consumer option deprecation as first step 
for ConsumerPerformance and ConsumerGroupCommand (as already done, pushing a 
PR<https://github.com/apache/kafka/pull/3537>, for ConsoleConsumer).


Thanks,

Paolo


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma 
<ism...@juma.me.uk>
Sent: Tuesday, July 18, 2017 1:09 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-176: Remove deprecated new-consumer option in 
ConsoleConsumer tool

On Mon, Jul 17, 2017 at 10:11 AM, Matthias J. Sax <matth...@confluent.io>
wrote:

> I thinks, for backward compatibility, this check should be there. The
> only change should be, that if `--bootstrap-servers` is used, one can
> omit `--new-consumer` (and if used, as warning is printed).
>

You can omit "--new-consumer" since 0.10.2. Paolo's PR will add the warning
so that we can eventually remove the switch altogether. To be honest, we
could just remove it at the same time as we remove the old consumer.

Ismael

Command tools : from Scala to Java, from Zookeeper utils to Admin Client API

[Paolo Patierno]

Hi committers,

in my understanding there is the common idea to move all tools from Scala to 
Java and then using the new Admin Client API instead of using the Zookeeper 
connection.

Regarding this subject I started to work on refactoring the TopicCommand tool 
but with two steps at same time :


  *   re-writing it in Java
  *   removing --zookeeper option (so no Zookeeper connection) and adding 
--broker-list option (so using the Admin Client API)

Of course, such option substitution is a breaking change for the tool (and the 
users who are using it).
In such a scenario, the two steps path should be needed : first deprecation, 
then removal (for the --zookeeper option)

It seems that in the "deprecation" phase we have two possible solutions :


  1.  Adding Admin Client API to the Scala tools and so the --broker-list 
option and a warning on --zookeeper for deprecation
  2.  Rewriting the tool in Java using the Admin Client API (so --broker-list) 
but at same time providing --zookeeper as well (so Zookeeper connection)

With the solution 2) we have the advantage to have the tool already in Java 
when the --zookeeper option will be removed in a next step. In any case we have 
to write the part related to use the Admin Client API so it make more sense to 
me option 2) porting the Zookeeper needed code from Scala to Java (then 
removing it in the next phase).

Is my understanding right on how we have to handle deprecation and removal for 
something that is a breaking change ?
Or this case is something "special" and we can live with a new Java based tool 
without zookeeper but with Admin Client API at same time ?

Of course having both Admin Client API and Zookeeper utils working at the same 
time in the tools means more complexity in the code but it's something that 
could be factorized.


Thanks,

Paolo.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

Consumer assign vs subscribe influence on __consumer_offset

[Paolo Patierno]

Hi devs,

I have one simple topic named "test" with only one partition. then ...

I have a consumer C1 using assign() for having assigned partition 0 of topic 
"test" and this consumer has group.id = "mygroup"

I have another consumer C2 using subscribe() for having assigned partitions 
from topic "test" (so it will have assigned partition 0) and this consumer has 
group.id = "mygroup" (the same as above)


USE CASE 1

Both consumers have auto commit enabled

Reading messages from the __consumer_offset I always see only ONE message of 
this type


[mygroup,test,0]::[OffsetMetadata[7,NO_METADATA],CommitTime 
1501149394800,ExpirationTime 1501235794800]


So even if both consumers receives messages from "test" and have auto commit 
enabled, only one offset commit message is there. Is that normal ?


USE CASE 2

Consumer C1 has auto commit enabled

Consumer C2 has auto commit disabled and it doesn't execute any offset commit 
manually

Both consumers get messages from "test" but, there are no offset commit 
messages into __consumer_offset. Is that normal ?

I expect that the C1 with auto commit enabled sends the offset commit message 
but it doesn't happen. It seems that the C2 settings about auto commit disabled 
influences the C1 settings on that.


USE CASE 3 (opposite of 2)

Consumer C1 has auto commit disabled

Consumer C2 has auto commit enabled

Both consumers get messages from "test" and I can see offset commit messages 
into __consumer_offset (from C2).


[mygroup,test,0]::[OffsetMetadata[7,NO_METADATA],CommitTime 
1501149394800,ExpirationTime 1501235794800]


In this use case, the auto commit disabled by C1 doesn't influence the enabled 
from C2. It happened in the use case 2 (in the opposite direction).


Conclusion ... it's possible that there is something that I didn't understand 
about this interaction between consumers in the same group but asking for 
partitions in two different way (assign vs subscribe).


Thanks,

Paolo


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

About "exclude.internal.topics" ... some thoughts

[Paolo Patierno]

Hi devs,


I noticed that the "exclude.internal.topics" consumer property has an effect 
only if the consumer subscribes specifying a "topic pattern" and not a topic 
name directly.

Of course if I run something like the console consumer passing "--topic 
__consumer_offsets", setting the above property does have no effect, you'll see 
messages sent to the internal __consumer_offsets topic. If you run same 
application using the "--whitelist" option then the property has its effect. We 
know that the difference is that passing --topic call a subscribe on specific 
topic name while passing --whitelist call a subscribe for a topic pattern.


Said that, the following comment on top of OffsetMessageFormatter class is 
misleading in my opinion :


"Consumer should also set exclude.internal.topics to false"


The same in the documentation about the property :


"Whether records from internal topics (such as offsets) should be exposed to 
the consumer. If set to true the only way to receive records from an internal 
topic is subscribing to it."


At least I would change in "... topic is subscribing directly to it." or 
something that explains the difference between subscribing using a topic filter 
(so the property takes an effect) or direct topic name (property does have no 
effect).


What do you think ?


Thanks,

Paolo.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

[jira] [Created] (KAFKA-5643) Using _DUCKTAPE_OPTIONS has no effect on executing tests

[Paolo Patierno (JIRA)]

Paolo Patierno created KAFKA-5643:
-

 Summary: Using _DUCKTAPE_OPTIONS has no effect on executing tests
 Key: KAFKA-5643
 URL: https://issues.apache.org/jira/browse/KAFKA-5643
 Project: Kafka
  Issue Type: Bug
  Components: system tests
Reporter: Paolo Patierno
Assignee: Paolo Patierno


Hi,
as described in the documentation, you should be able to enable debugging using 
the following line :

_DUCKTAPE_OPTIONS="--debug" bash tests/docker/run_tests.sh | tee debug_logs.txt

Instead the _DUCKTAPE_OPTIONS isn't available in the run_tests.sh script so 
it's not passed to the ducker-ak and finally on the ducktape command line.

Thanks,
Paolo.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

[Paolo Patierno]

Hi Tom,


I haven taken a look to the updated KIP, some thoughts :


  *   in the "Public Interfaces" section you wrote 
alterTopics(Set) but then a collection is used (instead of a set) 
in the Proposed Changes section. I'm ok with collection.
  *   in the summary of the alterTopics method you say "The request can change 
the number of partitions, replication factor and/or the partition assignments." 
I think that the "and/or" is misleading (at least for me). For the TopicCommand 
tool you can specify partitions AND replication factor ... OR partition 
assignments (but not for example partitions AND replication factor AND 
partition assignments). Maybe it could be "The request can change the number of 
partitions and the related replication factor or specifying a partition 
assignments."
  *   I know that it would be a breaking change in the Admin Client API but why 
having an AlteredTopic class which is quite similar to the already existing 
NewTopic class ? I know that using NewTopic for the alter method could be 
misleading. What about renaming NewTopic in something like AdminTopic ? At same 
time it means that the TopicDetails class (which you can get from the current 
NewTopic) should be outside the CreateTopicsRequest because it could be used in 
the AlterTopicsRequest as well.
  *   A typo in the ReplicaStatus : gpartition() instead of partition()
  *   In the AlterTopicRequets
 *   the replication factor should be INT16
 *   I would use same fields name as CreateTopicsRequest (they are quite 
similar)
  *   What's the broker id in the ReplicaStatusRequest ?
  *   Thinking aloud, could make sense having "Partition" in the 
ReplicaStatusRequest as an array so that I can specify in only one request the 
status for partitions I'm interested in, in order to avoid e request for each 
partition for the same topic. Maybe empty array could mean .. "give me status 
for all partitions of this topic". Of course it means that the 
ReplicaStatusResponse should change because we should have an array with 
partition, broker, lag and so on

Thanks,
Paolo


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>



From: Tom Bentley <t.j.bent...@gmail.com>
Sent: Tuesday, July 25, 2017 11:07 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use 
AdminClient

Hi Ismael and Paolo,

I've updated my KIP [1] to describe an alterTopics() API which would work
for kafka-reassign-partitions.sh. It's still a bit rough, but should be a
good basis for a KIP to cover both tools.

As a first step, if Paolo could review this and check it's compatible with
what he needs for the kafka-topics.sh tool that would be great. Then we can
add to describe his changes to the rest of the kafka-topics.sh tool,
assuming Paolo is happy with this arrangement, Paolo?

Cheers,

Tom

[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-179+-+Change+ReassignPartitionsCommand+to+use+AdminClient

On 25 July 2017 at 11:38, Ismael Juma <ism...@juma.me.uk> wrote:

> Hi Tom and Paolo,
>
> It's true that increasing the number of partitions is done via the
> kafka-topics tool, which is also being converted to use the AdminClient
> (but via a different JIRA). I also agree that it would be good to consider
> if alterTopics would be a sensible way to support all the use cases or if
> it's better to have separate APIs. I think it makes sense to have a single
> KIP though as they are related and it will be easier to evaluate as a
> whole.
>
> Does that make sense?
>
> Ismael
>
> On Tue, Jul 25, 2017 at 10:16 AM, Paolo Patierno <ppatie...@live.com>
> wrote:
>
> > Hi,
> >
> >
> > I was digging into it because I need something like an Admin Client alter
> > API for my work on rewriting the TopicCommand tool using them.
> >
> > The AlterConfigs API is used for changing topic level configuration (i.e.
> > retention.ms, retention.bytes and so on).
> >
> > A new AlterTopic API could be better in order to change topic "high
> level"
> > structure so number of partitions, replication factors and so on.
> >
> > My opinion is that we need separate API because from my point of view
> they
> > are different settings.
> >
> >
> > Thanks,
> >
> > Paolo.
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno&l