[jira] [Created] (KAFKA-14155) Kafka Stream - State transition from RUNNING to ERROR
Harsha Nadig created KAFKA-14155: Summary: Kafka Stream - State transition from RUNNING to ERROR Key: KAFKA-14155 URL: https://issues.apache.org/jira/browse/KAFKA-14155 Project: Kafka Issue Type: Bug Reporter: Harsha Nadig 2022-08-09 06:05:17 [-StreamThread-1] INFO c.g.o.d.k.c.KafkaStreamsConfig.lambda$null$0[82] State transition from RUNNING to ERROR 2022-08-09 06:05:17 [-StreamThread-1] ERROR o.a.k.streams.KafkaStreams.maybeSetError[443] stream-client [] All stream threads have died. The instance will be in error state and should be closed. 2022-08-09 06:05:17 [] INFO o.a.k.s.p.i.StreamThread.completeShutdown[935] stream-thread [-StreamThread-1] Shutdown complete 2022-08-09 06:05:17 [-StreamThread-1] ERROR c.g.o.d.k.c.KafkaStreamsConfig.uncaughtException[88] StreamsThread threadId: -StreamThread-1 TaskManager MetadataState: Tasks: throws exception: org.apache.kafka.streams.errors.StreamsException: Exception caught in process. taskId=12_1, processor=KSTREAM-SOURCE-, topic=, partition=1, offset=, stacktrace=java.lang.NullPointerException at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:696) at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:1033) at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:690) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) Caused by: java.lang.NullPointerException: null -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] KIP - 405: Kafka Tiered Storage.
+1 (binding). Thanks, Harsha On Thu, Feb 11, 2021 at 6:21 AM Satish Duggana wrote: > Hi All, > We would like to start voting on “KIP-405: Kafka Tiered Storage”. > > For reference here is the KIP: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage > > Thanks, > Satish. >
Re: [DISCUSS] KIP-688: Support dynamic update of delete.topic.enable config
Hi Gwen, Agree with you ACLs would be a better option. But not all installations will be able to have secure setup and authorizer. In a hosted service where they may not be ACLs or security setup this flag helps prevent accidental topic deletion. This config acted as a good gatekeeper to prevent accidents in deletion as there is no confirmation or there is time delay in execution of the deletion of a topic or a soft delete etc... It's pretty much a catastrophic event in case of accidental deletion. We also need to consider that not all installations will be secure and we need to provide a simpler way to gate keep certain functionalities such as delete topic. Thanks, Harsha On Wed, Nov 25, 2020 at 10:25 AM Gwen Shapira wrote: > Thanks for bringing this up! > > I disagree that " there is a valid use-case to keep the flag > "delete.topic.enable" as "false" in server.properties configs, so that > someone doesn't accidentally delete a topic,". > The flag was added and set to "false" back in the days when topic > deletion was not a stable feature, and enabling it meant a risk of > encountering serious bugs. Things changed in the last 4-5 years: > - Topic deletion is now a stable and reliable feature. We have it > enabled by default in Confluent Cloud and over a few years and > hundreds of clusters, there was maybe one issue related to deletion of > large number of partitions, and that was fixed. > - We have authorization and ACLs, so the permissions to delete topics > can be granted on an individual basis. This is the correct way to > handle concerns about someone accidentally deleting data valuable > data. Note that ACLs are dynamic. > > IMO, the best solution is to use the upcoming major release (3.0) to > enable topic deletion by default and remove the flag entirely. > > Gwen > > On Wed, Nov 25, 2020 at 9:36 AM Prateek Agarwal > wrote: > > > > Hi, > > > > I've created KIP-688 to enable support for dynamic update of > > delete.topic.enable config. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-688%3A+Support+dynamic+update+of+delete.topic.enable+config > > > > Please take a look and let me know if you have any feedback. > > > > Thanks > > -- > > Prateek Agarwal > > > > -- > Gwen Shapira > Engineering Manager | Confluent > 650.450.2760 | @gwenshap > Follow us: Twitter | blog >
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
Thanks Kowshik for the link. Seems reasonable, as we discussed on the call, code and completion of this KIP will be taken up by us. Regarding Milestone 2, what you think it needs to be clarified there? I believe what we are promising in the KIP along with unit tests, systems tests will be delivered and we can call that as preview. We will be running this in our production and continue to provide the data and metrics to push this feature to GA. On Tue, Nov 10, 2020 at 10:07 AM, Kowshik Prakasam wrote: > Hi Harsha/Satish, > > Thanks for the discussion today. Here is a link to the KIP-405 development > milestones google doc we discussed in the meeting today: https://docs. > google.com/document/d/1B5_jaZvWWb2DUpgbgImq0k_IPZ4DWrR8Ru7YpuJrXdc/edit > . I have shared it with you. Please have a look and share your > feedback/improvements. As we discussed, things are clear until milestone 1. > Beyond that, we can discuss it again (perhaps in next sync or later), once > you have thought through the implementation plan/milestones and release > into preview in 3.0. > > Cheers, > Kowshik > > On Tue, Nov 10, 2020 at 6:56 AM Satish Duggana > wrote: > > Hi Jun, > Thanks for your comments. Please find the inline replies below. > > 605.2 "Build the local leader epoch cache by cutting the leader epoch > sequence received from remote storage to [LSO, ELO]." I mentioned an issue > earlier. Suppose the leader's local start offset is 100. The follower finds > a remote segment covering offset range [80, 120). The producerState with > this remote segment is up to offset 120. To trim the producerState to > offset 100 requires more work since one needs to download the previous > producerState up to offset 80 and then replay the messages from 80 to 100. > It seems that it's simpler in this case for the follower just to take the > remote segment as it is and start fetching from offset 120. > > We chose that approach to avoid any edge cases here. It may be possible > that the remote log segment that is received may not have the same leader > epoch sequence from 100-120 as it contains on the leader(this can happen > due to unclean leader). It is safe to start from what the leader returns > here.Another way is to find the remote log segment > > 5016. Just to echo what Kowshik was saying. It seems that > RLMM.onPartitionLeadershipChanges() is only called on the replicas for a > partition, not on the replicas for the __remote_log_segment_metadata > partition. It's not clear how the leader of __remote_log_segment_metadata > obtains the metadata for remote segments for deletion. > > RLMM will always receive the callback for the remote log metadata topic > partitions hosted on the local broker and these will be subscribed. I will > make this clear in the KIP. > > 5100. KIP-516 has been accepted and is being implemented now. Could you > update the KIP based on topicID? > > We mentioned KIP-516 and how it helps. We will update this KIP with all > the changes it brings with KIP-516. > > 5101. RLMM: It would be useful to clarify how the following two APIs are > used. According to the wiki, the former is used for topic deletion and the > latter is used for retention. It seems that retention should use the former > since remote segments without a matching epoch in the leader (potentially > due to unclean leader election) also need to be garbage collected. The > latter seems to be used for the new leader to determine the last tiered > segment. > default Iterator > listRemoteLogSegments(TopicPartition topicPartition) > Iterator listRemoteLogSegments(TopicPartition > topicPartition, long leaderEpoch); > > Right,.that is what we are currently doing. We will update the javadocs > and wiki with that. Earlier, we did not want to remove the segments which > are not matched with leader epochs from the ladder partition as they may be > used later by a replica which can become a leader (unclean leader election) > and refer those segments. But that may leak these segments in remote > storage until the topic lifetime. We decided to cleanup the segments with > the oldest incase of size based retention also. > > 5102. RSM: > 5102.1 For methods like fetchLogSegmentData(), it seems that they can use > RemoteLogSegmentId instead of RemoteLogSegmentMetadata. > > It will be useful to have metadata for RSM to fetch log segment. It may > create location/path using id with other metadata too. > > 5102.2 In fetchLogSegmentData(), should we use long instead of Long? > > Wanted to keep endPosition as optional to read till the end of the segment > and avoid sentinels. > > 5102.3 Why only some of the methods have default implementation and others > Don't? > > Actually, RSM will not have any defaul
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
Hi All, We are all working through the last meeting feedback. I'll cancel the tomorrow 's meeting and we can meanwhile continue our discussion in mailing list. We can start the regular meeting from next week onwards. Thanks, Harsha On Fri, Sep 04, 2020 at 8:41 AM, Satish Duggana < satish.dugg...@gmail.com > wrote: > > > > Hi Jun, > Thanks for your thorough review and comments. Please find the inline > replies below. > > > > 600. The topic deletion logic needs more details. > 600.1 The KIP mentions "The controller considers the topic partition is > deleted only when it determines that there are no log segments for that > topic partition by using RLMM". How is this done? > > > > It uses RLMM#listSegments() returns all the segments for the given topic > partition. > > > > 600.2 "If the delete option is enabled then the leader will stop RLM task > and stop processing and it sets all the remote log segment metadata of > that partition with a delete marker and publishes them to RLMM." We > discussed this earlier. When a topic is being deleted, there may not be a > leader for the deleted partition. > > > > This is a good point. As suggested in the meeting, we will add a separate > section for topic/partition deletion lifecycle and this scenario will be > addressed. > > > > 601. Unclean leader election > 601.1 Scenario 1: new empty follower > After step 1, the follower restores up to offset 3. So why does it have > LE-2 at offset 5? > > > > Nice catch. It was showing the leader epoch fetched from the remote > storage. It should be shown with the truncated till offset 3. Updated the > KIP. > > > > 601.2 senario 5: After Step 3, leader A has inconsistent data between its > local and the tiered data. For example. offset 3 has msg 3 LE-0 locally, > but msg 5 LE-1 in the remote store. While it's ok for the unclean leader > to lose data, it should still return consistent data, whether it's from > the local or the remote store. > > > > There is no inconsistency here as LE-0 offsets are [0, 4] and LE-2: > [5, ]. It will always get the right records for the given offset and > leader epoch. In case of remote, RSM is invoked to get the remote log > segment that contains the given offset with the leader epoch. > > > > 601.4 It seems that retention is based on > listRemoteLogSegments(TopicPartition topicPartition, long leaderEpoch). > When there is an unclean leader election, it's possible for the new leader > to not to include certain epochs in its epoch cache. How are remote > segments associated with those epochs being cleaned? > > > > That is a good point. This leader will also cleanup the epochs earlier to > its start leader epoch and delete those segments. It gets the earliest > epoch for a partition and starts deleting segments from that leader epoch. > We need one more API in RLMM to get the earliest leader epoch. > > > > 601.5 The KIP discusses the handling of unclean leader elections for user > topics. What about unclean leader elections on > __remote_log_segment_metadata? > This is the same as other system topics like consumer_offsets, > __transaction_state topics. As discussed in the meeting, we will add the > behavior of __remote_log_segment_metadata topic’s unclean leader > truncation. > > > > 602. It would be useful to clarify the limitations in the initial release. > The KIP mentions not supporting compacted topics. What about JBOD and > changing the configuration of a topic from delete to compact after remote. > log. storage. enable ( http://remote.log.storage.enable/ ) is enabled? > > > > This was updated in the KIP earlier. > > > > 603. RLM leader tasks: > 603.1"It checks for rolled over LogSegments (which have the last message > offset less than last stable offset of that topic partition) and copies > them along with their offset/time/transaction indexes and leader epoch > cache to the remote tier." It needs to copy the producer snapshot too. > > > > Right. It copies producer snapshots too as mentioned in LogSegmentData. > > > > 603.2 "Local logs are not cleaned up till those segments are copied > successfully to remote even though their retention time/size is reached" > This seems weird. If the tiering stops because the remote store is not > available, we don't want the local data to grow forever. > > > > It was clarified in the discussion that the comment was more about the > local storage goes beyond the log.retention. The above statement is about > local.log.retention but not for the complete log.retention. When it > reaches the
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
Updated the KIP with Meeting Notes section https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-MeetingNotes On Tue, Aug 25, 2020 at 1:03 PM Jun Rao wrote: > Hi, Harsha, > > Thanks for the summary. Could you add the summary and the recording link to > the last section of > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > ? > > Jun > > On Tue, Aug 25, 2020 at 11:12 AM Harsha Chintalapani > wrote: > > > Thanks everyone for attending the meeting today. > > Here is the recording > > > > > https://drive.google.com/file/d/14PRM7U0OopOOrJR197VlqvRX5SXNtmKj/view?usp=sharing > > > > Notes: > > > >1. KIP is updated with follower fetch protocol and ready to reviewed > >2. Satish to capture schema of internal metadata topic in the KIP > >3. We will update the KIP with details of different cases > >4. Test plan will be captured in a doc and will add to the KIP > >5. Add a section "Limitations" to capture the capabilities that will > be > >introduced with this KIP and what will not be covered in this KIP. > > > > Please add to it I missed anything. Will produce a formal meeting notes > > from next meeting onwards. > > > > Thanks, > > Harsha > > > > > > > > On Mon, Aug 24, 2020 at 9:42 PM, Ying Zheng > > wrote: > > > > > We did some basic feature tests at Uber. The test cases and results are > > > shared in this google doc: > > > https://docs.google.com/spreadsheets/d/ > > > 1XhNJqjzwXvMCcAOhEH0sSXU6RTvyoSf93DHF-YMfGLk/edit?usp=sharing > > > > > > The performance test results were already shared in the KIP last month. > > > > > > On Mon, Aug 24, 2020 at 11:10 AM Harsha Ch > wrote: > > > > > > "Understand commitments towards driving design & implementation of the > > KIP > > > further and how it aligns with participant interests in contributing to > > the > > > efforts (ex: in the context of Uber’s Q3/Q4 roadmap)." What is that > > about? > > > > > > On Mon, Aug 24, 2020 at 11:05 AM Kowshik Prakasam < > > kpraka...@confluent.io> > > > wrote: > > > > > > Hi Harsha, > > > > > > The following google doc contains a proposal for temporary agenda for > the > > > KIP-405 <https://issues.apache.org/jira/browse/KIP-405> sync meeting > > > tomorrow: > > > > > > https://docs.google.com/document/d/ > > > 1pqo8X5LU8TpwfC_iqSuVPezhfCfhGkbGN2TqiPA3LBU/edit > > > > > > . > > > Please could you add it to the Google calendar invite? > > > > > > Thank you. > > > > > > Cheers, > > > Kowshik > > > > > > On Thu, Aug 20, 2020 at 10:58 AM Harsha Ch > wrote: > > > > > > Hi All, > > > > > > Scheduled a meeting for Tuesday 9am - 10am. I can record and upload for > > > community to be able to follow the discussion. > > > > > > Jun, please add the required folks on confluent side. > > > > > > Thanks, > > > > > > Harsha > > > > > > On Thu, Aug 20, 2020 at 12:33 AM, Alexandre Dupriez < > alexandre.dupriez@ > > > gmail.com > wrote: > > > > > > Hi Jun, > > > > > > Many thanks for your initiative. > > > > > > If you like, I am happy to attend at the time you suggested. > > > > > > Many thanks, > > > Alexandre > > > > > > Le mer. 19 août 2020 à 22:00, Harsha Ch < harsha. ch@ gmail. com ( > > harsha. > > > c...@gmail.com ) > a écrit : > > > > > > Hi Jun, > > > Thanks. This will help a lot. Tuesday will work for us. > > > -Harsha > > > > > > On Wed, Aug 19, 2020 at 1:24 PM Jun Rao < jun@ confluent. io ( jun@ > > > confluent.io ) > wrote: > > > > > > Hi, Satish, Ying, Harsha, > > > > > > Do you think it would be useful to have a regular virtual meeting to > > > discuss this KIP? The goal of the meeting will be sharing > > > design/development progress and discussing any open issues to > > > > > > accelerate > > > > > > this KIP. If so, will every Tuesday (from next week) 9am-10am > > > > > > PT > > > > > > work for you? I can help set up a Zoom meeting, invite everyone who > > > > > > might > > > > > >
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
Thanks everyone for attending the meeting today. Here is the recording https://drive.google.com/file/d/14PRM7U0OopOOrJR197VlqvRX5SXNtmKj/view?usp=sharing Notes: 1. KIP is updated with follower fetch protocol and ready to reviewed 2. Satish to capture schema of internal metadata topic in the KIP 3. We will update the KIP with details of different cases 4. Test plan will be captured in a doc and will add to the KIP 5. Add a section "Limitations" to capture the capabilities that will be introduced with this KIP and what will not be covered in this KIP. Please add to it I missed anything. Will produce a formal meeting notes from next meeting onwards. Thanks, Harsha On Mon, Aug 24, 2020 at 9:42 PM, Ying Zheng wrote: > We did some basic feature tests at Uber. The test cases and results are > shared in this google doc: > https://docs.google.com/spreadsheets/d/ > 1XhNJqjzwXvMCcAOhEH0sSXU6RTvyoSf93DHF-YMfGLk/edit?usp=sharing > > The performance test results were already shared in the KIP last month. > > On Mon, Aug 24, 2020 at 11:10 AM Harsha Ch wrote: > > "Understand commitments towards driving design & implementation of the KIP > further and how it aligns with participant interests in contributing to the > efforts (ex: in the context of Uber’s Q3/Q4 roadmap)." What is that about? > > On Mon, Aug 24, 2020 at 11:05 AM Kowshik Prakasam > wrote: > > Hi Harsha, > > The following google doc contains a proposal for temporary agenda for the > KIP-405 <https://issues.apache.org/jira/browse/KIP-405> sync meeting > tomorrow: > > https://docs.google.com/document/d/ > 1pqo8X5LU8TpwfC_iqSuVPezhfCfhGkbGN2TqiPA3LBU/edit > > . > Please could you add it to the Google calendar invite? > > Thank you. > > Cheers, > Kowshik > > On Thu, Aug 20, 2020 at 10:58 AM Harsha Ch wrote: > > Hi All, > > Scheduled a meeting for Tuesday 9am - 10am. I can record and upload for > community to be able to follow the discussion. > > Jun, please add the required folks on confluent side. > > Thanks, > > Harsha > > On Thu, Aug 20, 2020 at 12:33 AM, Alexandre Dupriez < alexandre.dupriez@ > gmail.com > wrote: > > Hi Jun, > > Many thanks for your initiative. > > If you like, I am happy to attend at the time you suggested. > > Many thanks, > Alexandre > > Le mer. 19 août 2020 à 22:00, Harsha Ch < harsha. ch@ gmail. com ( harsha. > c...@gmail.com ) > a écrit : > > Hi Jun, > Thanks. This will help a lot. Tuesday will work for us. > -Harsha > > On Wed, Aug 19, 2020 at 1:24 PM Jun Rao < jun@ confluent. io ( jun@ > confluent.io ) > wrote: > > Hi, Satish, Ying, Harsha, > > Do you think it would be useful to have a regular virtual meeting to > discuss this KIP? The goal of the meeting will be sharing > design/development progress and discussing any open issues to > > accelerate > > this KIP. If so, will every Tuesday (from next week) 9am-10am > > PT > > work for you? I can help set up a Zoom meeting, invite everyone who > > might > > be interested, have it recorded and shared, etc. > > Thanks, > > Jun > > On Tue, Aug 18, 2020 at 11:01 AM Satish Duggana < > > satish. duggana@ gmail. com ( satish.dugg...@gmail.com ) > > > wrote: > > Hi Kowshik, > > Thanks for looking into the KIP and sending your comments. > > 5001. Under the section "Follower fetch protocol in detail", the > next-local-offset is the offset upto which the segments are copied > > to > > remote storage. Instead, would last-tiered-offset be a better name > > than > > next-local-offset? last-tiered-offset seems to naturally align well > > with > > the definition provided in the KIP. > > Both next-local-offset and local-log-start-offset were introduced > > to > > talk > > about offsets related to local log. We are fine with > > last-tiered-offset > > too as you suggested. > > 5002. After leadership is established for a partition, the leader > > would > > begin uploading a segment to remote storage. If successful, the > > leader > > would write the updated RemoteLogSegmentMetadata to the metadata > > topic > > (via > > RLMM.putRemoteLogSegmentData). However, for defensive reasons, it > > seems > > useful that before the first time the segment is uploaded by the > > leader > > for > > a partition, the leader should ensure to catch up to all the > > metadata > > events written so far in the metadata topic for that partition (ex: > > by > > previous leader). To achieve this, the leader could start a lease > > (using > > an
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
"Understand commitments towards driving design & implementation of the KIP further and how it aligns with participant interests in contributing to the efforts (ex: in the context of Uber’s Q3/Q4 roadmap)." What is that about? On Mon, Aug 24, 2020 at 11:05 AM Kowshik Prakasam wrote: > Hi Harsha, > > The following google doc contains a proposal for temporary agenda for the > KIP-405 <https://issues.apache.org/jira/browse/KIP-405> sync meeting > tomorrow: > https://docs.google.com/document/d/1pqo8X5LU8TpwfC_iqSuVPezhfCfhGkbGN2TqiPA3LBU/edit > . > Please could you add it to the Google calendar invite? > > Thank you. > > > Cheers, > Kowshik > > On Thu, Aug 20, 2020 at 10:58 AM Harsha Ch wrote: > >> Hi All, >> >> Scheduled a meeting for Tuesday 9am - 10am. I can record and upload for >> community to be able to follow the discussion. >> >> Jun, please add the required folks on confluent side. >> >> Thanks, >> >> Harsha >> >> On Thu, Aug 20, 2020 at 12:33 AM, Alexandre Dupriez < >> alexandre.dupr...@gmail.com > wrote: >> >> > >> > >> > >> > Hi Jun, >> > >> > >> > >> > Many thanks for your initiative. >> > >> > >> > >> > If you like, I am happy to attend at the time you suggested. >> > >> > >> > >> > Many thanks, >> > Alexandre >> > >> > >> > >> > Le mer. 19 août 2020 à 22:00, Harsha Ch < harsha. ch@ gmail. com ( >> > harsha...@gmail.com ) > a écrit : >> > >> > >> >> >> >> >> >> Hi Jun, >> >> Thanks. This will help a lot. Tuesday will work for us. >> >> -Harsha >> >> >> >> >> >> >> >> On Wed, Aug 19, 2020 at 1:24 PM Jun Rao < jun@ confluent. io ( >> >> j...@confluent.io ) > wrote: >> >> >> >> >> >>> >> >>> >> >>> Hi, Satish, Ying, Harsha, >> >>> >> >>> >> >>> >> >>> Do you think it would be useful to have a regular virtual meeting to >> >>> discuss this KIP? The goal of the meeting will be sharing >> >>> design/development progress and discussing any open issues to >> accelerate >> >>> this KIP. If so, will every Tuesday (from next week) 9am-10am >> >>> >> >>> >> >> >> >> >> >> >> >> PT >> >> >> >> >> >>> >> >>> >> >>> work for you? I can help set up a Zoom meeting, invite everyone who >> might >> >>> be interested, have it recorded and shared, etc. >> >>> >> >>> >> >>> >> >>> Thanks, >> >>> >> >>> >> >>> >> >>> Jun >> >>> >> >>> >> >>> >> >>> On Tue, Aug 18, 2020 at 11:01 AM Satish Duggana < >> >>> >> >>> >> >> >> >> >> >> >> >> satish. duggana@ gmail. com ( satish.dugg...@gmail.com ) > >> >> >> >> >> >>> >> >>> >> >>> wrote: >> >>> >> >>> >> >>>> >> >>>> >> >>>> Hi Kowshik, >> >>>> >> >>>> >> >>>> >> >>>> Thanks for looking into the KIP and sending your comments. >> >>>> >> >>>> >> >>>> >> >>>> 5001. Under the section "Follower fetch protocol in detail", the >> >>>> next-local-offset is the offset upto which the segments are copied to >> >>>> remote storage. Instead, would last-tiered-offset be a better name >> than >> >>>> next-local-offset? last-tiered-offset seems to naturally align well >> >>>> >> >>>> >> >>> >> >>> >> >> >> >> >> >> >> >> with >> >> >> >> >> >>> >> >>>> >> >>>> >> >>>> the definition provided in the KIP. >> >>>> >> >>>> >> >>>> >> >>>> Both next-local-offset and local-log-start-offset were introduced to >> talk >> >>>
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
Hi Jun, Thanks. This will help a lot. Tuesday will work for us. -Harsha On Wed, Aug 19, 2020 at 1:24 PM Jun Rao wrote: > Hi, Satish, Ying, Harsha, > > Do you think it would be useful to have a regular virtual meeting to > discuss this KIP? The goal of the meeting will be sharing > design/development progress and discussing any open issues to > accelerate this KIP. If so, will every Tuesday (from next week) 9am-10am PT > work for you? I can help set up a Zoom meeting, invite everyone who might > be interested, have it recorded and shared, etc. > > Thanks, > > Jun > > On Tue, Aug 18, 2020 at 11:01 AM Satish Duggana > wrote: > > > Hi Kowshik, > > > > Thanks for looking into the KIP and sending your comments. > > > > 5001. Under the section "Follower fetch protocol in detail", the > > next-local-offset is the offset upto which the segments are copied to > > remote storage. Instead, would last-tiered-offset be a better name than > > next-local-offset? last-tiered-offset seems to naturally align well with > > the definition provided in the KIP. > > > > Both next-local-offset and local-log-start-offset were introduced to > > talk about offsets related to local log. We are fine with > > last-tiered-offset too as you suggested. > > > > 5002. After leadership is established for a partition, the leader would > > begin uploading a segment to remote storage. If successful, the leader > > would write the updated RemoteLogSegmentMetadata to the metadata topic > (via > > RLMM.putRemoteLogSegmentData). However, for defensive reasons, it seems > > useful that before the first time the segment is uploaded by the leader > for > > a partition, the leader should ensure to catch up to all the metadata > > events written so far in the metadata topic for that partition (ex: by > > previous leader). To achieve this, the leader could start a lease (using > an > > establish_leader metadata event) before commencing tiering, and wait > until > > the event is read back. For example, this seems useful to avoid cases > where > > zombie leaders can be active for the same partition. This can also prove > > useful to help avoid making decisions on which segments to be uploaded > for > > a partition, until the current leader has caught up to a complete view of > > all segments uploaded for the partition so far (otherwise this may cause > > same segment being uploaded twice -- once by the previous leader and then > > by the new leader). > > > > We allow copying segments to remote storage which may have common > > offsets. Please go through the KIP to understand the follower fetch > > protocol(1) and follower to leader transition(2). > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-FollowerReplication > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-Followertoleadertransition > > > > > > 5003. There is a natural interleaving between uploading a segment to > remote > > store, and, writing a metadata event for the same (via > > RLMM.putRemoteLogSegmentData). There can be cases where a remote segment > is > > uploaded, then the leader fails and a corresponding metadata event never > > gets written. In such cases, the orphaned remote segment has to be > > eventually deleted (since there is no confirmation of the upload). To > > handle this, we could use 2 separate metadata events viz. copy_initiated > > and copy_completed, so that copy_initiated events that don't have a > > corresponding copy_completed event can be treated as garbage and deleted > > from the remote object store by the broker. > > > > We are already updating RMM with RemoteLogSegmentMetadata pre and post > > copying of log segments. We had a flag in RemoteLogSegmentMetadata > > whether it is copied or not. But we are making changes in > > RemoteLogSegmentMetadata to introduce a state field in > > RemoteLogSegmentMetadata which will have the respective started and > > finished states. This includes for other operations like delete too. > > > > 5004. In the default implementation of RLMM (using the internal topic > > __remote_log_metadata), a separate topic called > > __remote_segments_to_be_deleted is going to be used just to track > failures > > in removing remote log segments. A separate topic (effectively another > > metadata stream) introduces some maintenance overhead and design > > complexity. It seems to me that the same can be achieved jus
Re: [VOTE] KIP-651 - Support PEM format for SSL certificates and private key
+1 binding. Thanks, Harsha On Sat, Aug 8, 2020 at 2:07 AM Manikumar wrote: > +1 (binding) > > Thanks for the KIP. > > > On Fri, Aug 7, 2020 at 12:56 AM David Jacot wrote: > > > Supporting PEM is really nice. Thanks, Rajini. > > > > +1 (non-binding) > > > > On Thu, Aug 6, 2020 at 9:18 PM Gwen Shapira wrote: > > > > > +1 (binding) > > > Thank you for driving this, Rajini > > > > > > On Thu, Aug 6, 2020 at 10:43 AM Rajini Sivaram < > rajinisiva...@gmail.com> > > > wrote: > > > > > > > Hi all, > > > > > > > > I would like to start vote on KIP-651 to support SSL key stores and > > trust > > > > stores in PEM format: > > > > > > > >- > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+certificates+and+private+key > > > > > > > > > > > > Thank you... > > > > > > > > Regards, > > > > > > > > Rajini > > > > > > > > > > > > > -- > > > Gwen Shapira > > > Engineering Manager | Confluent > > > 650.450.2760 | @gwenshap > > > Follow us: Twitter | blog > > > > > >
Re: [VOTE] KIP-578: Add configuration to limit number of partitions
Thanks for the updates Gokul. +1 (binding). Thanks, Harsha On Wed, Aug 05, 2020 at 8:18 AM, Gokul Ramanan Subramanian < gokul24...@gmail.com > wrote: > > > > Hi. > > > > I request the community to kindly resume the voting process for KIP-578. > If you have further comments, we can address those as well. > > > > Thanks and cheers. > > > > On Tue, Jul 21, 2020 at 2:07 PM Gokul Ramanan Subramanian < gokul2411s@ gmail. > com ( gokul24...@gmail.com ) > wrote: > > >> >> >> Hi. >> >> >> >> Can we resume the voting process for KIP-578? I have addressed additional >> comments by Boyang and Ismael. >> >> >> >> Thanks. >> >> >> >> On Mon, Jun 8, 2020 at 9:09 AM Gokul Ramanan Subramanian < gokul2411s@ gmail. >> com ( gokul24...@gmail.com ) > wrote: >> >> >>> >>> >>> Hi. Can we resume the voting process for KIP-578? Thanks. >>> >>> >>> >>> On Mon, Jun 1, 2020 at 11:09 AM Gokul Ramanan Subramanian < gokul2411s@ >>> gmail. >>> com ( gokul24...@gmail.com ) > wrote: >>> >>> >>>> >>>> >>>> Thanks Colin. Have updated the KIP per your recommendations. Let me know >>>> what you think. >>>> >>>> >>>> >>>> Thanks Harsha for the vote. >>>> >>>> >>>> >>>> On Wed, May 27, 2020 at 8:17 PM Colin McCabe < cmccabe@ apache. org ( >>>> cmcc...@apache.org ) > wrote: >>>> >>>> >>>>> >>>>> >>>>> Hi Gokul Ramanan Subramanian, >>>>> >>>>> >>>>> >>>>> Thanks for the KIP. >>>>> >>>>> >>>>> >>>>> Can you please modify the KIP to remove the reference to the deprecated >>>>> --zookeeper flag? This is not how kafka-configs. sh ( >>>>> http://kafka-configs.sh/ ) is supposed to be used in new versions of >>>>> Kafka. >>>>> You get a warning message if you do use this deprecated flag. As described >>>>> in KIP-604, we are removing the --zookeeper flag in the Kafka 3.0 release. >>>>> It also causes problems when people use the deprecated access mode-- for >>>>> example, as you note in this KIP, it bypasses resource limits such as the >>>>> ones described here. >>>>> >>>>> >>>>> >>>>> Instead of WILL_EXCEED_PARTITION_LIMITS, how about RESOURCE_LIMIT_REACHED? >>>>> Then the error string can contain the detailed message about which >>>>> resource limit was hit (per broker limit, per cluster limit, whatever.) It >>>>> would also be good to spell out that CreateTopicsPolicy plugins can also >>>>> throw this exception, for consistency. >>>>> >>>>> >>>>> >>>>> I realize that 2 billion partitions seems like a very big number. However, >>>>> filesystems have had to transition to 64 bit inode numbers as time has >>>>> gone on. There doesn't seem to be any performance reason why this should >>>>> be a 31 bit number, so let's just make these configurations longs, not >>>>> ints. >>>>> >>>>> >>>>> >>>>> best, >>>>> Colin >>>>> >>>>> >>>>> >>>>> On Wed, May 27, 2020, at 09:48, Harsha Chintalapani wrote: >>>>> >>>>> >>>>>> >>>>>> >>>>>> Thanks for the KIP Gokul. This will be really useful for our use >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> cases as >>>>> >>>>> >>>>>> >>>>>> >>>>>> well. >>>>>> +1 (binding). >>>>>> >>>>>> >>>>>> >>>>>> -Harsha >>>>>> >>>>>> >>>>>> >>>>>> On Tue, May 26, 2020 at 12:33 AM, Gokul Ramanan Subramanian < >>>>>> gokul2411s@ gmail. >>>>>> com ( gokul24...@gmail.com ) > wrote: >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> Hi. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Any votes for this? >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, May 12, 2020 at 11:36 AM Gokul Ramanan Subramanian < >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> gokul2411s@ >>>>> >>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> gmail. com ( http://gmail.com/ ) > wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> >>>>>>> >>>>>>> I'd like to initialize voting on KIP-578: >>>>>>> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ ( >>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/ ) >>>>>>> KIP-578%3A+Add+configuration+to+limit+number+of+partitions >>>>>>> >>>>>>> . >>>>>>> >>>>>>> >>>>>>> >>>>>>> Got some good feedback from Stanislav Kozlovski, Alexandre Dupriez >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> and Tom >>>>> >>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> Bentley on the discussion thread. I have addressed their comments. >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> I want >>>>> >>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> to thank them for their time. >>>>>>> >>>>>>> >>>>>>> >>>>>>> If there are any more concerns about the KIP, I am happy to discuss >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> them >>>>> >>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> further. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > > >
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
Hi Colin, Thats not what we said in the previous email. RLMM is pluggable storage and by running numbers even 1PB data you do not need more than 10GB local storage. If in future this becomes a blocker for any users we can revisit but this does not warrant another implementation at this point to push the data to remote storage. We can ofcourse implement another RLMM that is optional for users to configure to push to remote. But that doesn't need to be addressed in this KIP. Thanks, Harsha On Wed, Jul 15, 2020 at 5:50 PM Colin McCabe wrote: > Hi Ying, > > Thanks for the response. > > It sounds like you agree that storing the metadata in the remote storage > would be a better design overall. Given that that's true, is there any > reason to include the worse implementation based on RocksDB? > > Choosing a long-term metadata store is not something that we should do > lightly. It can take users years to migrate from metadata store to the > other. I also don't think it's realistic or desirable for users to write > their own metadata stores. Even assuming that they could do a good job at > this, it would create huge fragmentation in the Kafka ecosystem. > > best, > Colin > > > On Tue, Jul 14, 2020, at 09:39, Ying Zheng wrote: > > Hi Jun, > > Hi Colin, > > > > Satish and I are still discussing some details about how to handle > > transactions / producer ids. Satish is going to make some minor changes > to > > RLMM API and other parts. Other than that, we have finished updating the > KIP > > > > I agree with Colin that the current design of using rocksDB is not > > optimal. But this design is simple and should work for almost all the > > existing Kafka users. RLMM is a plugin. Users can replace rocksDB with > > their own RLMM implementation, if needed. So, I think we can keep rocksDB > > for now. What do you think? > > > > > > Thanks, > > Ying > > > > > > > > On Tue, Jul 7, 2020 at 10:35 AM Jun Rao wrote: > > > > > Hi, Ying, > > > > > > Thanks for the update. It's good to see the progress on this. Please > let us > > > know when you are done updating the KIP wiki. > > > > > > Jun > > > > > > On Tue, Jul 7, 2020 at 10:13 AM Ying Zheng > wrote: > > > > > > > Hi Jun, > > > > > > > > Satish and I have added more design details in the KIP, including > how to > > > > keep consistency between replicas (especially when there is > leadership > > > > changes / log truncations) and new metrics. We also made some other > minor > > > > changes in the doc. We will finish the KIP changes in the next > couple of > > > > days. We will let you know when we are done. Most of the changes are > > > > already updated to the wiki KIP. You can take a look. But it's not > the > > > > final version yet. > > > > > > > > As for the implementation, the code is mostly done and we already had > > > some > > > > feature tests / system tests. I have added the performance test > results > > > in > > > > the KIP. However the recent design changes (e.g. leader epoch info > > > > management / log truncation / some of the new metrics) have not been > > > > implemented yet. It will take about 2 weeks for us to implement > after you > > > > review and agree with those design changes. > > > > > > > > > > > > > > > > On Tue, Jul 7, 2020 at 9:23 AM Jun Rao wrote: > > > > > > > > > Hi, Satish, Harsha, > > > > > > > > > > Any new updates on the KIP? This feature is one of the most > important > > > and > > > > > most requested features in Apache Kafka right now. It would be > helpful > > > if > > > > > we can make sustained progress on this. Could you share how far > along > > > is > > > > > the design/implementation right now? Is there anything that other > > > people > > > > > can help to get it across the line? > > > > > > > > > > As for "transactional support" and "follower > requests/replication", no > > > > > further comments from me as long as the producer state and leader > epoch > > > > can > > > > > be restored properly from the object store when needed. > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > On
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
Hi Jun, Thanks for the replies and feedback on design and giving input. We are coming close to finish the implementation. We also did several perf tests as well at our peak production loads and with tiered storage we didn't see any degradation on write throughputs and latencies. Ying already added some of the perf tests results in the KIP itself. It will be great if we can get design and code reviews from you and others in the community as we make progress. Thanks, Harsha On Tue, Jul 7, 2020 at 10:34 AM Jun Rao wrote: > Hi, Ying, > > Thanks for the update. It's good to see the progress on this. Please let > us know when you are done updating the KIP wiki. > > Jun > > On Tue, Jul 7, 2020 at 10:13 AM Ying Zheng wrote: > >> Hi Jun, >> >> Satish and I have added more design details in the KIP, including how to >> keep consistency between replicas (especially when there is leadership >> changes / log truncations) and new metrics. We also made some other minor >> changes in the doc. We will finish the KIP changes in the next couple of >> days. We will let you know when we are done. Most of the changes are >> already updated to the wiki KIP. You can take a look. But it's not the >> final version yet. >> >> As for the implementation, the code is mostly done and we already had some >> feature tests / system tests. I have added the performance test results in >> the KIP. However the recent design changes (e.g. leader epoch info >> management / log truncation / some of the new metrics) have not been >> implemented yet. It will take about 2 weeks for us to implement after you >> review and agree with those design changes. >> >> >> >> On Tue, Jul 7, 2020 at 9:23 AM Jun Rao wrote: >> >> > Hi, Satish, Harsha, >> > >> > Any new updates on the KIP? This feature is one of the most important >> and >> > most requested features in Apache Kafka right now. It would be helpful >> if >> > we can make sustained progress on this. Could you share how far along is >> > the design/implementation right now? Is there anything that other people >> > can help to get it across the line? >> > >> > As for "transactional support" and "follower requests/replication", no >> > further comments from me as long as the producer state and leader epoch >> can >> > be restored properly from the object store when needed. >> > >> > Thanks, >> > >> > Jun >> > >> > On Tue, Jun 9, 2020 at 3:39 AM Satish Duggana > > >> > wrote: >> > >> > > We did not want to add many implementation details in the KIP. But we >> > > decided to add them in the KIP as appendix or sub-sections(including >> > > follower fetch protocol) to describe the flow with the main cases. >> > > That will answer most of the queries. I will update on this mail >> > > thread when the respective sections are updated. >> > > >> > > Thanks, >> > > Satish. >> > > >> > > On Sat, Jun 6, 2020 at 7:49 PM Alexandre Dupriez >> > > wrote: >> > > > >> > > > Hi Satish, >> > > > >> > > > A couple of questions specific to the section "Follower >> > > > Requests/Replication", pages 16:17 in the design document [1]. >> > > > >> > > > 900. It is mentioned that followers fetch auxiliary states from the >> > > > remote storage. >> > > > >> > > > 900.a Does the consistency model of the external storage impacts >> reads >> > > > of leader epochs and other auxiliary data? >> > > > >> > > > 900.b What are the benefits of using a mechanism to store and access >> > > > the leader epochs which is different from other metadata associated >> to >> > > > tiered segments? What are the benefits of retrieving this >> information >> > > > on-demand from the follower rather than relying on propagation via >> the >> > > > topic __remote_log_metadata? What are the advantages over using a >> > > > dedicated control structure (e.g. a new record type) propagated via >> > > > this topic? Since in the document, different control paths are >> > > > operating in the system, how are the metadata stored in >> > > > __remote_log_metadata [which also include the epoch of the leader >> > > > which offloaded a segment] and the remote auxiliary states, kept in >>
Re: [VOTE] KIP-578: Add configuration to limit number of partitions
Thanks for the KIP Gokul. This will be really useful for our use cases as well. +1 (binding). -Harsha On Tue, May 26, 2020 at 12:33 AM, Gokul Ramanan Subramanian < gokul24...@gmail.com> wrote: > Hi. > > Any votes for this? > > Thanks. > > On Tue, May 12, 2020 at 11:36 AM Gokul Ramanan Subramanian < gokul2411s@ > gmail.com> wrote: > > Hello, > > I'd like to initialize voting on KIP-578: > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-578%3A+Add+configuration+to+limit+number+of+partitions > . > > Got some good feedback from Stanislav Kozlovski, Alexandre Dupriez and Tom > Bentley on the discussion thread. I have addressed their comments. I want > to thank them for their time. > > If there are any more concerns about the KIP, I am happy to discuss them > further. > > Thanks. > >
Re: [VOTE] KIP-602: Change default value for client.dns.lookup
+1 (binding). -Harsha On Fri, May 22, 2020 at 10:04 AM Ismael Juma wrote: > Thanks for the KIP, +1 (binding). > > Ismael > > On Fri, May 22, 2020 at 1:40 AM Badai Aqrandista > wrote: > > > Hi All, > > > > I would like to start the vote on KIP-602: Change default value for > > client.dns.lookup > > > > For reference, here is the KIP wiki: > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-602%3A+Change+default+value+for+client.dns.lookup > > > > And discussion thread: > > > > > > > https://lists.apache.org/thread.html/r0e70d3757267c4158f12c05a4e5ac9eb33f2d11ce99d5878b3b4b3f7%40%3Cdev.kafka.apache.org%3E > > > > -- > > Thanks, > > Badai > > >
Re: [VOTE] KIP-545 support automated consumer offset sync across clusters in MM 2.0
+1 (binding). Good addition to MM 2. -Harsha On Thu, May 21, 2020 at 8:36 AM, Manikumar < manikumar.re...@gmail.com > wrote: > > > > +1 (binding). > > > > Thanks for the KIP. > > > > On Thu, May 21, 2020 at 9:49 AM Maulin Vasavada < maulin. vasavada@ gmail. > com ( maulin.vasav...@gmail.com ) > wrote: > > >> >> >> Thank you for the KIP. I sincerely hope we get enough votes on this KIP. I >> was thinking of similar changes while working on DR capabilities and >> offsets are Achilles Heels and this KIP addresses it. >> >> >> >> On Mon, May 18, 2020 at 6:10 PM Maulin Vasavada < maulin. vasavada@ gmail. >> com ( maulin.vasav...@gmail.com ) >> >> >> >> wrote: >> >> >>> >>> >>> +1 (non-binding) >>> >>> >>> >>> On Mon, May 18, 2020 at 9:41 AM Ryanne Dolan < ryannedolan@ gmail. com ( >>> ryannedo...@gmail.com ) > wrote: >>> >>> >>>> >>>> >>>> Bump. Looks like we've got 6 non-binding votes and 1 binding. >>>> >>>> >>>> >>>> On Thu, Feb 20, 2020 at 11:25 AM Ning Zhang < ning2008wisc@ gmail. com ( >>>> ning2008w...@gmail.com ) > wrote: >>>> >>>> >>>>> >>>>> >>>>> Hello committers, >>>>> >>>>> >>>>> >>>>> I am the author of the KIP-545 and if we still miss votes from the >>>>> committers, please review the KIP and vote for it, so that the >>>>> corresponding PR will be reviewed soon. >>>>> >>>>> >>>> >>>> >>> >>> >> >> >> >> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ >> KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2. >> 0 ( >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0 >> ) >> >> >>> >>>> >>>>> >>>>> >>>>> Thank you >>>>> >>>>> >>>>> >>>>> On 2020/02/06 17:05:41, Edoardo Comar < edoardlists@ gmail. com ( >>>>> edoardli...@gmail.com ) > wrote: >>>>> >>>>> >>>>>> >>>>>> >>>>>> +1 (non-binding) >>>>>> thanks for the KIP ! >>>>>> >>>>>> >>>>>> >>>>>> On Tue, 14 Jan 2020 at 13:57, Navinder Brar < >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> >> >> navinder_brar@ yahoo. com ( navinder_b...@yahoo.com ) >> >> >>> >>>> >>>>> >>>>> >>>>> .invalid> >>>>> >>>>> >>>>>> >>>>>> >>>>>> wrote: >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> +1 (non-binding) >>>>>>> Navinder >>>>>>> On Tuesday, 14 January, 2020, 07:24:02 pm IST, Ryanne Dolan < >>>>>>> ryannedolan@ >>>>>>> gmail. com ( ryannedo...@gmail.com ) > wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> Bump. We've got 4 non-binding and one binding vote. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Ryanne >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Dec 13, 2019, 1:44 AM Tom Bentley < tbentley@ redhat. com ( >>>>>>> tbent...@redhat.com ) > >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> wrote: >>>> >>>> >>>>> >>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> +1 (non-binding) >>>>>>>> >>>>>>>> >>>>>>
Re: [VOTE] KIP-612: Ability to Limit Connection Creation Rate on Brokers
+1 (binding) On Mon, May 18 2020 at 4:54 PM, Anna Povzner < a...@confluent.io > wrote: > > > > Hi All, > > > > I would like to start the vote on KIP-612: Ability to limit connection > creation rate on brokers. > > > > For reference, here is the KIP wiki: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-612%3A+Ability+to+Limit+Connection+Creation+Rate+on+Brokers > > > > > And discussion thread: > https://lists.apache.org/thread.html/r61162661fa307d0bc5c8326818bf223a689c49e1c828c9928ee26969%40%3Cdev.kafka.apache.org%3E > > > > > Thanks, > > > > Anna > > >
Re: [VOTE] KIP-519: Make SSL context/engine configuration extensible
Thanks for putting the KIP together. This is going to help quite a bit in customizing SSL. +1 (binding). Thanks, Harsha On Mon, Mar 30, 2020 at 7:44 PM Maulin Vasavada wrote: > Thanks Jun Rao for your vote and comments. > > For 1) Earlier it was the security.ssl package but after a review I changed > it to .auth since there are some public interfaces in that package. I am > open to move it under .ssl package. > > For 2) Sure. Will document in Javadocs for the method. > > Thanks > Maulin > > On Mon, Mar 30, 2020 at 5:46 PM Jun Rao wrote: > > > Hi, Maulin, > > > > Thanks for the KIP. +1 from me. Just a couple of minor comments below. > > > > 1. Should the package name of the new > > interface SslEngineFactory be org.apache.kafka.common.security.ssl > instead > > of org.apache.kafka.common.security.auth? > > 2. Could you document when shouldBeRebuilt() will be called? > > > > Jun > > > > On Mon, Mar 30, 2020 at 5:07 PM Maulin Vasavada < > maulin.vasav...@gmail.com > > > > > wrote: > > > > > ^^^ bump ^^^ The vote is open for 2-3 days and gotten 1 Binding vote > so > > > far, can you please vote so that we can try to move forward with > changes? > > > > > > On Thu, Mar 26, 2020 at 4:11 PM Zhou, Thomas > > > > wrote: > > > > > > > +1 (non-binding) > > > > > > > > Regards, > > > > Thomas > > > > > > > > On 3/26/20, 12:36 PM, "Rajini Sivaram" > > wrote: > > > > > > > > +1 (binding) > > > > Thanks for the KIP, Maulin! > > > > > > > > Regards, > > > > > > > > Rajini > > > > > > > > On Thu, Mar 26, 2020 at 4:14 PM Maulin Vasavada < > > > > maulin.vasav...@gmail.com> > > > > wrote: > > > > > > > > > FYI - we have updated the KIP documentation also with > appropriate > > > > code > > > > > samples for interfaces and few important changes. > > > > > > > > > > Thanks > > > > > Maulin > > > > > > > > > > On Wed, Mar 25, 2020 at 10:21 AM Maulin Vasavada < > > > > > maulin.vasav...@gmail.com> > > > > > wrote: > > > > > > > > > > > bump > > > > > > > > > > > > On Wed, Mar 25, 2020 at 10:20 AM Maulin Vasavada < > > > > > > maulin.vasav...@gmail.com> wrote: > > > > > > > > > > > >> Hi all > > > > > >> > > > > > >> After much await on the approach conclusion we have a PR > > > > > >> > > > > > > > > > > https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Fpull%2F8338data=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=1ydk0OMaucb8QhTyyQ8Ua3ereGzcS4usRlavU1RixkE%3Dreserved=0 > > > > . > > > > > >> > > > > > >> Can you please provide your vote so that we can more this > > > forward? > > > > > >> > > > > > >> Thanks > > > > > >> Maulin > > > > > >> > > > > > >> On Sun, Jan 26, 2020 at 11:03 PM Maulin Vasavada < > > > > > >> maulin.vasav...@gmail.com> wrote: > > > > > >> > > > > > >>> Hi all > > > > > >>> > > > > > >>> After a good discussion on the KIP at > > > > > >>> > > > > > > > > > > https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mail-archive.com%2Fdev%40kafka.apache.org%2Fmsg101011.htmldata=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=qsvbqkoxL6NSPDV6rm9B9xqZG5xvYaZkj0cYrTM6bPw%3Dreserved=0 > > > > I > > > > > >>> think we are ready to start voting. > > > > > >>> > > > > > >>> KIP: > > > > > >>> > > > > > > > > > > > > > > > https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D128650952data=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=rcqWc2inIbrWlMj2jssHPKcMlHuDuLvicmYHHDYWrF8%3Dreserved=0 > > > > > >>> > > > > > >>> The KIP proposes - Making SSLEngine creation pluggable to > > > support > > > > > >>> customization of various security related aspects. > > > > > >>> > > > > > >>> Thanks > > > > > >>> Maulin > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
ct stores may support that already. For example, S3 supports > > events > > notification ( > > https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html). > > Otherwise one could use a separate metadata store that supports > > push-based > > change propagation. Other people have mentioned using a Kafka > > topic. The > > best approach may depend on the object store and the operational > environment (e.g. whether an external metadata store is already > > available). > > The above discussion is based on the assumption that we need to > > cache the > > object metadata locally in every broker. I mentioned earlier that > > an > > alternative is to just store/retrieve those metadata in an external > metadata store. That may simplify the implementation in some cases. > > Thanks, > > Jun > > On Thu, Dec 5, 2019 at 7:01 AM Satish Duggana < > > satish.dugg...@gmail.com> > > wrote: > > Hi Jun, > Thanks for your reply. > > Currently, `listRemoteSegments` is called at the configured interval(not > every second, defaults to 30secs). Storing remote > > log > > metadata in a strongly consistent store for S3 RSM is raised in > PR-comment[1]. > RLM invokes RSM at regular intervals and RSM can give remote > > segment > > metadata if it is available. RSM is responsible for maintaining > > and > > fetching those entries. It should be based on whatever mechanism > > is > > consistent and efficient with the respective remote storage. > > Can you give more details about push based mechanism from RSM? > > 1. > > https://github.com/apache/kafka/pull/7561#discussion_r344576223 > > Thanks, > Satish. > > On Thu, Dec 5, 2019 at 4:23 AM Jun Rao wrote: > > Hi, Harsha, > > Thanks for the reply. > > 40/41. I am curious which block storages you have tested. S3 > > seems > > to be > > one of the popular block stores. The concerns that I have with > > pull > > based > > approach are the following. > (a) Cost: S3 list object requests cost $0.005 per 1000 > > requests. If > > you > > have 100,000 partitions and want to pull the metadata for each > > partition > > at > > the rate of 1/sec. It can cost $0.5/sec, which is roughly $40K > > per > > day. > > (b) Semantics: S3 list objects are eventually consistent. So, > > when > > you > > do a > > list object request, there is no guarantee that you can see all > > uploaded > > objects. This could impact the correctness of subsequent > > logics. > > (c) Efficiency: Blindly pulling metadata when there is no > > change adds > > unnecessary overhead in the broker as well as in the block > > store. > > So, have you guys tested S3? If so, could you share your > > experience > > in > > terms of cost, semantics and efficiency? > > Jun > > On Tue, Dec 3, 2019 at 10:11 PM Harsha Chintalapani < > > ka...@harsha.io > > wrote: > > Hi Jun, > Thanks for the reply. > > On Tue, Nov 26, 2019 at 3:46 PM, Jun Rao > > wrote: > > Hi, Satish and Ying, > > Thanks for the reply. > > 40/41. There are two different ways that we can approach > > this. > > One is > > what > > you said. We can have an opinionated way of storing and > > populating > > the > > tier > > metadata that we think is good enough for everyone. I am > > not > > sure if > > this > > is the case based on what's currently proposed in the KIP. > > For > > example, I > > am not sure that (1) everyone always needs local metadata; > > (2) > > the > > current > > local storage format is general enough and (3) everyone > > wants to > > use > > the > > pull based approach to propagate the metadata. Another > > approach > > is to > > make > > this pluggable and let the implementor implements the best > > approach > > for a > > particular block storage. I haven't seen any comments from > > Slack/AirBnb > > in > > the mailing list on this topic. It would be great if they > > can > > provide > > feedback directly here. > > The current interfaces are designed with most popular block > > storages > > available today and we did 2 implementations with these > > interfaces and > > they both are yielding good results as we going through the > > testing of > > it. > > If there is ever a need for pull based approach we can > > definitely > > evolve > > the interface. > In the past we did mark int
[jira] [Created] (KAFKA-9578) Kafka Tiered Storage - System Tests
Harsha created KAFKA-9578: - Summary: Kafka Tiered Storage - System Tests Key: KAFKA-9578 URL: https://issues.apache.org/jira/browse/KAFKA-9578 Project: Kafka Issue Type: Sub-task Reporter: Harsha Assignee: Alexandre Dupriez Initial test cases set up by [~Ying Zheng] [https://docs.google.com/spreadsheets/d/1gS0s1FOmcjpKYXBddejXAoJAjEZ7AdEzMU9wZc-JgY8/edit#gid=0] -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Possible to create Scrum board under Kafka project in JIRA?
All the dependency JIRAs are created and linked to the EPIC here https://issues.apache.org/jira/browse/KAFKA-7739 . Lets drive it through that. -Harsha On Fri, Feb 14, 2020 at 2:28 AM, Alexandre Dupriez < alexandre.dupr...@gmail.com> wrote: > Good morning, > > Would it be possible to allow the the Apache Kafka project in JIRA to be > included in a new Scrum board? > > I can see there is already a Kanban board for Cloudera and tried to create > a Scrum board for Tiered-Storage but don't have the permissions to include > Apache Kafka. > > Thank you, > Alexandre >
Re: [DISCUSS] KIP-501 Avoid out-of-sync or offline partitions when follower fetch requests not processed in time
Hi Lucas, Yes the case you mentioned is true. I do understand KIP-501 might not fully solve this particular use case where there might blocked fetch requests. But the issue we noticed multiple times and continue to notice is 1. Fetch request comes from Follower 2. Leader tries to fetch data from disk which takes longer than replica.lag.time.max.ms 3. Async thread on leader side which checks the ISR marks the follower who sent a fetch request as not in ISR 4. Leader dies during this request due to disk errors and now we have offline partitions because Leader kicked out healthy followers out of ISR Instead of considering this from a disk issue. Lets look at how we maintain the ISR 1. Currently we do not consider a follower as healthy even when its able to send fetch requests 2. ISR is controlled on how healthy a broker is, ie if it takes longer than replica.lag.time.max.ms we mark followers out of sync instead of relinquishing the leadership. What we are proposing in this KIP, we should look at the time when a follower sends a fetch request and keep that as basis for marking a follower out of ISR or to keep it in the ISR and leave the disk read time on leader side out of this. Thanks, Harsha On Mon, Feb 10, 2020 at 9:26 PM, Lucas Bradstreet wrote: > Hi Harsha, > > Is the problem you'd like addressed the following? > > Assume 3 replicas, L and F1 and F2. > > 1. F1 and F2 are alive and sending fetch requests to L. > 2. L starts encountering disk issues, any requests being processed by the > request handler threads become blocked. > 3. L's zookeeper connection is still alive so it remains the leader for > the partition. > 4. Given that F1 and F2 have not successfully fetched, L shrinks the ISR > to itself. > > While KIP-501 may help prevent a shrink in partitions where a replica > fetch request has started processing, any fetch requests in the request > queue will have no effect. Generally when these slow/failing disk issues > occur, all of the request handler threads end up blocked and requests queue > up in the request queue. For example, all of the request handler threads > may end up stuck in > KafkaApis.handleProduceRequest handling produce requests, at which point > all of the replica fetcher fetch requests remain queued in the request > queue. If this happens, there will be no tracked fetch requests to prevent > a shrink. > > Solving this shrinking issue is tricky. It would be better if L resigns > leadership when it enters a degraded state rather than avoiding a shrink. > If L is no longer the leader in this situation, it will eventually become > blocked fetching from the new leader and the new leader will shrink the > ISR, kicking out L. > > Cheers, > > Lucas >
Re: [DISCUSS] KIP-501 Avoid out-of-sync or offline partitions when follower fetch requests not processed in time
Hi Jason & Jun, Do you have any feedback on the KIP or is it ok take it to voting?. Its good to have this config in Kafka to address disk failure scenarios as described in the KIP. Thanks, Harsha On Mon, Feb 10, 2020 at 5:10 PM, Brian Sang < bais...@yelp.com.invalid > wrote: > > > > Hi, > > > > Just wanted to bump this discussion, since it happened to us again at Yelp > > > > > It's particularly nasty since it can happen right before a disk failure, > so right as the leader for the partition becomes the only ISR, the leader > becomes unrecoverable right after, forcing us to do an unclean leader > election to resolve the situation. Having offline partitions due to a > single failure is really annoying. I'm curious if others have experienced > this as well, but weren't able to trace it to this specific error. > > > > Best, > Brian > > > > On 2020/01/22 03:28:34, Satish Duggana < satish. duggana@ gmail. com ( > satish.dugg...@gmail.com ) > wrote: > > >> >> >> Hi Jun, >> Can you please review the KIP and let us know your comments? >> >> >> >> If there are no comments/questions, we can start a vote thread. >> >> >> >> It looks like Yelp folks also encountered the same issue as mentioned in >> JIRA comment[1]. >> >> >>> >>>> >>>> >>>> Flavien Raynaud added a comment - Yesterday >>>> >>>> >>> >>> >> >> >> >> We've seen offline partitions happening for the same reason in one of our >> clusters too, where only the broker leader for the offline partitions was >> having disk issues. It looks like there has not been much progress/look on >> the PR submitted since December 9th. Is there anything blocking this >> change from moving forward? >> >> >> >> 1. https:/ / issues. apache. org/ jira/ browse/ >> KAFKA-8733?focusedCommentId=17020083=com. >> atlassian. jira. plugin. system. >> issuetabpanels%3Acomment-tabpanel#comment-17020083 >> ( >> https://issues.apache.org/jira/browse/KAFKA-8733?focusedCommentId=17020083=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17020083 >> ) >> >> >> >> Thanks, >> Satish. >> >> >> >> On Thu, Dec 5, 2019 at 10:38 AM Harsha Chintalapani < kafka@ harsha. io ( >> ka...@harsha.io ) > wrote: >> >> >>> >>> >>> Hi Jason, >>> As Satish said just increase replica max lag will not work in this case. >>> Just before a disk dies the reads becomes really slow and its hard to >>> estimate how much this is, as we noticed range is pretty wide. Overall it >>> doesn't make sense to knock good replicas out of just because a leader is >>> slower in processing reads or serving the fetch requests which may be due >>> to disk issues in this case but could be other issues as well. I think >>> this kip addresses in general all of these issues. >>> Do you still have questions on the current approach if not we can take it >>> vote. >>> Thanks, >>> Harsha >>> >>> >>> >>> On Mon, Nov 18, 2019 at 7:05 PM, Satish Duggana < satish. duggana@ gmail. >>> com >>> ( satish.dugg...@gmail.com ) > wrote: >>> >>> >>>> >>>> >>>> Hi Jason, >>>> Thanks for looking into the KIP. Apologies for my late reply. Increasing >>>> replica max lag to 30-45 secs did not help as we observed that a few fetch >>>> requests took more than 1-2 minutes. We do not want to increase further as >>>> it increases upper bound on commit latency. We have strict SLAs on some of >>>> the clusters on end to end(producer to consumer) latency. This proposal >>>> improves the availability of partitions when followers are trying their >>>> best to be insync even when leaders are slow in processing those requests. >>>> I have updated the KIP to have a single config for giving backward >>>> compatibility and I guess this config is more comprehensible than earlier. >>>> But I believe there is no need to have config because the suggested >>>> proposal in the KIP is an enhancement to the existing behavior. Please let >>>> me know your comments. >>>> >>>> >>>> >>>> Thanks, >>>> Satish. >>>> >>>> >>>> >>>&g
Re: [VOTE] KIP-559: Make the Kafka Protocol Friendlier with L7 Proxies
+1 ( binding). Much needed! -Harsha On Thu, Jan 23, 2020 at 7:17 PM, Guozhang Wang wrote: > +1 （binding） > > On Thu, Jan 23, 2020 at 1:55 PM Guozhang Wang wrote: > > Yeah that makes sense, it is a good-to-have if we can push through this in > 2.5 but if we do not have bandwidth that's fine too :) > > Guozhang > > On Thu, Jan 23, 2020 at 1:40 PM David Jacot wrote: > > Hi Guozhang, > > Thank you for your input. > > 1) You're right. I've put it there due to the version bump only. I'll make > it clearer. > > 2) I'd rather prefer to keep the scope as it is because 1) that field is > not related to > the problem that we are solving here and 2) I am not sure that I will have > the > bandwidth to do this before the feature freeze. The PR is already ready. > That being > said, as the addition of that field is part of KIP-429 and KIP-429 has > already been > accepted, we could give it a shot to avoid having to bump the version > twice. I could > try putting together a PR before the feature freeze but without guarantee. > Does that > make sense? > > David > > On Thu, Jan 23, 2020 at 9:44 AM Guozhang Wang wrote: > > Hello David, > > Thanks for the KIP! I have read through the proposal and had one minor > > and > > one meta comment. But overall it looks good to me! > > 1) The JoinGroupRequest format does not have any new fields proposed, > > so we > > could either clarify that it is listed here but without modifications > > (only > > version bumps) or just remove it from the wiki. > > 2) Could we consider adding a "protocol version" to allow brokers to > > select > > the leader with the highest version? This thought is brought up in > > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol#KIP-429:KafkaConsumerIncrementalRebalanceProtocol-LookingintotheFuture:AssignorVersion > > . > I'm fine with keeping this KIP's scope as is, just wondering if you feel > comfortable piggy-backing this change as well if we are going to bump up > the JoinGroupReq/Response anyways. > > Guozhang > > On Wed, Jan 22, 2020 at 9:10 AM Eno Thereska > wrote: > > This is awesome! +1 (non binding) > Eno > > On Tue, Jan 21, 2020 at 10:00 PM Gwen Shapira > > wrote: > > Thank you for the KIP. Awesomely cloud-native improvement :) > > +1 (binding) > > On Tue, Jan 21, 2020, 9:35 AM David Jacot > > wrote: > > Hi all, > > I would like to start a vote on KIP-559: Make the Kafka Protocol > > Friendlier > > with L7 Proxies. > > The KIP is here: > > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-559%3A+Make+the+Kafka+Protocol+Friendlier+with+L7+Proxies > > Thanks, > David > > -- > -- Guozhang > > -- > -- Guozhang > > -- > -- Guozhang >
Re: [DISCUSS] KIP-501 Avoid out-of-sync or offline partitions when follower fetch requests not processed in time
Hi Jason, As Satish said just increase replica max lag will not work in this case. Just before a disk dies the reads becomes really slow and its hard to estimate how much this is, as we noticed range is pretty wide. Overall it doesn't make sense to knock good replicas out of just because a leader is slower in processing reads or serving the fetch requests which may be due to disk issues in this case but could be other issues as well. I think this kip addresses in general all of these issues. Do you still have questions on the current approach if not we can take it vote. Thanks, Harsha On Mon, Nov 18, 2019 at 7:05 PM, Satish Duggana wrote: > Hi Jason, > Thanks for looking into the KIP. Apologies for my late reply. Increasing > replica max lag to 30-45 secs did not help as we observed that a few fetch > requests took more than 1-2 minutes. We do not want to increase further as > it increases upper bound on commit latency. We have strict SLAs on some of > the clusters on end to end(producer to consumer) latency. This proposal > improves the availability of partitions when followers are trying their > best to be insync even when leaders are slow in processing those requests. > I have updated the KIP to have a single config for giving backward > compatibility and I guess this config is more comprehensible than earlier. > But I believe there is no need to have config because the suggested > proposal in the KIP is an enhancement to the existing behavior. Please let > me know your comments. > > Thanks, > Satish. > > On Thu, Nov 14, 2019 at 10:57 AM Jason Gustafson > wrote: > > Hi Satish, > > Thanks for the KIP. I'm wondering how much of this problem can be > addressed just by increasing the replication max lag? That was one of the > purposes of KIP-537 (the default increased from 10s to 30s). Also, the new > configurations seem quite low level. I think they will be hard for users to > understand (even reading through a couple times I'm not sure I understand > them fully). I think if there's a way to improve this behavior without > requiring any new configurations, it would be much more attractive. > > Best, > Jason > > On Wed, Nov 6, 2019 at 8:14 AM Satish Duggana > wrote: > > Hi Dhruvil, > Thanks for looking into the KIP. > > 10. I have an initial sketch of the KIP-500 in commit[a] which discusses > tracking the pending fetch requests. Tracking is not done in > Partition#readRecords because if it takes longer in reading any of the > partitions then we do not want any of the replicas of this fetch request to > go out of sync. > > 11. I think `Replica` class should be thread-safe to handle the remote > scenario of concurrent requests running for a follower replica. Or I may be > missing something here. This is a separate issue from KIP-500. I will file > a separate JIRA to discuss that issue. > > a - > https://github.com/satishd/kafka/commit/ > c69b525abe8f6aad5059236076a003cdec4c4eb7 > > Thanks, > Satish. > > On Tue, Oct 29, 2019 at 10:57 AM Dhruvil Shah > wrote: > > Hi Satish, > > Thanks for the KIP, those seems very useful. Could you elaborate on how > pending fetch requests are tracked? > > Thanks, > Dhruvil > > On Mon, Oct 28, 2019 at 9:43 PM Satish Duggana > wrote: > > Hi All, > I wrote a short KIP about avoiding out-of-sync or offline partitions when > follower fetch requests are not processed in time by the leader replica. > KIP-501 is located at https://s.apache.org/jhbpn > > Please take a look, I would like to hear your feedback and suggestions. > > JIRA: https://issues.apache.org/jira/browse/KAFKA-8733 > > Thanks, > Satish. > >
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
Hi Jun, Thanks for the reply. On Tue, Nov 26, 2019 at 3:46 PM, Jun Rao wrote: > Hi, Satish and Ying, > > Thanks for the reply. > > 40/41. There are two different ways that we can approach this. One is what > you said. We can have an opinionated way of storing and populating the tier > metadata that we think is good enough for everyone. I am not sure if this > is the case based on what's currently proposed in the KIP. For example, I > am not sure that (1) everyone always needs local metadata; (2) the current > local storage format is general enough and (3) everyone wants to use the > pull based approach to propagate the metadata. Another approach is to make > this pluggable and let the implementor implements the best approach for a > particular block storage. I haven't seen any comments from Slack/AirBnb in > the mailing list on this topic. It would be great if they can provide > feedback directly here. > The current interfaces are designed with most popular block storages available today and we did 2 implementations with these interfaces and they both are yielding good results as we going through the testing of it. If there is ever a need for pull based approach we can definitely evolve the interface. In the past we did mark interfaces to be evolving to make room for unknowns in the future. If you have any suggestions around the current interfaces please propose we are happy to see if we can work them into it. 43. To offer tier storage as a general feature, ideally all existing > capabilities should still be supported. It's fine if the uber > implementation doesn't support all capabilities for internal usage. > However, the framework should be general enough. > We agree on that as a principle. But all of these major features mostly coming right now and to have a new big feature such as tiered storage to support all the new features will be a big ask. We can document on how do we approach solving these in future iterations. Our goal is to make this tiered storage feature work for everyone. 43.3 This is more than just serving the tier-ed data from block storage. > With KIP-392, the consumer now can resolve the conflicts with the replica > based on leader epoch. So, we need to make sure that leader epoch can be > recovered properly from tier storage. > We are working on testing our approach and we will update the KIP with design details. 43.4 For JBOD, if tier storage stores the tier metadata locally, we need to > support moving such metadata across disk directories since JBOD supports > moving data across disks. > KIP is updated with JBOD details. Having said that JBOD tooling needs to evolve to support production loads. Most of the users will be interested in using tiered storage without JBOD support support on day 1. Thanks, Harsha As for meeting, we could have a KIP e-meeting on this if needed, but it > will be open to everyone and will be recorded and shared. Often, the > details are still resolved through the mailing list. > > Jun > > On Tue, Nov 19, 2019 at 6:48 PM Ying Zheng > wrote: > > > Please ignore my previous email > I didn't know Apache requires all the discussions to be "open" > > > On Tue, Nov 19, 2019, 5:40 PM Ying Zheng wrote: > > Hi Jun, > > Thank you very much for your feedback! > > Can we schedule a meeting in your Palo Alto office in December? I think a > face to face discussion is much more efficient than emails. Both Harsha > > and > > I can visit you. Satish may be able to join us remotely. > > On Fri, Nov 15, 2019 at 11:04 AM Jun Rao wrote: > > Hi, Satish and Harsha, > > The following is a more detailed high level feedback for the KIP. > > Overall, > > the KIP seems useful. The challenge is how to design it such that it’s > general enough to support different ways of implementing this feature > > and > > support existing features. > > 40. Local segment metadata storage: The KIP makes the assumption that > > the > > metadata for the archived log segments are cached locally in every > > broker > > and provides a specific implementation for the local storage in the > framework. We probably should discuss this more. For example, some tier > storage providers may not want to cache the metadata locally and just > > rely > > > upon a remote key/value store if such a store is already present. If a > local store is used, there could be different ways of implementing it > (e.g., based on customized local files, an embedded local store like > RocksDB, etc). An alternative of designing this is to just provide an > interface for retrieving the tier segment metadata and leave the details of > how to get the metadata outside of the framework. > > > 41. RemoteStorageManager
Re: [ANNOUNCE] New committer: John Roesler
Congrats, John! -Harsha On Wed, Nov 13, 2019 at 5:42 AM, aishwarya kumar wrote: > Congratulations John!! > > On Wed, Nov 13, 2019 at 8:17 AM Rajini Sivaram > wrote: > > Congratulations, John! > > Regards, > > Rajini > > On Wed, Nov 13, 2019 at 8:49 AM Mickael Maison > wrote: > > Congratulations John! > > On Wed, Nov 13, 2019 at 8:29 AM Tom Bentley wrote: > > Congratulations John! > > On Wed, Nov 13, 2019 at 8:16 AM Bruno Cadonna > > wrote: > > Congrats, John! > > Best, > Bruno > > On Tue, Nov 12, 2019 at 10:56 PM Guozhang Wang > > wrote: > > Hi Everyone, > > The PMC of Apache Kafka is pleased to announce a new Kafka > > committer, > > John > > Roesler. > > John has been contributing to Apache Kafka since early 2018. His > > main > > contributions are primarily around Kafka Streams, but have also > > included > > improving our test coverage beyond Streams as well. Besides his own > > code > > contributions, John has also actively participated on community > > discussions > > and reviews including several other contributors' big proposals > > like > > foreign-key join in Streams (KIP-213). He has also been writing, > > presenting > > and evangelizing Apache Kafka in many venues. > > Congratulations, John! And look forward to more collaborations with > > you > > on > > Apache Kafka. > > Guozhang, on behalf of the Apache Kafka PMC > >
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
Hi Jun, Can you please take a look at Satish's reply. Let us know if that answers your question. I would like to get yours and the rest of the community thoughts on the general direction we are going as we continue to make progress. Thanks, Harsha On Fri, Nov 1, 2019 at 3:06 AM Satish Duggana wrote: > Hi Jun, > Thanks for looking into the updated KIP and clarifying our earlier queries. > > >20. It's fine to keep the HDFS binding temporarily in the PR. We just need > to remove it before it's merged to trunk. As Victor mentioned, we can > provide a reference implementation based on a mocked version of remote > storage. > > Sure, sounds good. > > >21. I am not sure that I understood the need for RemoteLogIndexEntry and > its relationship with RemoteLogSegmentInfo. It seems > that RemoteLogIndexEntry are offset index entries pointing to record > batches inside a segment. That seems to be the same as the .index file? > > That is a good point. `RemoteLogManager` does not put a restriction on > `RemoteStorageManager(RSM)` for maintaining positions in the remote > segment same as the local segments or keeping a correlation between > local segment's positions to the remote segment positions. RSM gives > back the respective entries for a given log segment, call RSM to fetch > the data by giving the respective entry. This allows RSM to have > better control in managing the given log segments. > > Thanks, > Satish. > > On Fri, Nov 1, 2019 at 2:28 AM Jun Rao wrote: > > > > Hi, Harsha, > > > > I am still looking at the KIP and the PR. A couple of quick > > comments/questions. > > > > 20. It's fine to keep the HDFS binding temporarily in the PR. We just > need > > to remove it before it's merged to trunk. As Victor mentioned, we can > > provide a reference implementation based on a mocked version of remote > > storage. > > > > 21. I am not sure that I understood the need for RemoteLogIndexEntry and > > its relationship with RemoteLogSegmentInfo. It seems > > that RemoteLogIndexEntry are offset index entries pointing to record > > batches inside a segment. That seems to be the same as the .index file? > > > > Thanks, > > > > Jun > > > > On Mon, Oct 28, 2019 at 9:11 PM Satish Duggana > > > wrote: > > > > > Hi Viktor, > > > >1. Can we allow RLM Followers to serve read requests? After all > segments > > > on > > > the cold storage are closed ones, no modification is allowed. Besides > > > KIP-392 ( > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica > > > ) > > > would introduce follower fetching too, so I think it would be nice to > > > prepare RLM for this as well. > > > > > > That is a good point. We plan to support fetching remote storage from > > > followers too. Current code in the PR work fine for this scenario > > > though there may be some edge cases to be handled. We have not yet > > > tested this scenario. > > > > > > >2. I think the remote.log.storage.enable config is redundant. By > > > specifying > > > remote.log.storage.manager.class.name one already declares that they > want > > > to use remote storage. Would it make sense to remove > > > the remote.log.storage.enable config? > > > > > > I do not think it is really needed. `remote.log.storage.enable` > > > property can be removed. > > > > > > Thanks, > > > Satish. > > > > > > > > > On Thu, Oct 24, 2019 at 2:46 PM Viktor Somogyi-Vass > > > wrote: > > > > > > > > Hi Harsha, > > > > > > > > A couple more questions: > > > > 1. Can we allow RLM Followers to serve read requests? After all > segments > > > on > > > > the cold storage are closed ones, no modification is allowed. Besides > > > > KIP-392 ( > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica > > > ) > > > > would introduce follower fetching too, so I think it would be nice to > > > > prepare RLM for this as well. > > > > 2. I think the remote.log.storage.enable config is redundant. By > > > specifying > > > > remote.log.storage.manager.class.name one already declares that they > > > want > > > > to use remote storage. Would it make sense to remove > > > > the remote.log.storage.enable config? > > > > >
Re: [VOTE] KIP-514 Add a bounded flush() API to Kafka Producer
+1 (binding) -Harsha On Fri, Oct 25 2019 at 11:01 AM, wrote: > > +1 > > On Thu, Oct 24, 2019 at 9:33 PM radai wrote: > > Hello, > > > > > I'd like to initiate a vote on KIP-514. > > > > > links: > the kip - > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-514%3A+Add+a+bounded+flush%28%29+API+to+Kafka+Producer > the PR - https://github.com/apache/kafka/pull/7569 > > > > > Thank you > > > >
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
Hi Jun, Thanks for the feedback. Given the no.of engineers involved in cross-team effort it would be great to have this as feature branch. Irrespective of if its in my fork or in Apache Kafka's branch it needs to be constantly rebased from trunk to keep it current. Our proposal is to merge it in feature branch and open a PR so its no different than current PR except that its in central repo rather my fork. Having it in Kafka's branch makes it easier for everyone to collaborate on this important feature in kafka. Let me know if you still think otherwise. KIP is updated and we can go through the discussion. Regarding the HDFS dependency its not a direct dependency rather its implementing the RemoteStorageManager interface. We packaged it along with core to make it more convenient to test it. We can move this to external module and keep it there. Let me know what you think. Thanks, Harsha On Tue, Oct 22, 2019 at 3:53 PM Jun Rao wrote: > Hi, Harsha, > > Historically, we tried using a feature branch in 0.8. The experience > actually wasn't great. Merging the feature branch to the main branch > required additional review work and each merge with the main branch added > the risk of introducing new bugs. So, we have been avoiding feature > branches since then, even for some major features. > > It's also going to be weird to have a feature branch before a KIP is > accepted. > > The KIP hasn't been updated much since the initial reviews. Is it ready for > discussion again? > > Looking at the PR, it seems to have direct dependency on HDFS. My > understanding is that the goal of the KIP is to make it more general such > that it can bind to different types of block storage. If so, we should > avoid introducing a direct dependency to any specific block storage in > Apache Kafka. > > Thanks, > > Jun > > On Mon, Oct 21, 2019 at 8:46 AM Harsha wrote: > > > Hi All, > > Thanks for the initial feedback on the KIP-405. We opened a PR > > here https://github.com/apache/kafka/pull/7561 . > > Please take a look and let us know if you have any questions. > > Since this feature is being developed by engineers from different > > companies we would like to open a feature branch in apache kafka git. It > > will allow us collaborate in open source community rather than in private > > branches. Please let me know if you have any objections to opening a > > feature branch in kafka's git repo. > > > > Thanks, > > Harsha > > > > On Mon, Apr 8, 2019, at 10:04 PM, Harsha wrote: > > > Thanks, Ron. Updating the KIP. will add answers here as well > > > > > > 1) If the cold storage technology can be cross-region, is there a > > > possibility for a disaster recovery Kafka cluster to share the > messages > > in > > > cold storage? My guess is the answer is no, and messages replicated > to > > the > > > D/R cluster have to be migrated to cold storage from there > > independently. > > > (The same cross-region cold storage medium could be used, but every > > message > > > would appear there twice). > > > > > > If I understand the question correctly, what you are saying is Kafka A > > > cluster (active) shipping logs to remote storage which cross-region > > > replication and another Kafka Cluster B (Passive) will it be able to > > > use the remote storage copied logs directly. > > > For the initial version my answer is No. We can handle this in > > > subsequent changes after this one. > > > > > > 2) Can/should external (non-Kafka) tools have direct access to the > > messages > > > in cold storage. I think this might have been addressed when someone > > asked > > > about ACLs, and I believe the answer is "no" -- if some external tool > > needs > > > to operate on that data then that external tool should read that data > by > > > acting as a Kafka consumer. Again, just asking to get the answer > clearly > > > documented in case it is unclear. > > > > > > The answer is No. All tools/clients must go through broker APIs to > > > access any data (local or remote). > > > Only Kafka broker user will have access to remote storage logs and > > > Security/ACLs will work the way it does today. > > > Tools/Clients going directly to the remote storage might help in terms > > > of efficiency but this requires Protocol changes and some way of > > > syncing ACLs in Kafka to the Remote storage. > > > > > > > > > Thanks, > > > Harsha > > > > > > On Mon, Apr 8, 2019, at 8:48
Re: [VOTE] KIP-537: Increase default zookeeper session timeout
+1 (binding). Thanks, Harsha On Tue, Oct 22, 2019 at 3:20 PM Colin McCabe wrote: > +1 (binding). > > Thanks, Jason. > > best, > Colin > > > On Mon, Oct 21, 2019, at 17:27, Jason Gustafson wrote: > > I'd like to start a vote for KIP-537: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-537%3A+Increase+default+zookeeper+session+timeout > > . > > > > +1 from me > > > > Thanks, > > Jason > > >
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
Hi All, Thanks for the initial feedback on the KIP-405. We opened a PR here https://github.com/apache/kafka/pull/7561 . Please take a look and let us know if you have any questions. Since this feature is being developed by engineers from different companies we would like to open a feature branch in apache kafka git. It will allow us collaborate in open source community rather than in private branches. Please let me know if you have any objections to opening a feature branch in kafka's git repo. Thanks, Harsha On Mon, Apr 8, 2019, at 10:04 PM, Harsha wrote: > Thanks, Ron. Updating the KIP. will add answers here as well > > 1) If the cold storage technology can be cross-region, is there a > possibility for a disaster recovery Kafka cluster to share the messages in > cold storage? My guess is the answer is no, and messages replicated to the > D/R cluster have to be migrated to cold storage from there independently. > (The same cross-region cold storage medium could be used, but every message > would appear there twice). > > If I understand the question correctly, what you are saying is Kafka A > cluster (active) shipping logs to remote storage which cross-region > replication and another Kafka Cluster B (Passive) will it be able to > use the remote storage copied logs directly. > For the initial version my answer is No. We can handle this in > subsequent changes after this one. > > 2) Can/should external (non-Kafka) tools have direct access to the messages > in cold storage. I think this might have been addressed when someone asked > about ACLs, and I believe the answer is "no" -- if some external tool needs > to operate on that data then that external tool should read that data by > acting as a Kafka consumer. Again, just asking to get the answer clearly > documented in case it is unclear. > > The answer is No. All tools/clients must go through broker APIs to > access any data (local or remote). > Only Kafka broker user will have access to remote storage logs and > Security/ACLs will work the way it does today. > Tools/Clients going directly to the remote storage might help in terms > of efficiency but this requires Protocol changes and some way of > syncing ACLs in Kafka to the Remote storage. > > > Thanks, > Harsha > > On Mon, Apr 8, 2019, at 8:48 AM, Ron Dagostino wrote: > > Hi Harsha. A couple of questions. I think I know the answers, but it > > would be good to see them explicitly documented. > > > > 1) If the cold storage technology can be cross-region, is there a > > possibility for a disaster recovery Kafka cluster to share the messages in > > cold storage? My guess is the answer is no, and messages replicated to the > > D/R cluster have to be migrated to cold storage from there independently. > > (The same cross-region cold storage medium could be used, but every message > > would appear there twice). > > > > 2) Can/should external (non-Kafka) tools have direct access to the messages > > in cold storage. I think this might have been addressed when someone asked > > about ACLs, and I believe the answer is "no" -- if some external tool needs > > to operate on that data then that external tool should read that data by > > acting as a Kafka consumer. Again, just asking to get the answer clearly > > documented in case it is unclear. > > > > Ron > > > > > > On Thu, Apr 4, 2019 at 12:53 AM Harsha wrote: > > > > > Hi Viktor, > > > > > > > > > "Now, will the consumer be able to consume a remote segment if: > > > - the remote segment is stored in the remote storage, BUT > > > - the leader broker failed right after this AND > > > - the follower which is to become a leader didn't scan yet for a new > > > segment?" > > > > > > If I understand correctly, after a local log segment copied to remote and > > > leader is failed to write the index files and leadership changed to a > > > follower. In this case we consider the log segment copy failed and newly > > > elected leader will start copying the data from last the known offset in > > > the remote to copy. Consumers who are looking for the offset which might > > > be in the failed copy log segment will continue to be read the data from > > > local disk since the local log segment will only be deleted once a > > > successful copy of the log segment. > > > > > > "As a follow-up question, what are your experiences, does a failover in a > > > broker causes bigger than usual churn in the consumers? (I'm thinking > > > about > > > the time re
Re: [VOTE] KIP-511: Collect and Expose Client's Name and Version in the Brokers
+1 (binding). Thanks for the KIP this is going to be super useful. Thanks, Harsha On Fri, Oct 11, 2019 at 2:57 PM Guozhang Wang wrote: > Hi David, > > Thanks for the KIP. I know I'm late on voting here but I'd be +1 on this > too! > > Just a quick clarification on the tag "name=Connections" of the third > metric, what would be the format of "Connections" here? Would that be the > connection listener name? > > > Guozhang > > > On Fri, Oct 11, 2019 at 1:49 PM Jun Rao wrote: > > > Hi, David, > > > > Thanks for the KIP. Just a minor comment below. > > > > 100. It seems that the new flexible fields need tag numbers. Could you > add > > them to the wiki? > > > > Jun > > > > On Mon, Sep 23, 2019 at 11:37 AM Jason Gustafson > > wrote: > > > > > Thanks David for the clarification. That sounds good. > > > > > > On Mon, Sep 23, 2019 at 12:35 AM David Jacot > > wrote: > > > > > > > Hi all, > > > > > > > > The vote has passed with +3 binding votes (Colin McCabe, Gwen > Shapira, > > > > Jason Gustafson) and +3 non binding votes (Mickael Maison, > Konstantine > > > > Karantasis, Kevin Lu). \o/ > > > > > > > > Thanks to everyone that reviewed and helped improve this proposal, > and > > > > huge thanks to Colin for his great feedback. > > > > > > > > Best, > > > > David > > > > > > > > On Mon, Sep 23, 2019 at 9:28 AM David Jacot > > wrote: > > > > > > > > > Hi Jason, > > > > > > > > > > The response will be a flexible version but the response header > won't > > > be > > > > > (only for the api versions response). I have forgotten to change > this > > > > point > > > > > in the KIP. I will make this point clearer. > > > > > > > > > > I didn't know that INVALID_REQUEST already exists. Yes, it makes > > sense > > > to > > > > > reuse it then. > > > > > > > > > > Best, > > > > > David > > > > > > > > > > On Sat, Sep 21, 2019 at 3:02 AM Kevin Lu > > > wrote: > > > > > > > > > >> +1 (non-binding) > > > > >> > > > > >> Definitely needed this before as it would have saved me some time > > from > > > > >> trying to guess a client's version from api version/source code. > > > Thanks > > > > >> for > > > > >> the KIP! > > > > >> > > > > >> Regards, > > > > >> Kevin > > > > >> > > > > >> On Fri, Sep 20, 2019 at 4:29 PM Jason Gustafson < > ja...@confluent.io > > > > > > > >> wrote: > > > > >> > > > > >> > +1 from me. This is a clever solution. Kind of a pity we > couldn't > > > work > > > > >> > flexible version support into the response, but I understand why > > it > > > is > > > > >> > difficult. > > > > >> > > > > > >> > One minor nitpick: the INVALID_REQUEST error already exists. Are > > you > > > > >> > intending to reuse it? > > > > >> > > > > > >> > Thanks, > > > > >> > Jason > > > > >> > > > > > >> > On Fri, Sep 20, 2019 at 3:50 PM Konstantine Karantasis < > > > > >> > konstant...@confluent.io> wrote: > > > > >> > > > > > >> > > Quite useful KIP from an operational standpoint and I also > like > > it > > > > in > > > > >> its > > > > >> > > most recent revised form. > > > > >> > > > > > > >> > > +1 (non-binding). > > > > >> > > > > > > >> > > Konstantine > > > > >> > > > > > > >> > > > > > > >> > > On Fri, Sep 20, 2019 at 9:55 AM Gwen Shapira < > g...@confluent.io > > > > > > > >> wrote: > > > > >> > > > > > > >> > > > +1 (binding) > > > > >> > > > > > > > >> > > > Thanks for the KIP, David and for the help with the design, > > > > Colin.
Re: [VOTE] KIP-416: Notify SourceTask of ACK'd offsets, metadata
+1 (binding).
Thanks,
Harsha
On Fri, Oct 04, 2019 at 6:53 AM, Manikumar
wrote:
> Hi All,
>
> Please vote here for the formal approval of this KIP.
> https://github.com/apache/kafka/pull/6295#discussion_r328867657
>
> Thanks,
>
>
> On Wed, Oct 2, 2019 at 5:50 AM Ryanne Dolan wrote:
>
> Hey y'all, resurrecting an old KIP for the benefit of KIP-382, which
> depends on an additional parameter in SourceTask.commitRecord(). I've
> updated KIP-416 according to consensus reached in PR-6295. Let's finish the
> vote so we can formally approve this minor KIP, please!
>
> Ryanne
>
> On Mon, Jan 21, 2019, 4:25 PM Ryanne Dolan wrote:
>
> > Andrew, I agree it's a better commitRecord, but with the slightly
> > different semantics you mentioned. I suppose we could document that well
> > enough that reusing the same name would be fine.
> >
> > I'll resend the discussion email. Maybe it got lost somehow.
> >
> > Ryanne
> >
> > On Mon, Jan 21, 2019, 4:37 AM Andrew Schofield com
> > wrote:
> >
> >> Hi,
> >> I'm not quite sure about the etiquette here but I wonder whether the KIP
> >> could be improved. I think I missed the DISCUSS thread.
> >>
> >> I think that really your recordLogged(SourceRecord, RecordMetadata)
> >> method is actually a better version of commitRecord() and perhaps it
> ought
> >> to be an overload. This is similar to the situation in which the
> Serializer
> >> interface was enhanced when record headers were added.
> >>
> >> public abstract class SourceTask implements Task {
> >> public void commitRecord(SourceRecord sourceRecord, RecordMetadata
> >> recordMetadata) {
> >> this.commitRecord();
> >> }
> >>
> >> public void commitRecord() {
> >> }
> >> }
> >>
> >> Or something like that. I do understand that the KIP mentions that
> >> recordLogged() is only called for records that are actually ACKed, but
> it's
> >> very similar in intent to commitRecord() in my view.
> >>
> >> Just my 2 cents.
> >>
> >> Andrew Schofield
> >> IBM Event Streams
> >>
> >>
> >> On 17/01/2019, 23:54, "Ryanne Dolan" wrote:
> >>
> >> Hey y'all, please vote for KIP-416 by replying +1 to this thread.
> >>
> >> Right now, there is no way for a SourceConnector/Task to know:
> >>
> >> - whether a record was successfully sent to Kafka, vs filtered out
> or
> >> skipped.
> >> - the downstream offsets and metadata of sent records
> >>
> >> KIP-416 proposes adding a recordLogged() callback for this purpose.
> >>
> >>
> >> https://nam05.safelinks.protection.outlook.com/
> ?url=https%3A%2F%2Fcwiki.apache.
> org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-416%253A%2BNotify%2BSourceTask%2Bof%2BACK%2527d%2Boffsets%252C%2Bmetadatadata=02%7C01%7C%7C9fa617754cce4bab7ba508d67cd7128f%7C84df9e7fe9f640afb435%7C1%7C0%7C636833660500817715sdata=udEP27%2FrshuP5sWthvZmUIdt13whM5XqKMoia1wE93c%3Dreserved=0
> >>
> >> Thanks!
> >> Ryanne
> >>
> >>
> >>
>
>
Re: [VOTE] KIP-524: Allow users to choose config source when describing configs
Thanks for the KIP. +1 (binding). -Harsha On Wed, Sep 25, 2019 at 11:55 AM Satish Duggana wrote: > Thanks Jason for the KIP. It looks to be a minor improvement but very > useful. > > +1 (non-binding) > > On Wed, Sep 25, 2019 at 9:29 PM Rajini Sivaram > wrote: > > > > Thanks for the KIP, Jason! > > > > +1 (binding) > > > > Regards, > > > > Rajini > > > > On Wed, Sep 25, 2019 at 4:41 PM Colin McCabe wrote: > > > > > Looks good. +1 (binding) > > > > > > best, > > > Colin > > > > > > On Tue, Sep 24, 2019, at 09:42, Jason Gustafson wrote: > > > > Hi All, > > > > > > > > I'm starting a vote for KIP-524, which is a small change to the > config > > > > tool: > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-524%3A+Allow+users+to+choose+config+source+when+describing+configs > > > > . > > > > > > > > +1 from me. > > > > > > > > Thanks, > > > > Jason > > > > > > > >
Re: [VOTE] KIP-525 - Return topic metadata and configs in CreateTopics response
+1 (binding). -Harsha On Sat, Sep 21, 2019 at 12:13 AM, Manikumar wrote: > +1 (binding), Thanks for the KIP. > > Thanks, > > On Sat, Sep 21, 2019 at 2:49 AM Colin McCabe wrote: > > +1 (binding). Thanks, Rajini. > > best, > Colin > > On Fri, Sep 20, 2019, at 00:43, Rajini Sivaram wrote: > > Hi all, > > I would like to start vote on KIP-525 to return configs in CreateTopics > response. This is a minor KIP that returns additional data in the > > response > > without breaking compatibility. > > - > > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-525+-+Return+topic+metadata+and+configs+in+CreateTopics+response > > Thank you, > > Rajini > >
Re: [VOTE] KIP-517: Add consumer metrics to observe user poll behavior
+1 (binding). Thanks for the KIP. -Harsha On Wed, Sep 18, 2019 at 9:07 AM Mickael Maison wrote: > +1 (non binding) > Thanks for the KIP, I can see this being really useful! > > On Wed, Sep 18, 2019 at 4:40 PM Kevin Lu wrote: > > > > Hi All, > > > > Since we have a bit of support, I'd like to start the vote on KIP-517: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-517%3A+Add+consumer+metrics+to+observe+user+poll+behavior > > > > Thanks! > > > > Regards, > > Kevin >
Re: [VOTE] KIP-373: Allow users to create delegation tokens for other users
+1 (binding). Thanks for the KIP Viktor Thanks, Harsha On Mon, Sep 16, 2019 at 3:02 AM, Viktor Somogyi-Vass < viktorsomo...@gmail.com > wrote: > > > > Hi All, > > > > I'd like to bump this again in order to get some more binding votes and/or > feedback in the hope we can push this in for 2.4. > > > > Thank you Manikumar, Gabor and Ryanne so far for the votes! (the last two > were on the discussion thread after starting the vote but I think it still > counts :) ) > > > > Thanks, > Viktor > > > > On Wed, Aug 21, 2019 at 1:44 PM Manikumar < manikumar. reddy@ gmail. com ( > manikumar.re...@gmail.com ) > wrote: > > >> >> >> Hi, >> >> >> >> +1 (binding). >> >> >> >> Thanks for the updated KIP. LGTM. >> >> >> >> Thanks, >> Manikumar >> >> >> >> On Tue, Aug 6, 2019 at 3:14 PM Viktor Somogyi-Vass < viktorsomogyi@ gmail. >> com ( viktorsomo...@gmail.com ) > >> wrote: >> >> >>> >>> >>> Hi All, >>> >>> >>> >>> Bumping this, I'd be happy to get some additional feedback and/or votes. >>> >>> >>> >>> Thanks, >>> Viktor >>> >>> >>> >>> On Wed, Jul 31, 2019 at 11:04 AM Viktor Somogyi-Vass < viktorsomogyi@ gmail. >>> com ( viktorsomo...@gmail.com ) > wrote: >>> >>> >>>> >>>> >>>> Hi All, >>>> >>>> >>>> >>>> I'd like to start a vote on this KIP. >>>> >>>> >>> >>> >> >> >> >> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ >> KIP-373%3A+Allow+users+to+create+delegation+tokens+for+other+users >> ( >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-373%3A+Allow+users+to+create+delegation+tokens+for+other+users >> ) >> >> >>> >>>> >>>> >>>> To summarize it: the proposed feature would allow users (usually >>>> superusers) to create delegation tokens for other users. This is >>>> >>>> >>> >>> >>> >>> especially >>> >>> >>>> >>>> >>>> helpful in Spark where the delegation token created this way can be >>>> distributed to workers. >>>> >>>> >>>> >>>> I'd be happy to receive any votes or additional feedback. >>>> >>>> >>>> >>>> Viktor >>>> >>>> >>> >>> >> >> > > >
Re: [VOTE] KIP-309: Add toUpperCase support to sasl.kerberos.principal.to.local rule
+1 (binding). Thanks, Harsha On Mon, Sep 16, 2019 at 4:21 AM, Manikumar wrote: > Hi all, > > I would like to start vote on this trivial KIP-309: https://cwiki.apache. > org/confluence/display/KAFKA/KIP > -309%3A+Add+toUpperCase+support+to+sasl.kerberos.principal.to.local+rule > > Thanks, >
Re: [DISCUSS] KIP-517: Add consumer metric indicating time between poll calls
Thanks. +1 LGTM. On Mon, Sep 16, 2019 at 9:19 AM, Kevin Lu wrote: > Hi Harsha, > > Thanks for the feedback. I have added *last-poll-seconds-ago* to the KIP > (being consistent with *last-heartbeat-seconds-ago*). > > Regards, > Kevin > > On Sat, Sep 14, 2019 at 9:44 AM Harsha Chintalapani > wrote: > > Thanks Kevin for the KIP. Overall LGTM. > On you second point, I think the metric will be really useful to indicate > the perf bottlenecks on user code vs kakfa consumer/broker. > > Thanks, > Harsha > > On Fri, Sep 13, 2019 at 2:41 PM, Kevin Lu wrote: > > Hi Radai & Jason, > > Thanks for the support and suggestion. > > 1. I think ratio is a good additional metric since the current proposed > metrics are only absolute times which may not be useful in all scenarios. > > I have added this to the KIP: > * - poll-idle-ratio*: The fraction of time the consumer spent waiting for > the user to process records from poll. > > Thoughts on the metric name/description? > > 2. Would it be useful to include a metric measuring the time since poll > was last called? Similar to *heartbeat-last-seconds-ago*, it would be > *poll-last-ms-ago. > *This could be useful if (1) the user has a very high *max.poll.interval. > ms > <http://max.poll.interval.ms>* configured and typically spends a long > time processing, or (2) comparing this metric with others such as > *heartbeat-last-seconds-ago* or something else for gathering data in root > cause analyses (or identifying potential consumer bugs related to poll). > > Regards, > Kevin > > On Fri, Sep 13, 2019 at 10:39 AM Jason Gustafson > wrote: > > Hi Kevin, > > This looks reasonable to me. I'd also +1 Radai's suggestion if you're > willing. Something like an idle ratio for the consumer would be helpful. > > Thanks, > Jason > > On Fri, Sep 13, 2019 at 10:08 AM radai > wrote: > > while youre at it another metric that we have found to be useful is % > > time > > spent in user code vs time spent in poll() (so time between poll calls / > time inside poll calls) - the higher the % value the more indicative of > user code being the cause of performance bottlenecks. > > On Fri, Sep 13, 2019 at 9:14 AM Kevin Lu wrote: > > Hi All, > > Happy Friday! Bumping this. Any thoughts? > > Thanks. > > Regards, > Kevin > > On Thu, Sep 5, 2019 at 9:35 AM Kevin Lu wrote: > > Hi All, > > I'd like to propose a new consumer metric that measures the time > > between > > calls to poll() for use in issues related to hitting > > max.poll.interval.ms > > due to long processing time. > > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-517%3A+Add+consumer+metric+indicating+time+between+poll+calls > > Please give it a read and let me know what you think. > > Thanks! > > Regards, > Kevin > >
Re: [DISCUSS] KIP-491: Preferred Leader Deprioritized List (Temporary Blacklist)
Hi Stanislav, Thanks for the comments. The proposal we are making is not about optimizing Big-O but instead provide a simpler way of stopping a broker becoming leader. If we want to go with making this an option and providing a tool which abstracts moving the broker to end preferred leader list , it needs to do it for all the partitions that broker is leader for. As said in the above comment a broker i.e leader for 1000 partitions we have to this for all the partitions. Instead of having a blacklist will help simplify this process and we can provide monitoring/alerts on such list. "This sounds like a bit of a hack. If that is the concern, why not propose a KIP that addresses the specific issue?" Do you mind shedding some light what issue you are talking to propose a KIP for? Replication is a challenge when we are bringing up a new node. If you have retention period of 3 days there is honestly no way to do it via online replication without taking a hit on latency SLAs. Is your ask to find a way to fix the replication itself when we are bringing a new broker from no data. "Having a blacklist you control still seems like a workaround given that Kafka itself knows when the topic retention would allow you to switch that replica to a leader" Not sure how its making it any complicated by having a single zk path to have a list of brokers. Thanks, Harsha On Mon, Sep 09, 2019 at 3:55 PM, Stanislav Kozlovski < stanis...@confluent.io > wrote: > > > > I agree with Colin that the same result should be achievable through > proper abstraction in a tool. Even if that might be "4xO(N)" operations, > that is still not a lot - it is still classified as O(N) > > > > Let's say a healthy broker hosting 3000 partitions, and of which 1000 are > > >> >> >> the preferred leaders (leader count is 1000). There is a hardware failure >> (disk/memory, etc.), and kafka process crashed. We swap this host with >> another host but keep the same broker. id ( http://broker.id/ ) , when this >> new broker coming up, it has no historical data, and we manage to have the >> current last offsets of all partitions set in the >> replication-offset-checkpoint (if we don't set them, it could cause crazy >> ReplicaFetcher pulling of historical data from other brokers and cause >> cluster high latency and other instabilities), so when Kafka is brought >> up, it is quickly catching up as followers in the ISR. Note, we have >> auto.leader.rebalance.enable disabled, so it's not serving any traffic as >> leaders (leader count = 0), even there are 1000 partitions that this >> broker is the Preferred Leader. We need to make this broker not serving >> traffic for a few hours or days depending on the SLA of the topic >> retention requirement until after it's having enough historical data. >> >> > > > > This sounds like a bit of a hack. If that is the concern, why not propose > a KIP that addresses the specific issue? Having a blacklist you control > still seems like a workaround given that Kafka itself knows when the topic > retention would allow you to switch that replica to a leader > > > > I really hope we can come up with a solution that avoids complicating the > controller and state machine logic further. > Could you please list out the main drawbacks of abstract this away in the > reassignments tool (or a new tool)? > > > > On Mon, Sep 9, 2019 at 7:53 AM Colin McCabe < cmccabe@ apache. org ( > cmcc...@apache.org ) > wrote: > > >> >> >> On Sat, Sep 7, 2019, at 09:21, Harsha Chintalapani wrote: >> >> >>> >>> >>> Hi Colin, >>> Can you give us more details on why you don't want this to be part of the >>> Kafka core. You are proposing KIP-500 which will take away zookeeper and >>> writing this interim tools to change the zookeeper metadata doesn't make >>> sense to me. >>> >>> >> >> >> >> Hi Harsha, >> >> >> >> The reassignment API described in KIP-455, which will be part of Kafka >> 2.4, doesn't rely on ZooKeeper. This API will stay the same after KIP-500 >> is implemented. >> >> >>> >>> >>> As George pointed out there are >>> several benefits having it in the system itself instead of asking users to >>> hack bunch of json files to deal with outage scenario. >>> >>> >> >> >> >> In both cases, the user just has to run a shell command, right? In both >> cases, the user has to remember to undo the command later when they want >> the broker
Re: [DISCUSS] KIP-517: Add consumer metric indicating time between poll calls
Thanks Kevin for the KIP. Overall LGTM. On you second point, I think the metric will be really useful to indicate the perf bottlenecks on user code vs kakfa consumer/broker. Thanks, Harsha On Fri, Sep 13, 2019 at 2:41 PM, Kevin Lu wrote: > Hi Radai & Jason, > > Thanks for the support and suggestion. > > 1. I think ratio is a good additional metric since the current proposed > metrics are only absolute times which may not be useful in all scenarios. > > I have added this to the KIP: > * - poll-idle-ratio*: The fraction of time the consumer spent waiting for > the user to process records from poll. > > Thoughts on the metric name/description? > > 2. Would it be useful to include a metric measuring the time since poll > was last called? Similar to *heartbeat-last-seconds-ago*, it would be > *poll-last-ms-ago. > *This could be useful if (1) the user has a very high *max.poll.interval. > ms > <http://max.poll.interval.ms>* configured and typically spends a long > time processing, or (2) comparing this metric with others such as > *heartbeat-last-seconds-ago* or something else for gathering data in root > cause analyses (or identifying potential consumer bugs related to poll). > > Regards, > Kevin > > On Fri, Sep 13, 2019 at 10:39 AM Jason Gustafson > wrote: > > Hi Kevin, > > This looks reasonable to me. I'd also +1 Radai's suggestion if you're > willing. Something like an idle ratio for the consumer would be helpful. > > Thanks, > Jason > > On Fri, Sep 13, 2019 at 10:08 AM radai > wrote: > > while youre at it another metric that we have found to be useful is % time > spent in user code vs time spent in poll() (so time between poll calls / > time inside poll calls) - the higher the % value the more indicative of > user code being the cause of performance bottlenecks. > > On Fri, Sep 13, 2019 at 9:14 AM Kevin Lu wrote: > > Hi All, > > Happy Friday! Bumping this. Any thoughts? > > Thanks. > > Regards, > Kevin > > On Thu, Sep 5, 2019 at 9:35 AM Kevin Lu wrote: > > Hi All, > > I'd like to propose a new consumer metric that measures the time > > between > > calls to poll() for use in issues related to hitting > > max.poll.interval.ms > > due to long processing time. > > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-517%3A+Add+consumer+metric+indicating+time+between+poll+calls > > Please give it a read and let me know what you think. > > Thanks! > > Regards, > Kevin > >
Re: [DISCUSS] KIP-491: Preferred Leader Deprioritized List (Temporary Blacklist)
Hi Colin, Can you give us more details on why you don't want this to be part of the Kafka core. You are proposing KIP-500 which will take away zookeeper and writing this interim tools to change the zookeeper metadata doesn't make sense to me. As George pointed out there are several benefits having it in the system itself instead of asking users to hack bunch of json files to deal with outage scenario. Thanks, Harsha On Fri, Sep 6, 2019 at 4:36 PM George Li wrote: > Hi Colin, > > Thanks for the feedback. The "separate set of metadata about blacklists" > in KIP-491 is just the list of broker ids. Usually 1 or 2 or a couple in > the cluster. Should be easier than keeping json files? e.g. what if we > first blacklist broker_id_1, then another broker_id_2 has issues, and we > need to write out another json file to restore later (and in which order)? > Using blacklist, we can just add the broker_id_2 to the existing one. and > remove whatever broker_id returning to good state without worrying how(the > ordering of putting the broker to blacklist) to restore. > > For topic level config, the blacklist will be tied to > topic/partition(e.g. Configs: > topic.preferred.leader.blacklist=0:101,102;1:103where 0 & 1 is the > partition#, 101,102,103 are the blacklist broker_ids), and easier to > update/remove, no need for external json files? > > > Thanks, > George > > On Friday, September 6, 2019, 02:20:33 PM PDT, Colin McCabe < > cmcc...@apache.org> wrote: > > One possibility would be writing a new command-line tool that would > deprioritize a given replica using the new KIP-455 API. Then it could > write out a JSON files containing the old priorities, which could be > restored when (or if) we needed to do so. This seems like it might be > simpler and easier to maintain than a separate set of metadata about > blacklists. > > best, > Colin > > > On Fri, Sep 6, 2019, at 11:58, George Li wrote: > > Hi, > > > > Just want to ping and bubble up the discussion of KIP-491. > > > > On a large scale of Kafka clusters with thousands of brokers in many > > clusters. Frequent hardware failures are common, although the > > reassignments to change the preferred leaders is a workaround, it > > incurs unnecessary additional work than the proposed preferred leader > > blacklist in KIP-491, and hard to scale. > > > > I am wondering whether others using Kafka in a big scale running into > > same problem. > > > > > > Satish, > > > > Regarding your previous question about whether there is use-case for > > TopicLevel preferred leader "blacklist", I thought about one > > use-case: to improve rebalance/reassignment, the large partition will > > usually cause performance/stability issues, planning to change the say > > the New Replica will start with Leader's latest offset(this way the > > replica is almost instantly in the ISR and reassignment completed), and > > put this partition's NewReplica into Preferred Leader "Blacklist" at > > the Topic Level config for that partition. After sometime(retention > > time), this new replica has caught up and ready to serve traffic, > > update/remove the TopicConfig for this partition's preferred leader > > blacklist. > > > > I will update the KIP-491 later for this use case of Topic Level config > > for Preferred Leader Blacklist. > > > > > > Thanks, > > George > > > >On Wednesday, August 7, 2019, 07:43:55 PM PDT, George Li > > wrote: > > > > Hi Colin, > > > > > In your example, I think we're comparing apples and oranges. You > started by outlining a scenario where "an empty broker... comes up... > [without] any > leadership[s]." But then you criticize using reassignment > to switch the order of preferred replicas because it "would not actually > switch the leader > automatically." If the empty broker doesn't have any > leaderships, there is nothing to be switched, right? > > > > Let me explained in details of this particular use case example for > > comparing apples to apples. > > > > Let's say a healthy broker hosting 3000 partitions, and of which 1000 > > are the preferred leaders (leader count is 1000). There is a hardware > > failure (disk/memory, etc.), and kafka process crashed. We swap this > > host with another host but keep the same broker.id, when this new > > broker coming up, it has no historical data, and we manage to have the > > current last offsets of all partitions set in > > the replication-offset-checkpoint (if we don't set them, it could
Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields
LGTM. +1 (binding) -Harsha On Wed, Sep 04, 2019 at 1:46 AM, Satish Duggana wrote: > +1 (non-binding) Thanks for the nice KIP. > > You may want to update the KIP saying that optional tagged fields do not > support complex types(or structs). > > On Wed, Sep 4, 2019 at 3:43 AM Jose Armando Garcia Sancio > wrote: > > +1 (non-binding) > > Looking forward to this improvement. > > On Tue, Sep 3, 2019 at 12:49 PM David Jacot wrote: > > +1 (non-binding) > > Thank for the KIP. Great addition to the Kafka protocol! > > Best, > David > > Le mar. 3 sept. 2019 à 19:17, Colin McCabe a écrit : > > Hi all, > > I'd like to start the vote for KIP-482: The Kafka Protocol should Support > Optional Tagged Fields. > > KIP: > > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields > > Discussion thread here: > > https://lists.apache.org/thread.html/ > cdc801ae886491b73ef7efecac7ef81b24382f8b6b025899ee343f7a@%3Cdev.kafka. > apache.org%3E > > best, > Colin > > -- > -Jose > >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Thanks Pere. KIP looks good to me. -Harsha On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes wrote: > Not really, > my idea is to keep the JAAS parameter, so people don't see major > changes. But if you pass a properties file, then this takes precedence over > the other, with the idea that you can do sasl as well with the properties > files. > > Makes sense? > > -- Pere > > Missatge de Harsha Chintalapani del dia dv., 30 d’ag. > 2019 a les 19:00: > > Hi Pere, > Thanks for the KIP. Enabling SSL for zookeeper for Kafka makes > sense. > "The changes are planned to be introduced in a compatible way, by keeping > the current JAAS variable precedence." > Can you elaborate a bit here. If the user configures a JAAS file with > Client section it will take precedence over zookeeper SSL configs? > > Thanks, > Harsha > > > > On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes > wrote: > > Hi, > quick question, I saw in another mail that 2.4 release is planned for > September. I think it would be really awesome to have this for this > release, do you think we can make it? > > -- Pere > > Missatge de Pere Urbón Bayes del dia dj., 29 d’ag. > 2019 a les 20:10: > > Hi, > this is my first KIP for a change in Apache Kafka, so I'm really need to > the process. Looking forward to hearing from you and learn the best ropes > here. > > I would like to propose this KIP-515 to enable the ZookeeperClients to > take full advantage of the TLS communication in the new Zookeeper 3.5.5. > Specially interesting it the Zookeeper Security Migration, that without > this change will not work with TLS, disabling users to use ACLs when the > Zookeeper cluster use TLS. > > link: > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication > > Looking forward to hearing from you on this, > > /cheers > > -- > Pere Urbon-Bayes > Software Architect > http://www.purbon.com > https://twitter.com/purbon > https://www.linkedin.com/in/purbon/ > > -- > Pere Urbon-Bayes > Software Architect > http://www.purbon.com > https://twitter.com/purbon > https://www.linkedin.com/in/purbon/ > > > > > -- > Pere Urbon-Bayes > Software Architect > http://www.purbon.com > https://twitter.com/purbon > https://www.linkedin.com/in/purbon/ >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Pere, Thanks for the KIP. Enabling SSL for zookeeper for Kafka makes sense. "The changes are planned to be introduced in a compatible way, by keeping the current JAAS variable precedence." Can you elaborate a bit here. If the user configures a JAAS file with Client section it will take precedence over zookeeper SSL configs? Thanks, Harsha On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes wrote: > Hi, > quick question, I saw in another mail that 2.4 release is planned for > September. I think it would be really awesome to have this for this > release, do you think we can make it? > > -- Pere > > Missatge de Pere Urbón Bayes del dia dj., 29 d’ag. > 2019 a les 20:10: > > Hi, > this is my first KIP for a change in Apache Kafka, so I'm really need to > the process. Looking forward to hearing from you and learn the best ropes > here. > > I would like to propose this KIP-515 to enable the ZookeeperClients to > take full advantage of the TLS communication in the new Zookeeper 3.5.5. > Specially interesting it the Zookeeper Security Migration, that without > this change will not work with TLS, disabling users to use ACLs when the > Zookeeper cluster use TLS. > > link: > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication > > Looking forward to hearing from you on this, > > /cheers > > -- > Pere Urbon-Bayes > Software Architect > http://www.purbon.com > https://twitter.com/purbon > https://www.linkedin.com/in/purbon/ > > -- > Pere Urbon-Bayes > Software Architect > http://www.purbon.com > https://twitter.com/purbon > https://www.linkedin.com/in/purbon/ >
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Hi Maulin,
Use cases are clear now. I am +1 for moving forward
with the discussions on having such configurable option for users. But the
interfaces is proposed doesn't look right to me. We are still talking about
keystore interfaces. Given keystore's are used as filebased way of
transporting certificates I am not sure it will help the rest of the
user-base.
In short, I am +1 on the KIP's motivation and only have
questions around returning keystores instead of returning certs, private
keys etc. . If others in the community are ok with such interface we can
move forward.
Thanks,
Harsha
On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada
wrote:
> Hi Harsha
>
> As we synced-up offline on this topic, we hope you don't have any more
> clarifications that you are seeking. If that is the case, can you please
> help us move this forward and discuss what changes you would expect on the
> KIP design in order to make it valuable contribution?
>
> Just FYI - we verified our primary design change with the author of Sun's
> X509 Trustmanager implementation and the outcome is that what we are
> proposing makes sense at the heart of it - "Instead of writing TrustManager
> just plugin the Trust store". We are open to discuss additional changes
> that you/anybody else would like to see on the functionality however.
>
> Thanks
> Maulin
>
> On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada
> wrote:
>
> Hi Harsha
>
> Any response on my question? I feel this KIP is worth accommodating. Your
> help is much appreciated.
>
> Thanks
> Maulin
>
> On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada < maulin.vasavada@gmail.
> com> wrote:
>
> Hi Harsha
>
> I've examined the SPIFFE provider more and have one question -
>
> If SPIFFE didn't have a need to do checkSpiffeId() call at the below
> location, would you really still write the Provider? *OR* Would you just
> use TrustManagerFactory.init(KeyStore) signature to pass the KeyStore from
> set of certs returned by spiffeIdManager. getTrustedCerts()?
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> provider/CertificateUtils.java#L100
>
> /**
>
> * Validates that the SPIFFE ID is present and matches the SPIFFE ID
> configured in
> * the java.security property ssl.spiffe.accept
> *
> * If the authorized spiffe ids list is empty any spiffe id is authorized
> *
> * @param chain an array of X509Certificate that contains the Peer's SVID
> to be validated
> * @throws CertificateException when either the certificates doesn't have a
> SPIFFE ID or the SPIFFE ID is not authorized
> */
> static void checkSpiffeId(X509Certificate[] chain) throws
> CertificateException {
>
> Thanks
> Maulin
>
> On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani
> wrote:
>
> Maulin,
> The code parts you are pointing are specific for Spiffe and if
> you are talking about validate method which uses PKIX check like any other
> provider does.
> If you want to default to SunJSSE everywhere you can do so by delegating
> the calls in these methods to SunJSSE provider.
>
> TrustManagerFactory tmf = TrustManagerFactory
> .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
> tmf.chekServerTrusted()
> or use
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> TrustManagerFactory.html#getInstance(java.lang.String)if you want a
> specific provider.
>
> -Harsha
>
> On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada < maulin.vasavada@gmail.
> com>
> wrote:
>
> Okay, so I take that you guys agree that I have to write a 'custom'
> algorithm and a provider to make it work , correct?
>
> Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
> implementation per say , ..." , I diagree. You can refer to https://
>
> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
>
> SpiffeTrustManager.java#L91 <http://spiffetrustmanager.java/#L91> and
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>
> provider/CertificateUtils.java#L100
>
> "that code" is the customization you have for the custom way to check
> something on top of regular checks. That method is NOT doing custom
> truststore loading. It is validating/verifying something in the
>
> "custom"
>
> way with spiffeId.
> I bet that without that you won't have a need of the custom algorithm
>
> in
>
> the first place.
>
> Let me know if you agree to this.
>
> Thanks
> Maulin
>
> On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri
>
> wrote:
>
> Hi Maulin, thanks for the discussion. As Harsha pointed out, to use the
&g
Re: [VOTE] KIP-352: Distinguish URPs caused by reassignment
+1 (binding) Thanks, Harsha On Wed, Aug 21, 2019 at 4:23 PM, Robert Barrett < bob.barr...@confluent.io > wrote: > > > > +1 (non-binding) > > > > This will be great to have, thanks Jason! > > > > On Wed, Aug 21, 2019 at 4:29 AM Manikumar < manikumar. reddy@ gmail. com ( > manikumar.re...@gmail.com ) > wrote: > > >> >> >> +1 (binding). >> >> >> >> Thanks for the KIP. LGTM. >> >> >> >> On Wed, Aug 21, 2019 at 3:12 PM Satish Duggana < satish. duggana@ gmail. com >> ( satish.dugg...@gmail.com ) > wrote: >> >> >>> >>> >>> Hi Jason, >>> +1 (non binding) Thanks for the KIP! >>> >>> >>> >>> Do we need to have a separate JIRA to update the docs as it introduces >>> >>> >> >> >> >> new >> >> >>> >>> >>> metrics and a change in behavior for the existing metric? >>> >>> >>> >>> On Wed, Aug 21, 2019 at 2:41 PM Mickael Maison < mickael. maison@ gmail. com >>> ( mickael.mai...@gmail.com ) >>> >>> >>> >>> wrote: >>> >>> >>>> >>>> >>>> +1 (non binding) >>>> Thanks Jason >>>> >>>> >>>> >>>> On Wed, Aug 21, 2019 at 8:15 AM David Jacot < djacot@ confluent. io ( >>>> dja...@confluent.io ) > >>>> >>>> >>> >>> >> >> >> >> wrote: >> >> >>> >>>> >>>>> >>>>> >>>>> +1 (non-binding) >>>>> >>>>> >>>>> >>>>> Thanks for the KIP! >>>>> >>>>> >>>>> >>>>> Best, >>>>> David >>>>> >>>>> >>>>> >>>>> On Tue, Aug 20, 2019 at 7:55 PM Jason Gustafson < jason@ confluent. io ( >>>>> ja...@confluent.io ) > >>>>> >>>>> >>>> >>>> >>>> >>>> wrote: >>>> >>>> >>>>> >>>>>> >>>>>> >>>>>> Hi All, >>>>>> >>>>>> >>>>>> >>>>>> I'd like to start a vote on KIP-352, which is a follow-up to >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> >> >> KIP-455 >> >> >>> >>>> >>>> >>>> to fix >>>> >>>> >>>>> >>>>>> >>>>>> >>>>>> a long-known shortcoming of URP reporting and to improve >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> >> >> reassignment >> >> >>> >>>> >>>>> >>>>>> >>>>>> >>>>>> monitoring: >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> >> >> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ >> KIP-352%3A+Distinguish+URPs+caused+by+reassignment >> ( >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-352%3A+Distinguish+URPs+caused+by+reassignment >> ) >> >> >>> >>>> >>>>> >>>>>> >>>>>> >>>>>> . >>>>>> >>>>>> >>>>>> >>>>>> Note that I have added one new metric following the discussion. It >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> seemed >>>> >>>> >>>>> >>>>>> >>>>>> >>>>>> useful to have a lag metric for reassigning partitions. >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Jason >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > > >
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Maulin,
The code parts you are pointing are specific for Spiffe and if
you are talking about validate method which uses PKIX check like any
other provider does.
If you want to default to SunJSSE everywhere you can do so by delegating
the calls in these methods to SunJSSE provider.
TrustManagerFactory tmf = TrustManagerFactory
.getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
tmf.chekServerTrusted()
or use
https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#getInstance(java.lang.String)if
you want a specific provider.
-Harsha
On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada
wrote:
> Okay, so I take that you guys agree that I have to write a 'custom'
> algorithm and a provider to make it work , correct?
>
> Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
> implementation per say , ..." , I diagree. You can refer to https://
> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> SpiffeTrustManager.java#L91 and
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> provider/CertificateUtils.java#L100
>
> "that code" is the customization you have for the custom way to check
> something on top of regular checks. That method is NOT doing custom
> truststore loading. It is validating/verifying something in the "custom"
> way with spiffeId.
> I bet that without that you won't have a need of the custom algorithm in
> the first place.
>
> Let me know if you agree to this.
>
> Thanks
> Maulin
>
> On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri wrote:
>
> Hi Maulin, thanks for the discussion. As Harsha pointed out, to use the
> KIP492, you need to create a new provider, register a *new* custom
> algorithm for your keymanager and trustmanager factory implementations.
> After this, the kafka server configuration can be done as given below
>
> # Register the provider class with custom algorithm, say CUSTOM security.
> provider.classes=com.company.security.CustomProvider
> <http://security.provider.classes=com.company.security.customprovider/>
>
> # Register the keymanager and trustmanager algorithms
> # These algorithms indicate that the Keymanager and Trustmanagers
> registered under the algorithm "CUSTOM" needs to be used
> ssl.trustmanager.algorithm=CUSTOM
> ssl.keymanager.algorithm=CUSTOM
>
> And the customprovider looks like this...
>
> public class CustomProvider extends Provider {
> public CustomProvider() {
> super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
> super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
> super.put("TrustManagerFactory.CUSTOM",
> "customTrustManagerFactory");
> }
> }
>
> The PR for this is in review and can be found here: https://github.com/
> apache/kafka/pull/7090
> This PR includes the fixed insertProviderAt function call.
>
> On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani
> wrote:
>
> Answers are in-line
>
> On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada < maulin.vasavada@gmail.
> com
>
> wrote:
>
> Hi Colin
>
> When I refer to "standard" or "custom" algorithms I am following Java
> security Provider Terminology. You can refer to
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> StandardNames.html#TrustManagerFactory link I provided earlier in the
> emails. It says PKIX is the default Algorithm for TrustManagerFactory.
>
> 1. For SPIFFE, I am not sure why you are saying 'it does not implement
> custom algorithms' because the following file clearly indicates that it
> does use custom algorithm-
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>
> provider/SpiffeProvider.java#L17
>
> Algorithm value:
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>
> provider/SpiffeProviderConstants.java#L6
>
> @Harsha do you want to chime in since you use that provider?
>
> Here the "Custom" Algorithm is not an implementation per say , rather
>
> used
>
> to invoke the custom trust store factory and key manager factory. You are
> not moving away from "standard" alogrithms that are available.
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> provider/SpiffeTrustManager.java
>
> As you can see it delegates all the calls of verification of certificate
>
> to
>
> the default implementation available.
> So in our implementation we still use PKIX to verify the certificate
> chain. So you are not losing anything here and Spiffe is not reimplementing
> the verification pro
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Answers are in-line On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada wrote: > Hi Colin > > > When I refer to "standard" or "custom" algorithms I am following Java > security Provider Terminology. You can refer to > https://docs.oracle.com/javase/7/docs/technotes/guides/security/ > StandardNames.html#TrustManagerFactory link I provided earlier in the > emails. It says PKIX is the default Algorithm for TrustManagerFactory. > > > 1. For SPIFFE, I am not sure why you are saying 'it does not implement > custom algorithms' because the following file clearly indicates that it > does use custom algorithm- > > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ > provider/SpiffeProvider.java#L17 > > > Algorithm value: > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ > provider/SpiffeProviderConstants.java#L6 > > > @Harsha do you want to chime in since you use that provider? > Here the "Custom" Algorithm is not an implementation per say , rather used to invoke the custom trust store factory and key manager factory. You are not moving away from "standard" alogrithms that are available. https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/SpiffeTrustManager.java As you can see it delegates all the calls of verification of certificate to the default implementation available. So in our implementation we still use PKIX to verify the certificate chain. So you are not losing anything here and Spiffe is not reimplementing the verification process. 2. I already mentioned in my 3rd point, in my previous post, why using > ssl.provider does NOT work. I updated KIP-486 in "rejected alternatives" > also why ssl.provider does not work. > As mentioned before , provider is the right way to go. I am still not understanding the gap is. If I understand correctly your argument is , provider is going to ask to implement a custom algorithm. My answer to that is , no you are not re-implementing the algorithm. Please check the above link , TrustManager implementation it delegates the calls back. There is no need to implement your own here. 3. Security.insertProviderAt() comments were based on assumption if KIP-492 > changes are done and we use that mechanism to configure providers instead > of ssl.provider configuration. > KIP-492 has patch available and going through review. Can you read my all the points, I mentioned in my previous post, very > carefully? I am covering all the aspects in explaining. I am open to still > discuss more to clarify any doubts. > "3. If we just use existing ssl.provider kafka configuration then our provider will be used in SSLContext.getInstance(protocol, provider) call in SslFactory.java <http://sslfactory.java/> and if our provider does not have implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we tested it). Example: In MyProvider sample above you see that I didn't add SSLContext.TLSv1 as "Service+Algorithm" and that didn't work for me. In SPIFFE provider you don't have this challenge since you are planning to bypass ssl.provider as you mention in the KIP-492." Yes here you need to pass the protocol that your KeyManager/TrustManager registered with and in no way its deviating from TLS RFC spec. https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/javax/net/ssl/SSLContext.java#L134 My suggestion here is for you to implement a simple Security Provider as you did before and register a Algorithm. You can use the existing implementation in SpiffeProvider and plug in changes where you need to retrieve the certificates from by making RPC call. Run an end-to-end test with Kafka broker coming up with your provider making calls to RPC call. You do need to pass the "custom algorithm" that you registered in your key manager to invoke the provider. I think your concern is this approach is replacing the existing known ciphersuites and certificate validation provided by java. Which its not. Now test the TLS connection you can do so via openssl -s_client options to test if everything else is working. I am happy to share configs that we used if you are interested. Thanks, Harsha > Thanks > Maulin > > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe wrote: > > Hi Maulin, > > A lot of JSSE providers don't implement custom algorithms. Spire is a good > example of a JSSE provider that doesn't, and yet is still useful to many > people. Your JSSE provider can work fine even if it doesn't implement a > custom algorithm. > > > Maybe I'm missing something, but I don't understand the discussion of > Security.insertProviderAt() that you included. SslEngineBuilder doesn't use > that API to get the security provider. Instead, it calls > "SSLContext.getInsta
Re: [VOTE] KIP-504 - Add new Java Authorizer Interface
+1 (binding). Thanks, Harsha On Sat, Aug 17, 2019 at 2:53 AM, Manikumar wrote: > Hi, > > +1 (binding). > > Thanks for the KIP. LGTM. > > Regards, > Manikumar > > On Sat, Aug 17, 2019 at 4:41 AM Colin McCabe wrote: > > +1 (binding) > > Thanks, Rajini! > > best, > Colin > > On Fri, Aug 16, 2019, at 09:52, Rajini Sivaram wrote: > > Hi all, > > I would like to start the vote for KIP-504: > > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-504+-+Add+new+Java+Authorizer+Interface > > This KIP replaces the Scala Authorizer API with a new Java API similar to > other pluggable APIs in the broker and addresses known limitations in the > existing API. > > Thanks for all the feedback! > > Regards, > > Rajini > >
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Thanks for the KIP Mickael. LGTM +1 (binding). -Harsha On Wed, Aug 14, 2019 at 1:10 PM, Colin McCabe wrote: > Thanks, Mickael. +1 (binding) > > best, > Colin > > On Wed, Aug 14, 2019, at 12:07, Gabor Somogyi wrote: > > +1 (non-binding) > I've read it through in depth and as Jungtaek said Spark can make good use > of it. > > On Wed, 14 Aug 2019, 17:06 Jungtaek Lim, wrote: > > +1 (non-binding) > > I found it very useful for Spark's case. (Discussion on KIP-505 described > it.) > > Thanks for driving the effort! > > 2019년 8월 14일 (수) 오후 8:49, Mickael Maison 님이 작성: > > Hi Guozhang, > > Thanks for taking a look. > > 1. Right, I updated the titles of the code blocks > > 2. Yes that's a good idea. I've updated the KIP > > Thank you > > On Wed, Aug 14, 2019 at 11:05 AM Mickael Maison > wrote: > > Hi Colin, > > Thanks for raising these 2 valid points. I've updated the KIP > > accordingly. > > On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang > > wrote: > > Hi Mickael, > > Thanks for the KIP! > > Just some minor comments. > > 1. Java class names are stale, e.g. "CommitOffsetsOptions.java > <http://commitoffsetsoptions.java/>" > > should > > be > > "AlterOffsetsOptions". > > 2. I'd suggest we change the future structure of "AlterOffsetsResult" > > to > > *KafkaFuture>>* > > This is because we will have a hierarchy of two-layers of errors > > since > > we > > need to find out the group coordinator first and then issue the > > commit > > offset request (see e.g. the ListConsumerGroupOffsetsResult which > > exclude > > partitions that have errors, or the DeleteMembersResult as part of > > KIP-345). > > If the discover-coordinator returns non-triable error, we would set > > it > > on > > the first layer of the KafkaFuture, and the per-partition error would > > be > > set on the second layer of the KafkaFuture. > > Guozhang > > On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe > > wrote: > > Hi Mickael, > > Considering that KIP-496, which adds a way of deleting consumer > > offsets > > from AdminClient, looks like it is going to get in, this seems like > functionality we should definitely have. > > For alterConsumerGroupOffsets, is the intention to ignore > > partitions > > that > > are not specified in the map? If so, we should specify that in the > > JavaDoc. > > isolationLevel seems like it should be an enum rather than a > > string. The > > existing enum is in org.apache.kafka.common.requests, so we should > > probably > > create a new one which is public in org.apache.kafka.clients.admin. > > best, > Colin > > On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote: > > Bumping this thread once again > > Ismael, have I answered your questions? > While this has received a few non-binding +1s, no committers have voted > yet. If you have concerns or questions, please let me know. > > Thanks > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison > wrote: > > Bumping this thread as it's been a couple of weeks. > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison < > > mickael.mai...@gmail.com> wrote: > > Thanks Ismael for the feedback. I think your point has 2 > > parts: > > - Having the reset functionality in the AdminClient: The fact we have a > command line tool illustrate that this > > operation > > is > > relatively common. I seems valuable to be able to perform > > this > > operation directly via a proper API in addition of the CLI > > tool. > > - Sending an OffsetCommit directly instead of relying on > > KafkaConsumer: > > The KafkaConsumer requires a lot of stuff to commit offsets. > > Its > > group > > cannot change so you need to start a new Consumer every time, > > that > > creates new connections and overal sends more requests. Also > > there > > are > > already a bunch of AdminClient APIs that have logic very > > close to > > what needs to be done to send a commit request, keeping the > > code > > small > > and consistent. > > I've updated the KIP with these details and moved the 2nd > > part > > to > > "Proposed changes" as it's more an implementation detail. > > I hope this answers your question > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma < > > isma...@gmail.com > > wrote: > > The KIP doesn't discuss the option of using KafkaConsumer > > directly > > as far > > as I can tell. We have
Re: [VOTE] KIP-503: deleted topics metric
+1 (binding) Thanks, Harsha On Tue, Aug 13, 2019 at 12:08 PM, David Arthur wrote: > Hello all, > > I'd like to start the vote on KIP-503 > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-503%3A+Add+metric+for+number+of+topics+marked+for+deletion > > Thanks! > David >
Re: [DISCUSS] KIP-505 : Add new public method to only update assignment metadata in consumer
Hi Jungtaek, Have you looked into this interface https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/MetadataUpdater.java . Right now its not a public interface but does the methods available in this interface work for your needs? . The DefaultMeatadataUpdater responsible for making the metadata requests to brokers https://github.com/apache/kafka/blob/26814e060e98f9674127be13a28ce41a21ca6b3c/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L958 and if it can be invoked from client methods does that solve your requirements? Thanks, Harsha On Mon, Aug 12, 2019 at 2:53 PM, Jungtaek Lim wrote: > Thanks for the feedbacks Colin and Matthias. > > I agree with you regarding getting topics and partitions via AdminClient, > just curious how much the overhead would be. Would it be lighter, or > heavier? We may not want to list topics in regular intervals - in plan > phase we want to know up-to-date information so that the calculation from > Spark itself makes sense. > > On the other hands I'm not seeing any information regarding offset in > current AdminClient, which is also one of reason we leverage consumer and > call poll(0). Colin, as you mentioned there're KIPs addressing this, could > you refer KIPs so that we can see whether it would work for our case? > Without support of this we cannot replace our usage of consumer/poll with > AdminClient. > > ps. IMHO it seems to be helpful if there's overloaded `listTopics` which > receives regex same as consumer subscription via pattern. We would like to > provide same behavior what Kafka is basically providing as a source. > > On Tue, Aug 13, 2019 at 1:03 AM Matthias J. Sax > wrote: > > Thanks for the details Jungtaek! > > I tend to agree with Colin, that using the AdminClient seems to be the > better choice. > > You can get all topics via `listTopics()` (and you can refresh this > information on regular intervals) and match any pattern against the list of > available topics in the driver. > > As you use `assignment()` and store offsets in the Spark checkpoint, it > seems that using consumer group management is not a good fit for the use > case. > > Thoughts? > > -Matthias > > On 8/12/19 8:22 AM, Colin McCabe wrote: > > Hi, > > If there’s no need to consume records in the Spark driver, then the > > Consumer is probably the wrong thing to use. Instead, Spark should use > AdminClient to find out what partitions exist and where, manage their > offsets, and so on. There are some KIPs under discussion now that would add > the necessary APIs for managing offsets. > > Best, > Colin > > On Mon, Aug 12, 2019, at 07:39, Jungtaek Lim wrote: > > My feeling is that I didn't explain the use case for Spark properly and > hence fail to explain the needs. Sorry about this. > > Spark leverages the single instance of KafkaConsumer in the driver > > which is > > registered solely on the consumer group. This is used in the plan phase > > for > > each micro-batch to calculate the overall topicpartitions with its > > offset > > ranges for this batch, and split and assign (topicpartition, fromOffset, > untilOffset) to each input partition. After the planning is done and > > tasks > > are being distributed to executors, consumer per each input partition > > will > > be initialized from some executor (being assigned to the single > topicpartition), and pull the actual records. (Pooling consumers is > > applied > > for sure.) As plan phase is to determine the overall topicpartitions and > offset ranges to process, Spark is never interested on pulling the > > records > > in driver side. > > Spark mainly leverages poll(0) to get the latest assigned partitions and > adopt the changes or validate the expectation. That's not only use case > > for > > poll(0). Spark is also seeking the offset per topicpartition to the > earliest or the latest, or specific one (either provided by end user or > > the > > last committed offset) so that Spark can have actual offset or validate > > the > > provided offset. According to the javadoc (if I understand correctly), > > to > > get the offset immediately it seems to be required to call `poll` or > `position`. > > The way Spark interacts with Kafka in this plan phase in driver is > synchronous, as the phase should finish ASAP to run the next phase. > Registering ConsumerRebalanceListener and tracking the change will > > require > > some asynchronous handling which sounds to add unnecessary complexity. > Spark may be OK with deal with synchronous with timeout (that's what > methods in KafkaConsumer have been providing - they're
Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer
On Fri, Aug 09, 2019 at 11:12 AM, Ismael Juma wrote: > Hi all, > > A few points: > > 1. I think the way backwards compatibility is being used here is not > correct. Any functionality that is only enabled if set via a config is > backwards compatible. People may disagree with the functionality or the > config, but it's not a backwards compatibility issue. > We are talking about both broker and producer as a single entity and run by the same team/users. Allowing newer producer to create topics on a older broker when auto.create.topics.enable set to false, breaks server side contract that this config offered from the beginning. IMO, it clearly isn't backward compatible. User who set auto.create.topic.enable on broker will not be the same who will turn it on producer side . > 2. It's an interesting question if auto topic creation via the producer > should be a server driven choice or not. I can see the desire to have a > server-driven default, but it seems like this is often application > specific. Because the functionality is trivially available via AdminClient > (released 2 years ago), it's not quite possible to control what > applications do without the use of ACLs or policies today. > > > Producers & consumers are the majority of the clients in Kafka ecosystem. Just because AdminClient shipped a while back that doesn't mean all users adopting to it. To this day lot more users are aware of Producer & Consumer APIs and running them in production compare to AdminClient. > 3. Changing the create topics request in this way is highly unintuitive in > my opinion and it relies on every client to pass the new field. For > example, if librdkafka added auto create functionality by relying on their > AdminClient, it would behave differently than what is proposed here. > Forcing every client to implement this change when calling auto create from > the producer specifically seems odd > I am not sure why its unintuitive , protocols change. We add or upgrade the existing protocols all the time. Thanks, Harsha . > > Ismael > > On Thu, Aug 8, 2019 at 11:51 AM Jun Rao wrote: > > Hi, Justine, > > Thanks for the KIP. Overall, it seems to be a good improvement. > > However, I think Harsha's point seems reasonable. We had > auto.create.topics.enable config on the broker to allow admins to disable > topic creation from the producer/consumer clients before we had the > security feature. The need for that config is reduced with the security > feature, but may still be present since not all places have security > enabled. It's true that a non-secured environment is vulnerable to some > additional attacks, but producer/consumer are the most common way for a > user to interact with the broker. So, keeping that config for backward > compatibility could still be useful if it's not introducing too much effort > or extra confusion. > > > Here is a one potential alternative way that I was thinking. We add a new > field in the CreateTopicRequest to indicate whether it's from the producer > or not. If auto.create.topics.enable is false, CreateTopicRequest from the > producer will be rejected. We probably don't need to introduce the new > config (which seems a bit hard to explain) in the producer. Instead, the > new producer always uses MetadataRequest with AllowAutoTopicCreation set to > false to get the metadata and if the metadata is not present, send the new > CreateTopicRequest > (assuming the broker supports it) to try to create the topic > automatically. Whether the creation is allowed or not will be determined by > the broker. This will make the behavior backward compatible and we can > still achieve the main goal of the KIP, which is not relying on > MetadataRequest for topic creation. What do you think? > > > Thanks, > > Jun > > On Thu, Aug 8, 2019 at 1:34 AM M. Manna wrote: > > Hi, > > > If I may, perhaps you could simplify everything by using only > 'auto.create.topics.enable' as a value along with true. In other words, > > > the > > public interfaces section should only have > > [true,auto.create.topics.enable, > > false]. > > The reason for this is that auto.create.topics.enable is already known to > users as a "Server-SIde" config. So all you are saying is > > a) To avoid day 1 impact, it will follow whatever > > auto.create.topics.enable > > > value is set. > b) False means - no client side topic creation > c) True means client side topic creation. > > > It saves creating 2 more new strings :). But not too expensive anyway. > > Also, when you deprecate auto.create.topics.enable - you must provide > sufficient logic to ensure that things like rolling upgrade doesn't > temporarily break anything. I
Re: [VOTE] KIP-499 - Unify connection name flag for command line tool
+1 (binding). much needed!! On Thu, Aug 08, 2019 at 6:43 PM, Gwen Shapira wrote: > +1 (binding) THANK YOU. It would be +100 if I could. > > On Thu, Aug 8, 2019 at 6:37 PM Mitchell wrote: > > Hello Dev, > After the discussion I would like to start the vote for KIP-499 > > The following command line tools will have the `--bootstrap-server` > command line argument added: kafka-console-producer.sh, > kafka-consumer-groups.sh, kafka-consumer-perf-test.sh, > kafka-verifiable-consumer.sh, kafka-verifiable-producer.sh > > Thanks, > -Mitch > > -- > Gwen Shapira > Product Manager | Confluent > 650.450.2760 | @gwenshap > Follow us: Twitter | blog >
Re: Ask for contributor access and write permission on wiki
Hi Jungtaek, Gave you permissions on wiki. Please check. Thansk, Harsha On Thu, Aug 08, 2019 at 7:03 PM, Jungtaek Lim wrote: > Hi devs, > > I'd like to give a shot to make first contribution on Kafka community, as > I initiated thread on needs a new public API for metadata update only [1]. > > Could you please grant me contributor in JIRA as well as write permission > on wiki page? > > Thanks in advance! > Jungtaek Lim (HeartSaVioR) > > 1, > https://lists.apache.org/thread.html/ > 017cf631ef981ab1b494b1249be5c11d7edfe5f4867770a18188ebdc@%3Cdev.kafka. > apache.org%3E >
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Hi Maulin, With java security providers can be as custom you would like it to be. If you only want to to implement a custom way of loading the keystore and truststore and not implement any protocol/encryption handling you can leave them empty and no need to implement. Have you looked into the links I pasted before? https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java Can you please tell me which methods are too complex in above to implement or unnecessary? You are changing anything in SSL/TLS implementations provided by All of the implementations delegating the checks to the default implementation anyway. Spire agent is an example, its nothing but a GRPC server listening on a unix domain socket . Above code is making a RPC call to the local daemon to get the certificate and keys. The mechanics are pretty much same as what you are asking for. Thanks, Harsha On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada wrote: > Imagine a scenario like - We know we have a custom KMS and as a Kafka owner > we want to comply to using that KMS source to load keys/certs. As a Kafka > owner we know how to integrate with KMS but doesn't necessarily have to > know anything about cipher suites, algorithms, and SSL/TLS implementation. > Going the Provider way requires to know lot more than we should, isn't it? > Not that we would have concern/shy-away knowing those details - but if we > don't have to - why should we? > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada > wrote: > > > Hi Harsha > > > > We don't have spire (or similar) agents and we do not have keys/certs > > locally on any brokers. To elaborate more on my previous email, > > > > I agree that Java security Providers are used in much broader sense - to > > have a particular implementation of an algorithm, use specific cipher > > suites for SSL , OR in our current team's case have a particular way to > > leverage pre-generated SSL sessions. However, the scope of our KIP (486) > is > > much restricted than that. We merely intend to provide a custom > > keystore/truststore for our SSL connections and not really worry about > > underlying specific SSL/TLS implementation. This simplifies it a lot for > > us to keep the concerns separate and clear. > > > > I feel our approach is more complimentary such that it allows for using > > keystores of choice while retaining the flexibility to use any > > underlying/available Provider for actually making the SSL call. > > > > We agree with KIP-492's approach based on Providers (and Java's > > recommendation), but also strongly believe that our approach can > compliment > > it very effectively for reasons explained above. > > > > Thanks > > Maulin > > > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani > > wrote: > > > >> Hi Maulin, > >> > >> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada < > >> maulin.vasav...@gmail.com> > >> wrote: > >> > >> > Hi Harsha > >> > > >> > The reason we rejected the SslProvider route is that - we only needed > a > >> > custom way to load keys/certs. Not touch any policy that existing > >> Providers > >> > govern like SunJSSE Provider. > >> > > >> > >> We have exactly the same requirements to load certs and keys through > spire > >> agent. We used security.provider to do that exactly. I am not sure why > you > >> would be modifying any policies provided by default SunJSSE provider. > Can > >> you give me an example of having custom provider that will override an > >> existing policy in SunJSSE provider. > >> > >> As pointed out earlier, this kip > >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config > >> allows > >> you to load security.provider through config. > >> Take a look at the examples I gave before > >> > >> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java > >> It registers KeyManagerFactory and TrustManagerFactory and Keystore > >> algorithm. > >> Implement your custom way of loading Keystore in here > >&
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Hi Maulin, On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada wrote: > Hi Harsha > > The reason we rejected the SslProvider route is that - we only needed a > custom way to load keys/certs. Not touch any policy that existing Providers > govern like SunJSSE Provider. > We have exactly the same requirements to load certs and keys through spire agent. We used security.provider to do that exactly. I am not sure why you would be modifying any policies provided by default SunJSSE provider. Can you give me an example of having custom provider that will override an existing policy in SunJSSE provider. As pointed out earlier, this kip https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config allows you to load security.provider through config. Take a look at the examples I gave before https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java It registers KeyManagerFactory and TrustManagerFactory and Keystore algorithm. Implement your custom way of loading Keystore in here https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java and Trust manager like here https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java In your Kafka client you can set the security.provider to your custom implementation and with this fix https://issues.apache.org/jira/browse/KAFKA-8191 you can set keyManagerAlgorigthm and trustManagerAlgorithm configs. All of this is in your clients and broker side and do not need to touch any policy changes at JVM level. You'll register the providers in the priority order and can still have SunJSSE provider and have your custom provider to implement the key and trust managers. The ask here is different than KIP-492. We don't have any need to > modify/specify the algorithm parameter. Does that make sense? > The ask in KIP is introducing new interfaces where the KIP's goal/motivation can be achieved through the security.provider and we worked on similar goal without touching any Keystore or Truststore interfaces. My advise is against changing or introducing new interfaces when it can work through security.provider. Thanks, Harsha Thanks > Maulin > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani > wrote: > > In your KIP you added security. provider as rejected alternative and > specified "its not the correct way". Do you mind explaining why its not? I > didn't find any evidence in Java docs to say so. Contrary to your statement > it does say in the java docs > " However, please note that a provider can be used to implement any > security service in Java that uses a pluggable architecture with a choice > of implementations that fit underneath." > > Java Security Providers have been used by other projects to provide such > integration . I am not sure if you looked into Spiffe project to > efficiently distribute certificates but here is an example of Java provider > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider. > java which > obtains certificates from local daemons. > These integrations are being used in Tomcat, Jetty etc.. We are also using > Security provider to do the same in our Kafka clusters. So unless I see > more evidence why security.provider doesn't work for you adding new > interfaces while there exists more cleaner way of achieving the goals of > this KIP is unnecessary and breaks the well known security interfaces > provided by Java itself. > > Thanks, > Harsha > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani > wrote: > > Hi Maulin, > Not sure if you looked at my previous replies. This > > changes > > are not required as there is already security Provider to do what you are > proposing. This KIP https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config also > addresses easy registration of such providers. > > Thanks, > Harsha > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada com> wrote: > > Bump! Can somebody please review this? > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada < > > maulin.vasav...@gmail.com> > > wrote: > > Bump! Can somebody please review this? > >
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
In your KIP you added security. provider as rejected alternative and specified "its not the correct way". Do you mind explaining why its not? I didn't find any evidence in Java docs to say so. Contrary to your statement it does say in the java docs " However, please note that a provider can be used to implement any security service in Java that uses a pluggable architecture with a choice of implementations that fit underneath." Java Security Providers have been used by other projects to provide such integration . I am not sure if you looked into Spiffe project to efficiently distribute certificates but here is an example of Java provider https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java which obtains certificates from local daemons. These integrations are being used in Tomcat, Jetty etc.. We are also using Security provider to do the same in our Kafka clusters. So unless I see more evidence why security.provider doesn't work for you adding new interfaces while there exists more cleaner way of achieving the goals of this KIP is unnecessary and breaks the well known security interfaces provided by Java itself. Thanks, Harsha On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani wrote: > Hi Maulin, >Not sure if you looked at my previous replies. This changes > are not required as there is already security Provider to do what you are > proposing. This KIP https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config also > addresses easy registration of such providers. > > Thanks, > Harsha > > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada com> wrote: > > Bump! Can somebody please review this? > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada > wrote: > > Bump! Can somebody please review this? > >
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Hi Maulin, Not sure if you looked at my previous replies. This changes are not required as there is already security Provider to do what you are proposing. This KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config also addresses easy registration of such providers. Thanks, Harsha On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada wrote: > Bump! Can somebody please review this? > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada > wrote: > > Bump! Can somebody please review this? > >
Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer
On Wed, Aug 07, 2019 at 9:50 AM, Colin McCabe wrote: > On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote: > > On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe < cmcc...@apache.org > > wrote: > > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote: > > Hi Colin, > "Hmm... I'm not sure I follow. Users don't have to build their own > tooling, right? They can use any of the shell scripts that we've shipped in > the last few releases. For example, if any of your users run it, this shell > script will delete all of the topics from your non-security-enabled > cluster: > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) > --bootstrap-server localhost:9092 --list 2>/dev/null > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) > --bootstrap-server localhost:9092 --delete > --topic > > They will need to fill in the correct bootstrap servers list, of course, > not localhost. This deletion script will work on some pretty old brokers, > even back to the 0.10 releases. It seems a little odd to trust your users > with this power, but not trust them to avoid changing a particular > configuration key." > > The above will blocked by the server if we set delete.topic.enable to > false and thats exactly what I am asking for. > > Hi Harsha, > > I was wondering if someone was going to bring up that configuration :) > > it's an interesting complication, but globally disabling topic deletion is > not very practical for most use-cases. > > In any case, there are plenty of other bad things that users with full > permissions can do that aren't blocked by any server configuration. For > example, they can delete every record in every topic. I can write a script > for that too, and there's no server configuration you can set to disable > it. Or I could simply create hundreds of thousands of topics, until cluster > performance becomes unacceptable (this will be even more of a problem if > someone configured delete.topic.enable as false). Or publish bad data to > every topic, etc. etc. > > The point I'm trying to make here is that you can't rely on these kind of > server-side configurations for security. At most, they're a way to set up > certain very simple policies. But the policies are so simple that they're > hardly ever useful any more. > > For example, if the problem you want to solve is that you want a user to > only be able to create 50 topics and not delete anyone else's topics, you > can solve that with a CreateTopicsPolicy that limits the number of topics, > and some ACLs. There's no combination of auto.create.topics.enable and > delete.topic.enable that will help here. > > Hi Colin, > > Well you gave the example that a user can delete the topics > just by running that script :). > > I understand there are open APIs in Kafka and can lead to rogue clients > taking advantage of it without proper security in place. > > What I am asking so far in this thread is , this KIP is changing the > producer behavior and its not backward compatible. > > The change is backwards compatible. The default will still be server-side > topic auto-creation, just like now. > You will have to specifically change the producer config to get the new > behavior. > I disagree. Today server can turn off the topic creation neither producer or consumer can create a topic. With this KIP , producer can create a topic by turning on client side config when server side config is turned off. We can still achieve > the main goal of the KIP which is to change MetadataRequest creating > topics and send a CreateTopicRequest from Producer and also keep the server > side config to have precedence. This KIP originally written to have server > side preference and there is not much explanation why it changed to have > Producer side preference. > > Arguing that AdminClient can do that and so we are going to make Producer > do the same doesn't make sense. > > "The downside is that if we wanted to check a server side configuration > before sending the create topics request, the code would be more complex. > The behavior would also not be consistent with how topic auto-creation is > handled in Kafka Streams." > > I am not sure why you need to check server side configuration before > sending create topics request. A user enables producer side config to > create topics. > Producer sends a request to the broker and if the broker has > auto.topic.create.enable to true (default) it will allow creation of > topics. If it set to false it returns error back to the client. > > auto.topic.create.enable has never affected CreateTopicsRequest. If you > submit a CreateTopicsRequest and you are authorized, a topic will be > created, rega
Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer
On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe < cmcc...@apache.org > wrote: > > > > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote: > > > >> >> >> Hi Colin, >> "Hmm... I'm not sure I follow. Users don't have to build their own >> tooling, right? They can use any of the shell scripts that we've shipped >> in the last few releases. For example, if any of your users run it, this >> shell script will delete all of the topics from your non-security-enabled >> cluster: >> >> >> >> ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) --bootstrap-server >> localhost:9092 --list 2>/dev/null >> | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) >> --bootstrap-server >> localhost:9092 --delete >> --topic >> >> >> >> They will need to fill in the correct bootstrap servers list, of course, >> not localhost. This deletion script will work on some pretty old brokers, >> even back to the 0.10 releases. It seems a little odd to trust your users >> with this power, but not trust them to avoid changing a particular >> configuration key." >> >> >> >> >> The above will blocked by the server if we set delete.topic.enable to >> false and thats exactly what I am asking for. >> >> >> > > > > Hi Harsha, > > > > > I was wondering if someone was going to bring up that configuration :) > > > > > it's an interesting complication, but globally disabling topic deletion is > not very practical for most use-cases. > > > > > In any case, there are plenty of other bad things that users with full > permissions can do that aren't blocked by any server configuration. For > example, they can delete every record in every topic. I can write a script > for that too, and there's no server configuration you can set to disable > it. Or I could simply create hundreds of thousands of topics, until > cluster performance becomes unacceptable (this will be even more of a > problem if someone configured delete.topic.enable as false). Or publish > bad data to every topic, etc. etc. > > > > > The point I'm trying to make here is that you can't rely on these kind of > server-side configurations for security. At most, they're a way to set up > certain very simple policies. But the policies are so simple that they're > hardly ever useful any more. > > > > > For example, if the problem you want to solve is that you want a user to > only be able to create 50 topics and not delete anyone else's topics, you > can solve that with a CreateTopicsPolicy that limits the number of topics, > and some ACLs. There's no combination of auto.create.topics.enable and > delete.topic.enable that will help here. > > > > Hi Colin, Well you gave the example that a user can delete the topics just by running that script :). I understand there are open APIs in Kafka and can lead to rogue clients taking advantage of it without proper security in place. What I am asking so far in this thread is , this KIP is changing the producer behavior and its not backward compatible. We can still achieve the main goal of the KIP which is to change MetadataRequest creating topics and send a CreateTopicRequest from Producer and also keep the server side config to have precedence. This KIP originally written to have server side preference and there is not much explanation why it changed to have Producer side preference. Arguing that AdminClient can do that and so we are going to make Producer do the same doesn't make sense. > > > > > > > >> >> >> "The downside is that if we wanted to check a server side configuration >> before sending the create topics request, the code would be more complex. >> The behavior would also not be consistent with how topic auto-creation is >> handled in Kafka Streams." >> >> >> >> >> I am not sure why you need to check server side configuration before >> sending create topics request. A user enables producer side config to >> create topics. >> Producer sends a request to the broker and if the broker has >> auto.topic.create.enable to true (default) it will allow creation of >> topics. If it set to false it returns error back to the client. >> >> > > > > auto.topic.create.enable has never affected CreateTopicsRequest. If you > submit a CreateTopicsRequest and you are authorized, a topic will be > created, regardless of what the value of auto.topic.create.enable is. This > behavior goes back a long way, to about 2016
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Thanks for the KIP Rajini. Quick thought, it would be good to have a common module outside of clients that only applies to server side interfaces & changes. It looks like we are increasingly in favor of using Java interface for pluggable modules on the broker side. Thanks, Harsha On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram wrote: > Hi all, > > I have created a KIP to replace the Scala Authorizer API with a new Java > API: > > - > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-504+-+Add+new+Java+Authorizer+Interface > > This is replacement for KIP-50 which was accepted but never merged. Apart > from moving to a Java API consistent with other pluggable interfaces in the > broker, KIP-504 also attempts to address known limitations in the > authorizer. If you have come across other limitations that you would like > to see addressed in the new API, please raise these on the discussion > thread so that we can consider those too. All suggestions and feedback are > welcome. > > Thank you, > > Rajini >
Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer
Hi Colin, "Hmm... I'm not sure I follow. Users don't have to build their own tooling, right? They can use any of the shell scripts that we've shipped in the last few releases. For example, if any of your users run it, this shell script will delete all of the topics from your non-security-enabled cluster: ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list 2>/dev/null | xargs -l ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic They will need to fill in the correct bootstrap servers list, of course, not localhost. This deletion script will work on some pretty old brokers, even back to the 0.10 releases. It seems a little odd to trust your users with this power, but not trust them to avoid changing a particular configuration key." The above will blocked by the server if we set delete.topic.enable to false and thats exactly what I am asking for. "The downside is that if we wanted to check a server side configuration before sending the create topics request, the code would be more complex. The behavior would also not be consistent with how topic auto-creation is handled in Kafka Streams." I am not sure why you need to check server side configuration before sending create topics request. A user enables producer side config to create topics. Producer sends a request to the broker and if the broker has auto.topic.create.enable to true (default) it will allow creation of topics. If it set to false it returns error back to the client. I don't see how this behavior will be different in Kafka streams. By default server allows the topic creation and with this KIP, It will only allow creation of topic when both producer and server side are turned on. Its exactly the same behavior in KIP-361. "In general, it would be nice if we could keep the security and access control model simple and not introduce a lot of special cases and exceptions. Kafka has basically converged on using ACLs and CreateTopicPolicy classes to control who can create topics and where. Adding more knobs seems like a step backwards, especially when the proposed knobs don't work consistently across components, and don't provide true security." This is not about access control at all. Shipping sane defaults should be prioritized. We keep talking about CreateTopicPolicy and yet we don't have default one and asking every user of Kafka implement their own doesn't make sense here. I am asking about exact behavior that we shipped for consumer side https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation I still didn't get any response why this behavior shouldn't be exactly like Kafka consumer and why do we want to have producer to overrider server side config and while not allowing consumer to do so. We are not even allowing the same contract and this will cause more confusion from the users standpoint. Thanks, Harsha On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe wrote: > On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote: > > Not sure how the AdminClient applies here, It is an external API and > > is not part of KafkaProducer so any user who updates to latest version of > > Kafka with this feature get to create the topics. > > They have to build a tooling around AdminClient allow themselves to > create > > topics. > > Hi Harsha, > > Hmm... I'm not sure I follow. Users don't have to build their own > tooling, right? They can use any of the shell scripts that we've shipped > in the last few releases. For example, if any of your users run it, this > shell script will delete all of the topics from your non-security-enabled > cluster: > > ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list 2>/dev/null > | xargs -l ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete > --topic > > They will need to fill in the correct bootstrap servers list, of course, > not localhost. This deletion script will work on some pretty old brokers, > even back to the 0.10 releases. It seems a little odd to trust your users > with this power, but not trust them to avoid changing a particular > configuration key. > > > There is no behavior in Kafka producer that allowed users to > > delete the topics or delete the records. So citing them as an > > example doesn't makes sense in this context. > > I think Kafka Streams is relevant here. After all, it's software that we > ship as part of the official Kafka release. And it auto-creates topics > even when auto.create.topics.enable is set to false on the broker. > > I think that auto.create.topics.enable was never intended as a security > setting to control access. It was intended as a way to disable the > broker-side auto-create feature specifically, because there were some > problems with that specific feature. Broker-side auto-cre
Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer
Hi Justin, Thanks for making changes. I still have concern that we are prioritizing producer config over server side which is breaking the backward compatibility of broker's auto.topic.create.enable as far as producer is concerned. Also https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation only allows creation of topic if both allow.auto.create.topics set to true and auto.create.topics.enable on the broker side set to true. This KIP is changing that contract for producers and allow them to take control even broker side is turned off. Why can't we have the same behavior. You can still achieve the goal for this KIP by allowing auto.create.topics.enable to take precedence and have a similar behavior to that of consumer. Thanks, Harsha On Tue, Aug 06, 2019 at 6:06 PM, Harsha Ch wrote: > Hi Colin, > There is no behavior in Kafka producer that allowed users to > delete the topics or delete the records. So > citing them as an example doesn't makes sense in this context. But there > is a functionality which allowed creation of topics if they don't exist in > the cluster and this behavior could be controlled by a config on the server > side. Now with this KIP we are allowing producer to make that decision > without any gateway on the server via configs. Any user who is not aware of > such changes > can accidentally create these topics and we are essentially removing a > config that exists in brokers today to block this accidental creation and > allowing clients to take control. > Not sure how the AdminClient applies here, It is an external API and > is not part of KafkaProducer so any user who updates to latest version of > Kafka with this feature get to create the topics. > They have to build a tooling around AdminClient allow themselves to create > topics. > I still didn't get any positives of not having server side configs? > if you want to turn them on and allow any client to create topics set the > default to true > and allow users who doesn't want to have this feature let them turn them > off. It will be the exact behavior as it is today, as far as producer is > concerned. I am not > understanding why not having server side configs to gateways such a hard > requirement and this behavior exists today. As far I am concerned this > breaks the backward compatibility. > > Thanks, > Harsha > > > On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe wrote: > > Hi Harsha, > > Thanks for taking a look. > > I'm not sure I follow the discussion about AdminClient. KafkaAdminClient > has been around for about 2 years at this point as a public class. There > are many programs that use it to automatically create topics. Kafka > Streams does this, for example. If any of your users run Kafka Streams, > they will be auto-creating topics, regardless of what setting you use for > auto.create.topics.enable. > > A big part of the annoyance of auto-topic creation right now is that it is > on by default. The new configuration proposed by KIP-487 wouldn't be. > Users would have to explicitly opt in to the new behavior of client-side > topic creation. If you run without security, you're already putting a huge > amount of trust in your users. For example, you trust them not to delete > records with the kafka-delete-records.sh command, or delete topics with > kafka-topics.sh. Trusting them not to set a certain config value seems > minor in comparison, right? > > best, > Colin > > On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote: > > Hi, > > Even with policies one needs to implement that, so for every user who > > doesn't want a producer to create topics or have limits around > partitions & > > replication factor they have to implement a policy. > > The KIP is changing the behavior , it might not be introducing the > > new functionality but it will enable producers to override the create > topic > > config settings on the broker. What I am asking for to provide a config > > that will disable auto creation of topics and if its enabled set some > sane > > defaults so that clients can create a topic with in those limits. I don't > > see how this not related to this KIP. > > If the server config options as I suggested doesn't interest you at > > least have a default CreateTopicPolices in place. > >To give an example, In our environment we disable the > > auto.create.topic.enable and force users to go through a centralized > > service as we want capture more details about the topic creation and > > requirements. With this KIP, a producer can create a topic with no > bounds. > > Another example max.message.s
Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer
Hi Colin, There is no behavior in Kafka producer that allowed users to delete the topics or delete the records. So citing them as an example doesn't makes sense in this context. But there is a functionality which allowed creation of topics if they don't exist in the cluster and this behavior could be controlled by a config on the server side. Now with this KIP we are allowing producer to make that decision without any gateway on the server via configs. Any user who is not aware of such changes can accidentally create these topics and we are essentially removing a config that exists in brokers today to block this accidental creation and allowing clients to take control. Not sure how the AdminClient applies here, It is an external API and is not part of KafkaProducer so any user who updates to latest version of Kafka with this feature get to create the topics. They have to build a tooling around AdminClient allow themselves to create topics. I still didn't get any positives of not having server side configs? if you want to turn them on and allow any client to create topics set the default to true and allow users who doesn't want to have this feature let them turn them off. It will be the exact behavior as it is today, as far as producer is concerned. I am not understanding why not having server side configs to gateways such a hard requirement and this behavior exists today. As far I am concerned this breaks the backward compatibility. Thanks, Harsha On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe wrote: > Hi Harsha, > > Thanks for taking a look. > > I'm not sure I follow the discussion about AdminClient. KafkaAdminClient > has been around for about 2 years at this point as a public class. There > are many programs that use it to automatically create topics. Kafka > Streams does this, for example. If any of your users run Kafka Streams, > they will be auto-creating topics, regardless of what setting you use for > auto.create.topics.enable. > > A big part of the annoyance of auto-topic creation right now is that it is > on by default. The new configuration proposed by KIP-487 wouldn't be. > Users would have to explicitly opt in to the new behavior of client-side > topic creation. If you run without security, you're already putting a huge > amount of trust in your users. For example, you trust them not to delete > records with the kafka-delete-records.sh command, or delete topics with > kafka-topics.sh. Trusting them not to set a certain config value seems > minor in comparison, right? > > best, > Colin > > On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote: > > Hi, > > Even with policies one needs to implement that, so for every user who > > doesn't want a producer to create topics or have limits around > partitions & > > replication factor they have to implement a policy. > > The KIP is changing the behavior , it might not be introducing the > > new functionality but it will enable producers to override the create > topic > > config settings on the broker. What I am asking for to provide a config > > that will disable auto creation of topics and if its enabled set some > sane > > defaults so that clients can create a topic with in those limits. I don't > > see how this not related to this KIP. > > If the server config options as I suggested doesn't interest you at > > least have a default CreateTopicPolices in place. > >To give an example, In our environment we disable the > > auto.create.topic.enable and force users to go through a centralized > > service as we want capture more details about the topic creation and > > requirements. With this KIP, a producer can create a topic with no > bounds. > > Another example max.message.size we define that at cluster level and one > > can override max.messsage.size at topic level, any misconfiguration at > this > > will cause service degradation. Its not always about the rogue clients, > > users can easily misconfigure and can cause an outage. > > Again we can talk about CreateTopicPolicy but without having a default > > implementation and asking everyone to implement their own while changing > > the behavior in producer doesn't make sense to me. > > > > Thanks, > > Harsha > > > > > > On Tue, Aug 06, 2019 at 7:41 AM, Ismael Juma wrote: > > > > > Hi Harsha, > > > > > > I mentioned policies and the authorizer. For example, with > > > CreateTopicPolicy, you can implement the limits you describe. If you > have > > > ideas of how that should be improved, please submit a KIP. My point is > that > > > this KIP is not introducing any new functionality with regards to what > &g
Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer
Hi, Even with policies one needs to implement that, so for every user who doesn't want a producer to create topics or have limits around partitions & replication factor they have to implement a policy. The KIP is changing the behavior , it might not be introducing the new functionality but it will enable producers to override the create topic config settings on the broker. What I am asking for to provide a config that will disable auto creation of topics and if its enabled set some sane defaults so that clients can create a topic with in those limits. I don't see how this not related to this KIP. If the server config options as I suggested doesn't interest you at least have a default CreateTopicPolices in place. To give an example, In our environment we disable the auto.create.topic.enable and force users to go through a centralized service as we want capture more details about the topic creation and requirements. With this KIP, a producer can create a topic with no bounds. Another example max.message.size we define that at cluster level and one can override max.messsage.size at topic level, any misconfiguration at this will cause service degradation. Its not always about the rogue clients, users can easily misconfigure and can cause an outage. Again we can talk about CreateTopicPolicy but without having a default implementation and asking everyone to implement their own while changing the behavior in producer doesn't make sense to me. Thanks, Harsha On Tue, Aug 06, 2019 at 7:41 AM, Ismael Juma wrote: > Hi Harsha, > > I mentioned policies and the authorizer. For example, with > CreateTopicPolicy, you can implement the limits you describe. If you have > ideas of how that should be improved, please submit a KIP. My point is that > this KIP is not introducing any new functionality with regards to what > rogue clients can do. It's using the existing protocol that is already > exposed via the AdminClient. So, I don't think we need to address it in > this KIP. Does that make sense? > > Ismael > > On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani > wrote: > > Ismael, > Sure AdminClient can do that and we should've shipped a config or use the > existing one to block that. Not all users are yet to upgrade to AdminClient > and start using that to cause issues yet. In shared environment we should > allow server to set sane defaults and not allow every client to go ahead > create random no.of topic/partitions and replication factor. Even if the > users want to allow topic creation proposed in the KIP , it makes sense to > have some guards against the no.of partitions and replication factor. > Authorizer is not always an answer to block requests and having users set > server side configs to protect a multi-tenant environment is required. In a > non-secure environment Authorizer is a blunt instrument either you end up > blocking everyone or allowing everyone. > I am asking to have server side that allow clients to create topics or not > , if they are allowed set a ceiling on max no.of partitions and > replication-factor. > > -Harsha > > On Mon, Aug 5 2019 at 8:58 PM, wrote: > > Harsha, > > Rogue clients can use the admin client to create topics and partitions. > ACLs and policies can help in that case as well as this one. > > Ismael > > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani > > wrote: > > Hi Justine, > Thanks for the KIP. > "When server-side auto-creation is disabled, client-side auto-creation > will try to use client-side configurations" > If I understand correctly, this KIP is removing any server-side blocking > client auto creation of topic? > if so this will present potential issue of rogue client creating ton of > topic-partitions and potentially bringing down the service for everyone > > or > > degrade the service itself. > By reading the KIP its not clear to me that there is a clear way to block > auto creation topics of all together from clients by server side config. > Server side configs of default topic, partitions should take higher > precedence and client shouldn't be able to create a topic with higher > > no.of > > partitions, replication than what server config specifies. > > Thanks, > Harsha > > On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan > wrote: > > Hi all, > I made some changes to the KIP. Hopefully this configuration change > > will > > make things a little clearer. > > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer > > Please let me know if you have any feedback or questions! > > Thank you, > Justine > > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe > > wrote: > > Hi Mickael, > > I think
Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer
Ismael, Sure AdminClient can do that and we should've shipped a config or use the existing one to block that. Not all users are yet to upgrade to AdminClient and start using that to cause issues yet. In shared environment we should allow server to set sane defaults and not allow every client to go ahead create random no.of topic/partitions and replication factor. Even if the users want to allow topic creation proposed in the KIP , it makes sense to have some guards against the no.of partitions and replication factor. Authorizer is not always an answer to block requests and having users set server side configs to protect a multi-tenant environment is required. In a non-secure environment Authorizer is a blunt instrument either you end up blocking everyone or allowing everyone. I am asking to have server side that allow clients to create topics or not , if they are allowed set a ceiling on max no.of partitions and replication-factor. -Harsha On Mon, Aug 5 2019 at 8:58 PM, wrote: > Harsha, > > Rogue clients can use the admin client to create topics and partitions. > ACLs and policies can help in that case as well as this one. > > Ismael > > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani wrote: > > Hi Justine, > Thanks for the KIP. > "When server-side auto-creation is disabled, client-side auto-creation > will try to use client-side configurations" > If I understand correctly, this KIP is removing any server-side blocking > client auto creation of topic? > if so this will present potential issue of rogue client creating ton of > topic-partitions and potentially bringing down the service for everyone or > degrade the service itself. > By reading the KIP its not clear to me that there is a clear way to block > auto creation topics of all together from clients by server side config. > Server side configs of default topic, partitions should take higher > precedence and client shouldn't be able to create a topic with higher no.of > partitions, replication than what server config specifies. > > > > > Thanks, > Harsha > > > > > > > > > On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan > wrote: > > > > > > Hi all, > > I made some changes to the KIP. Hopefully this configuration change will > > make things a little clearer. > > > > https://cwiki.apache.org/confluence/display/KAFKA/ > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer > > > > Please let me know if you have any feedback or questions! > > > > Thank you, > > Justine > > > > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe > wrote: > > > > Hi Mickael, > > > > I think you bring up a good point. It would be better if we didn't ever > > have to set up client-side configuration for this feature, and KIP-464 > > would let us skip this entirely. > > > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote: > > > > Hi Mickael, > > I agree that KIP-464 works on newer brokers, but I was a bit worried how > > things would play out on older brokers that* do not *have KIP 464 > > > > included. > > > > Is it enough to throw an error in this case when producer configs are > not > > specified? > > > > I think the right thing to do would be to log an error message in the > > client. We will need to have that capability in any case, to cover > > scenarios like the client trying to auto-create a topic that they don't > > have permission to create. Or a client trying to create a topic on a > broker > > so old that CreateTopicsRequest is not supported. > > > > The big downside to relying on KIP-464 is that it is a very recent > feature > > -- so recent that it hasn't even made its way to any official Apache > > release. It's scheduled for the upcoming 2.4 release in a few months. > > > > So if you view this KIP as a step towards removing broker-side > > auto-create, you might want to support older brokers just to accelerate > > adoption, and hasten the day when we can finally flip broker-side > > auto-create to off (or even remove it entirely). > > > > I have to agree, though, that having client-side configurations for > number > > of partitions and replication factor is messy. Maybe it would be worth > it > > to restrict support to post-KIP-464 brokers, if we could avoid creating > > more configs. > > > > best, > > Colin > > > > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison > > > wrote: > > > > Hi Justine, > > > > We can rely on KIP-464 which allows to omit the partition count or > > replication factor when c
Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer
Hi Justine, Thanks for the KIP. "When server-side auto-creation is disabled, client-side auto-creation will try to use client-side configurations" If I understand correctly, this KIP is removing any server-side blocking client auto creation of topic? if so this will present potential issue of rogue client creating ton of topic-partitions and potentially bringing down the service for everyone or degrade the service itself. By reading the KIP its not clear to me that there is a clear way to block auto creation topics of all together from clients by server side config. Server side configs of default topic, partitions should take higher precedence and client shouldn't be able to create a topic with higher no.of partitions, replication than what server config specifies. Thanks, Harsha On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan wrote: > Hi all, > I made some changes to the KIP. Hopefully this configuration change will > make things a little clearer. > > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer > > Please let me know if you have any feedback or questions! > > Thank you, > Justine > > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe wrote: > > Hi Mickael, > > I think you bring up a good point. It would be better if we didn't ever > have to set up client-side configuration for this feature, and KIP-464 > would let us skip this entirely. > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote: > > Hi Mickael, > I agree that KIP-464 works on newer brokers, but I was a bit worried how > things would play out on older brokers that* do not *have KIP 464 > > included. > > Is it enough to throw an error in this case when producer configs are not > specified? > > I think the right thing to do would be to log an error message in the > client. We will need to have that capability in any case, to cover > scenarios like the client trying to auto-create a topic that they don't > have permission to create. Or a client trying to create a topic on a broker > so old that CreateTopicsRequest is not supported. > > The big downside to relying on KIP-464 is that it is a very recent feature > -- so recent that it hasn't even made its way to any official Apache > release. It's scheduled for the upcoming 2.4 release in a few months. > > So if you view this KIP as a step towards removing broker-side > auto-create, you might want to support older brokers just to accelerate > adoption, and hasten the day when we can finally flip broker-side > auto-create to off (or even remove it entirely). > > I have to agree, though, that having client-side configurations for number > of partitions and replication factor is messy. Maybe it would be worth it > to restrict support to post-KIP-464 brokers, if we could avoid creating > more configs. > > best, > Colin > > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison > wrote: > > Hi Justine, > > We can rely on KIP-464 which allows to omit the partition count or > replication factor when creating a topic. In that case, the broker defaults > are used. > > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan > wrote: > > Michael, > That makes sense to me! > To clarify, in the current state of the KIP, the producer does not > > rely > > on > > the broker to autocreate--if the broker's config is disabled, then > > the > > producer can autocreate on its own with a create topic request (the > > same > > type of request the admin client uses). > However, if both configs are enabled, the broker will autocreate > > through > > a > > metadata request before the producer gets a chance. Of course, the way to > avoid this, is to do as you suggested, and set > > the > > "allow_auto_topic_creation" field to false. > > I think the only thing we need to be careful with in this setup is > > without > > KIP 464, we can not use broker defaults for this topic. A user needs > > to > > specify the number of partition and replication factor in the config. An > alternative to this is to have coded defaults for when these > > configs > > are > > unspecified, but it is not immediately apparent what these defaults > > should > > be. > > Thanks again for reading my KIP, > Justine > > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison < > > mickael.mai...@gmail.com > > wrote: > > Hi Justine, > > Thanks for the response! > In my opinion, it would be better if the producer did not rely at > > all > > on the broker auto create feature as this is what we're aiming to > deprecate. When requesting metadata we can set the > "all
Re: [DISCUSS] KIP-503: deleted topics metric
Thanks for the KIP. Its useful metric to have. LGTM. -Harsha On Mon, Aug 05, 2019 at 11:24 AM, David Arthur wrote: > Hello all, I'd like to start a discussion for > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-503%3A+Add+metric+for+number+of+topics+marked+for+deletion > > Thanks! > David >
Re: [DISCUSS] KIP-499 - Unify connection name flag for command line tool
+1 for the KIP. -Harsha On Thu, Aug 1, 2019, at 3:07 PM, Colin McCabe wrote: > On Wed, Jul 31, 2019, at 05:26, Mitchell wrote: > > Hi Jason, > > Thanks for looking at this! > > > > I wasn't exactly sure what to put in the compatibility section. I wrote > > the KIP thinking that we should probably mark the old arguments for > > deprecation for a release or two before actually removing them. I'm happy > > to change this either way if it's preferred. > > I think Jason was proposing deprecating (but NOT removing) the old > arguments in the next release. > > Thanks for tackling this. +1 for the KIP. > > best, > Colin > > > -mitch > > > > On Tue, Jul 30, 2019 at 11:55 PM Jason Gustafson wrote: > > > > > Hey Mitch, thanks for the KIP! This command line inconsistency frustrates > > > me almost every day. I'm definitely +1 on this. > > > > > > One minor nitpick. The compatibility section mentions there will be no > > > deprecations, but it sounds like we are planning on deprecating the old > > > arguments? > > > > > > Thanks, > > > Jason > > > > > > On Tue, Jul 30, 2019 at 5:25 PM Mitchell wrote: > > > > > > > Hello, > > > > I have written a proposal to add the command line argument > > > > `--bootstrap-server` to 5 of the existing command line tools that do not > > > > currently use `--broker-list` for passing cluster connection > > > > information. > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-499+-+Unify+connection+name+flag+for+command+line+tool > > > > > > > > Please take a look and let me know what you think. > > > > Thanks, > > > > -Mitch > > > > > > > > > >
Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Hi Colin, Looks like KIP is missing the images , links are broken. Thanks, Harsha On Thu, Aug 1, 2019, at 2:05 PM, Colin McCabe wrote: > Hi all, > > I've written a KIP about removing ZooKeeper from Kafka. Please take a > look and let me know what you think: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum > > cheers, > Colin >
Re: [VOTE] KIP-492 Add java security providers in Kafka Security config
Thanks for the KIP Sandeep. +1 (binding) Thanks, Harsha On Jul 29, 2019, 12:22 PM -0700, Sandeep Mopuri , wrote: > Hi all, after some good discussion > <https://www.mail-archive.com/dev@kafka.apache.org/msg99419.html> about the > KIP > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config>, > I'm starting the voting. > > This KIP proposes adding new security configuration to accept custom > security providers that can provide algorithms for SSL or SASL. > > -- > Thanks, > M.Sai Sandeep
Re: Fwd: [DISCUSS] KIP-492 Add java security providers in Kafka Security config
Thanks Rajini . > 4) The main difference between SSL and SASL is that for SSL, you register a > provider with your own algorithm name and you specify your algorithm name > in a separate config. This algorithm name can be anything you choose. For > SASL, we register providers for standard SASL mechanism names. If this KIP > wants to address the SASL case, then basically you need to add a provider > ahead of the built-in provider for that standard mechanism name. See below: We can keep this to SSL only. Given for SASL users can configure a provider for LoginModule. Unless there is a usecase where we benefit from having a provider config for SASL. On Thu, Jul 25, 2019, at 5:25 AM, Rajini Sivaram wrote: > Hi Sandeep/Harsha, > > I don't have any major concerns about this KIP since it solves a specific > issue and is a relatively minor change. I am unconvinced about the SASL > case, but it probably is better to add as a config that can be used with > SASL as well in future anyway. > > Just to complete the conversation above: > > 4) The main difference between SSL and SASL is that for SSL, you register a > provider with your own algorithm name and you specify your algorithm name > in a separate config. This algorithm name can be anything you choose. For > SASL, we register providers for standard SASL mechanism names. If this KIP > wants to address the SASL case, then basically you need to add a provider > ahead of the built-in provider for that standard mechanism name. See below: > > 6) JVM starts off with an ordered list of providers, the `java.security` > file you quote above shows the ordering. When you dynamically add > providers, you have the choice of adding it anywhere in that list. > Particularly with SASL where you may have multiple providers with the same > name, this ordering is significant. Security.insertProviderAt(Provider > provider, int position) allows you to choose the position. I think the KIP > intends to add provider at the beginning using Security.addProvider(Provider > provider), which is basically inserting at position 0. We should clarify > the behaviour in the KIP. > > We should also clarify how this config will co-exist with existing dynamic > updates of security providers in the SASL login modules in Kafka. Or > clarify that you can't override those. What we don't want is > non-deterministic behaviour which is the biggest issue with these static > configs. > > > On Wed, Jul 24, 2019 at 5:50 PM Harsha wrote: > > > Thanks for the details. > > Rajini, Can you please take a look and let us know if these addresses your > > concerns. > > > > Thanks, > > Harsha > > > > On Mon, Jul 22, 2019, at 9:36 AM, Sandeep Mopuri wrote: > > > Hi Rajini, > > > Thanks for raising the above questions. Please find the > > > replies below > > > > > > On Wed, Jul 17, 2019 at 2:49 AM Rajini Sivaram > > > wrote: > > > > > > > Hi Sandeep, > > > > > > > > Thanks for the KIP. A few questions below: > > > > > > > >1. Is the main use case for this KIP adding security providers for > > SSL? > > > >If so, wouldn't a more generic solution like KIP-383 work for this? > > > > > > >We’re trying to solve this for both SSL and SASL. KIP-383 allows > > the > > > creation of custom SSLFactory implementation, however adding the provides > > > to new security algorithms doesn’t involve any new implementation of > > > SSLFactory. Even after the KIP 383, people still are finding a need for > > > loading custom keymanager and trustmanager implementations (KIP 486) > > > > > >2. Presumably this config would also apply to clients. If so, have we > > > >thought through the implications of changing static JVM-wide > > security > > > >providers in the client applications? > > > > > > >Yes, this config will be applied to clients as well and the > > > responsibility to face the consequences of adding the security providers > > > need to be taken by the clients. In cases of resource manager running > > > streaming applications such as Yarn, Mesos etc.. each user needs to make > > > sure they are passing these JVM arguments. > > > > > >3. Since client applications can programmatically invoke the Java > > > >Security API anyway, isn't the system property described in > > `Rejected > > > >Alternatives` a reasonable solution for brokers? > > > > > > > This is true in a kafka only environment, but with an eco-system
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
Hi Habib, Yes. Our approach is to have retention as you see it to day i.e delete the local log segments after configured amount of time or size is reached. We will be shipping logs to remote storage such as HDFS or S3 as soon as a log segment is rotated in a topic-partition. This will not trigger any deletion of local segments. Thanks, Harsha On Thu, Jul 25, 2019, at 6:01 AM, Habib Nahas wrote: > Hi, > > Under the proposed definition of RemoteTier, would it be possible to > have an implementation that transfers older log segments to a slower > storage tier, but one that is still local? > Examples of slower local(ie mounted locally) tiers being HDDs vs SSDs, > or NFS volumes. > > Let me know if I"m missing an existing solution for this usecase. > Thanks, > > Habib > > > On 2019/04/09 05:04:17, Harsha wrote: > > Thanks, Ron. Updating the KIP. will add answers here as well> > > > > 1) If the cold storage technology can be cross-region, is there a> > > possibility for a disaster recovery Kafka cluster to share the messages in> > > cold storage? My guess is the answer is no, and messages replicated to the> > > D/R cluster have to be migrated to cold storage from there independently.> > > (The same cross-region cold storage medium could be used, but every > > message> > > would appear there twice).> > > > > If I understand the question correctly, what you are saying is Kafka A > > cluster (active) shipping logs to remote storage which cross-region > > replication and another Kafka Cluster B (Passive) will it be able to use > > the remote storage copied logs directly.> > > > > > > For the initial version my answer is No. We can handle this in subsequent > > changes after this one.> > > > > 2) Can/should external (non-Kafka) tools have direct access to the > > messages> > > in cold storage. I think this might have been addressed when someone asked> > > about ACLs, and I believe the answer is "no" -- if some external tool > > needs> > > to operate on that data then that external tool should read that data by> > > acting as a Kafka consumer. Again, just asking to get the answer clearly> > > documented in case it is unclear.> > > > > The answer is No. All tools/clients must go through broker APIs to access > > any data (local or remote). > > > Only Kafka broker user will have access to remote storage logs and > > Security/ACLs will work the way it does today.> > > Tools/Clients going directly to the remote storage might help in terms of > > efficiency but this requires Protocol changes and some way of syncing ACLs > > in Kafka to the Remote storage. > > > > > > > > > > > Thanks,> > > Harsha> > > > > On Mon, Apr 8, 2019, at 8:48 AM, Ron Dagostino wrote:> > > > Hi Harsha. A couple of questions. I think I know the answers, but it> > > > would be good to see them explicitly documented.> > > > > > > > 1) If the cold storage technology can be cross-region, is there a> > > > possibility for a disaster recovery Kafka cluster to share the messages > > > in> > > > cold storage? My guess is the answer is no, and messages replicated to > > > the> > > > D/R cluster have to be migrated to cold storage from there > > > independently.> > > > (The same cross-region cold storage medium could be used, but every > > > message> > > > would appear there twice).> > > > > > > > 2) Can/should external (non-Kafka) tools have direct access to the > > > messages> > > > in cold storage. I think this might have been addressed when someone > > > asked> > > > about ACLs, and I believe the answer is "no" -- if some external tool > > > needs> > > > to operate on that data then that external tool should read that data by> > > > acting as a Kafka consumer. Again, just asking to get the answer clearly> > > > documented in case it is unclear.> > > > > > > > Ron> > > > > > > > > > > > On Thu, Apr 4, 2019 at 12:53 AM Harsha wrote:> > > > > > > > > Hi Viktor,> > > > >> > > > >> > > > > "Now, will the consumer be able to consume a remote segment if:> > > > > - the remote segment is stored in the remote storage, BUT> > > > > - the lea
Re: Fwd: [DISCUSS] KIP-492 Add java security providers in Kafka Security config
Thanks for the details. Rajini, Can you please take a look and let us know if these addresses your concerns. Thanks, Harsha On Mon, Jul 22, 2019, at 9:36 AM, Sandeep Mopuri wrote: > Hi Rajini, > Thanks for raising the above questions. Please find the > replies below > > On Wed, Jul 17, 2019 at 2:49 AM Rajini Sivaram > wrote: > > > Hi Sandeep, > > > > Thanks for the KIP. A few questions below: > > > >1. Is the main use case for this KIP adding security providers for SSL? > >If so, wouldn't a more generic solution like KIP-383 work for this? > > >We’re trying to solve this for both SSL and SASL. KIP-383 allows the > creation of custom SSLFactory implementation, however adding the provides > to new security algorithms doesn’t involve any new implementation of > SSLFactory. Even after the KIP 383, people still are finding a need for > loading custom keymanager and trustmanager implementations (KIP 486) > >2. Presumably this config would also apply to clients. If so, have we > >thought through the implications of changing static JVM-wide security > >providers in the client applications? > > >Yes, this config will be applied to clients as well and the > responsibility to face the consequences of adding the security providers > need to be taken by the clients. In cases of resource manager running > streaming applications such as Yarn, Mesos etc.. each user needs to make > sure they are passing these JVM arguments. > >3. Since client applications can programmatically invoke the Java > >Security API anyway, isn't the system property described in `Rejected > >Alternatives` a reasonable solution for brokers? > > > This is true in a kafka only environment, but with an eco-system of > streaming applications like flink, spark etc which might produce to kafka, > it’s difficult to make changes to all the clients > >4. We have SASL login modules in Kafka that automatically add security > >providers for SASL mechanisms not supported by the JVM. We should > > describe > >the impact of this KIP on those and whether we would now require a > > config > >to enable these security providers > > In a single JVM, one can register multiple security.providers. By default > JVM itself provides multiple providers and these will not stepped over each > other. The only way to activate a provider is through their registered names > Example: > > $ cat /usr/lib/jvm/jdk-8-oracle-x64/jre/lib/security/java.security > ... > security.provider.1=sun.security.provider.Sun > security.provider.2=sun.security.rsa.SunRsaSign > security.provider.3=sun.security.ec.SunEC > security.provider.4=com.sun.net.ssl.internal.ssl.Provider > security.provider.5=com.sun.crypto.provider.SunJCE > security.provider.6=sun.security.jgss.SunProvider > security.provider.7=com.sun.security.sasl.Provider > security.provider.8=org.jcp.xml.dsig.internal.dom.XMLDSigRI > security.provider.9=sun.security.smartcardio.SunPCSC > ... > >A user of a provider will refer through their registered provider > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java#L31 > >In the above example , we can register the SpiffeProvider and > multiple other providers into the JVM. When a client or a broker wants to > integrate with SpiffeProvider they need to add the config > ssl.keymanager.alogirhtm = "Spiffe" . Another client can refer to a > different provider or use a default one in the same JVM. > > > >5. We have been moving away from JVM-wide configs like the default JAAS > >config since they are hard to test reliably or update dynamically. The > >replacement config `sasl.jaas.config` doesn't insert a JVM-wide > >configuration. Have we investigated similar options for the specific > >scenario we are addressing here? > > >Yes, that is the case with jaas config, however in the case of > security providers, along with adding the security providers to JVM > properties, one also need to configure the provider algorithm. For example, > in the case of SSL configuration, besides adding the security provider to > the JVM, we need to configure the “ssl.trustmanager.algorithm” and > “ssl.keymanager.algorithm” inorder for the provider implementation to > apply. Different components can opt for different key and trustmanager > algorithms and can work independently simultaneously in the same JVM. This > case is different from the jaas config. > > > >6. Are we always going to insert new prov
Re: [DISCUSS] KIP-492 Add java security providers in Kafka Security config
Thanks for the KIP Sandeep. LGTM. Mani & Rajini, can you please look at the KIP as well. Thanks, Harsha On Tue, Jul 16, 2019, at 2:54 PM, Sandeep Mopuri wrote: > Thanks for the suggestions, made changes accordingly. > > On Tue, Jul 16, 2019 at 9:27 AM Satish Duggana > wrote: > > > Hi Sandeep, > > Thanks for the KIP, I have few comments below. > > > > >>“To take advantage of these custom algorithms, we want to support java > > security provider parameter in security config. This param can be used by > > kafka brokers or kafka clients(when connecting to the kafka brokers). The > > security providers can also be used for configuring security algorithms in > > SASL based communication.” > > > > You may want to mention use case like > > spiffe.provider.SpiffeProvider[1] in streaming applications like > > Flink, Spark or Storm etc. > > > > >>"We add new config parameter in KafkaConfig named > > “security.provider.class”. The value of “security.provider” is expected to > > be a string representing the provider’s full classname. This provider class > > will be added to the JVM properties through Security.addProvider api. > > Security class can be used to programmatically add the provider classes to > > the JVM." > > > > It is good to have this property as a list of providers instead of a > > single property. This will allow configuring multiple providers if it > > is needed in the future without introducing hacky solutions like > > security.provider.class.name.x, where x is a sequence number. You can > > change the property name to “security.provider.class.names” and its > > value is a list of fully qualified provider class names separated by > > ‘,'. > > For example: > > > > security.provider.class.names=spiffe.provider.SpiffeProvider,com.foo.MyProvider > > > > Typo in existing properties section: > > “ssl.provider” instead of “ssl.providers”. > > > > Thanks, > > Satish. > > > > 1. https://github.com/spiffe/java-spiffe > > > > > > On Mon, Jul 15, 2019 at 11:41 AM Sandeep Mopuri wrote: > > > > > > Hello all, > > > > > > I'd like to start a discussion thread for KIP-492. > > > This KIP plans on introducing a new security config parameter for a > > custom > > > security providers. Please take a look and let me know what do you think. > > > > > > More information can be found here: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config > > > -- > > > Thanks, > > > Sai Sandeep > > > > > -- > Thanks, > M.Sai Sandeep >
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
You can look at the implementation here for an example https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java On Tue, Jul 16, 2019, at 9:00 PM, Harsha wrote: > Hi Maulin, >This is not required. We are already addressing this. > One can write a KeyStoreProvider and TrustStoreProvider. Please look at > this JIRA https://issues.apache.org/jira/browse/KAFKA-8191 . It allows > users to add custom security provider in which you can write > KeyManagerFactory, TrustManagerFactory and add that your JVM settings > and pass that factory name via configs exposed in KAFKA-8191. These are > Java APIs and instead of adding custom apis like you are proposing in > the KIP. > > -Harsha > > On Tue, Jul 16, 2019, at 1:51 PM, Maulin Vasavada wrote: > > Bump! Can somebody please review this? > > >
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Hi Maulin, This is not required. We are already addressing this. One can write a KeyStoreProvider and TrustStoreProvider. Please look at this JIRA https://issues.apache.org/jira/browse/KAFKA-8191 . It allows users to add custom security provider in which you can write KeyManagerFactory, TrustManagerFactory and add that your JVM settings and pass that factory name via configs exposed in KAFKA-8191. These are Java APIs and instead of adding custom apis like you are proposing in the KIP. -Harsha On Tue, Jul 16, 2019, at 1:51 PM, Maulin Vasavada wrote: > Bump! Can somebody please review this? >
Re: Possible implementation for KAFKA-560
Hi Carlos, This is a really useful feature and we would like to have it as well. I think high_watermark == log_start_offset is a good starting point to consider but we may also have a case where the topic is empty and the clients producing it may be offline so we might end up garbage collecting which is still active. Having a configurable time period when an empty topic can be deleted will help in this case. Also, we should check if there are any consumers still reading from topics etc.. It will be good to have a KIP around this and add some edge cases handling. Thanks, Harsha On Sun, Jun 23, 2019, at 9:40 PM, Carlos Manuel Duclos-Vergara wrote: > Hi, > Thanks for the answer. Looking at high water mark, then the logic would be > to flag the partitions that have > > high_watermark == log_start_offset > > In addition, I'm thinking that having the leader fulfill that criteria is > enough to flag a partition, maybe check the replicas only if requested by > the user. > > > fre. 21. jun. 2019, 23:35 skrev Colin McCabe : > > > I don't think this requires a change in the protocol. It seems like you > > should be able to use the high water mark to figure something out here? > > > > best, > > Colin > > > > > > On Fri, Jun 21, 2019, at 04:56, Carlos Manuel Duclos-Vergara wrote: > > > Hi, > > > > > > This is an ancient task, but I feel it is still current today (specially > > > since as somebody that deals with a Kafka cluster I know that this > > happens > > > more often than not). > > > > > > The task is about garbage collection of topics in a sort of automated > > way. > > > After some consideration I started a prototype implementation based on a > > > manual process: > > > > > > 1. Using the cli, I can use the --describe-topic to get a list of topics > > > that have size 0 > > > 2. Massage that list into something that can be then fed into the cli and > > > remove the topics that have size 0. > > > > > > The guiding principle here is the assumption that abandoned topics will > > > eventually have size 0, because all records will expire. This is not true > > > for all topics, but it covers a large portion of them and having > > something > > > like this would help admins to find "suspicious" topics at least. > > > > > > I started implementing this change and I realized that it would require a > > > change in the protocol, because the sizes are never sent over the wire. > > > Funny enough we collect the sizes of the log files, but we do not send > > them. > > > > > > I think this kind of changes will require a KIP, but I wanted to ask what > > > others think about this. > > > > > > The in-progress implementation of this can be found here: > > > > > https://github.com/carlosduclos/kafka/commit/0dffe5e131c3bd32b77f56b9be8eded89a96df54 > > > > > > Comments? > > > > > > -- > > > Carlos Manuel Duclos Vergara > > > Backend Software Developer > > > > > >
Re: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol
+1 (binding). Thanks for the KIP looking forward for this to be avaiable in consumers. Thanks, Harsha On Wed, May 22, 2019, at 12:24 AM, Liquan Pei wrote: > +1 (non-binding) > > On Tue, May 21, 2019 at 11:34 PM Boyang Chen wrote: > > > Thank you Guozhang for all the hard work. > > > > +1 (non-binding) > > > > > > From: Guozhang Wang > > Sent: Wednesday, May 22, 2019 1:32 AM > > To: dev > > Subject: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol > > > > Hello folks, > > > > I'd like to start the voting for KIP-429 now, details can be found here: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol#KIP-429:KafkaConsumerIncrementalRebalanceProtocol-RebalanceCallbackErrorHandling > > > > And the on-going PRs available for review: > > > > Part I: https://github.com/apache/kafka/pull/6528 > > Part II: https://github.com/apache/kafka/pull/6778 > > > > > > Thanks > > -- Guozhang > > > > > -- > Liquan Pei > Software Engineer, Confluent Inc >
Re: [VOTE] KIP-461 Improve Replica Fetcher behavior at handling partition failure
Thanks for the KIP. +1(binding) Thanks, Harsha On May 9, 2019, 9:58 AM -0700, Jun Rao , wrote: > Hi, Aishwarya, > > Thanks for the KIP. +1 > > Jun > > On Wed, May 8, 2019 at 4:30 PM Aishwarya Gune > wrote: > > > Hi All! > > > > I would like to call for a vote on KIP-461 that would improve the behavior > > of replica fetcher in case of partition failure. The fetcher thread would > > just stop monitoring the crashed partition instead of terminating. > > > > Here's a link to the KIP - > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-461+-+Improve+Replica+Fetcher+behavior+at+handling+partition+failure > > > > Discussion thread - > > https://www.mail-archive.com/dev@kafka.apache.org/msg97559.html > > > > -- > > Thank you, > > Aishwarya > >
Re: [VOTE] KIP-434: Dead replica fetcher and log cleaner metrics
Thanks for the kip. LGTM +1. -Harsha On Mon, Apr 29, 2019, at 8:14 AM, Viktor Somogyi-Vass wrote: > Hi Jason, > > I too agree this is more of a problem in older versions and therefore we > could backport it. Were you thinking of any specific versions? I guess the > 2.x and 1.x versions are definitely targets here but I was thinking that we > might not want to further. > > Viktor > > On Mon, Apr 29, 2019 at 12:55 AM Stanislav Kozlovski > wrote: > > > Thanks for the work done, Viktor! +1 (non-binding) > > > > I strongly agree with Jason that this monitoring-focused KIP is worth > > porting back to older versions. I am sure users will find it very useful > > > > Best, > > Stanislav > > > > On Fri, Apr 26, 2019 at 9:38 PM Jason Gustafson > > wrote: > > > > > Thanks, that works for me. +1 > > > > > > By the way, we don't normally port KIPs to older releases, but I wonder > > if > > > it's worth making an exception here. From recent experience, it tends to > > be > > > the older versions that are more prone to fetcher failures. Thoughts? > > > > > > -Jason > > > > > > On Fri, Apr 26, 2019 at 5:18 AM Viktor Somogyi-Vass < > > > viktorsomo...@gmail.com> > > > wrote: > > > > > > > Let me have a second thought, I'll just add the clientId instead to > > > follow > > > > the convention, so it'll change DeadFetcherThreadCount but with the > > > > clientId tag. > > > > > > > > On Fri, Apr 26, 2019 at 11:29 AM Viktor Somogyi-Vass < > > > > viktorsomo...@gmail.com> wrote: > > > > > > > > > Hi Jason, > > > > > > > > > > Yea I think it could make sense. In this case I would rename the > > > > > DeadFetcherThreadCount to DeadReplicaFetcherThreadCount and introduce > > > the > > > > > metric you're referring to as DeadLogDirFetcherThreadCount. > > > > > I'll update the KIP to reflect this. > > > > > > > > > > Viktor > > > > > > > > > > On Thu, Apr 25, 2019 at 8:07 PM Jason Gustafson > > > > > wrote: > > > > > > > > > >> Hi Viktor, > > > > >> > > > > >> This looks good. Just one question I had is whether we may as well > > > cover > > > > >> the log dir fetchers as well. > > > > >> > > > > >> Thanks, > > > > >> Jason > > > > >> > > > > >> > > > > >> On Thu, Apr 25, 2019 at 7:46 AM Viktor Somogyi-Vass < > > > > >> viktorsomo...@gmail.com> > > > > >> wrote: > > > > >> > > > > >> > Hi Folks, > > > > >> > > > > > >> > This thread sunk a bit but I'd like to bump it hoping to get some > > > > >> feedback > > > > >> > and/or votes. > > > > >> > > > > > >> > Thanks, > > > > >> > Viktor > > > > >> > > > > > >> > On Thu, Mar 28, 2019 at 8:47 PM Viktor Somogyi-Vass < > > > > >> > viktorsomo...@gmail.com> > > > > >> > wrote: > > > > >> > > > > > >> > > Sorry, the end of the message cut off. > > > > >> > > > > > > >> > > So I tried to be consistent with the convention in LogManager, > > > hence > > > > >> the > > > > >> > > hyphens and in AbstractFetcherManager, hence the camel case. It > > > > would > > > > >> be > > > > >> > > nice though to decide with one convention across the whole > > > project, > > > > >> > however > > > > >> > > it requires a major refactor (especially for the components that > > > > >> leverage > > > > >> > > metrics for monitoring). > > > > >> > > > > > > >> > > Thanks, > > > > >> > > Viktor > > > > >> > > > > > > >> > > On Thu, Mar 28, 2019 at 8:44 PM Viktor Somogyi-Vass < > > > > >> > > viktorsomo...@gmail.com> wrote: > > > > >> > > > > > > >> > >> Hi Dhruvil, > > > >
Re: [VOTE] KIP-433: Block old clients on brokers
Hi Gwen & Ismael, Do you have any feedback on with the proposed approach, min.api.version allowing users to specify versions for every request. Thanks, Harsha On Fri, Apr 19, 2019, at 10:24 AM, Harsha wrote: > Thanks Ying for updating the KIP. > Hi Ismael, > Given min.api.version allows admin/users to specifiy > min.version for each request this should address your concerns right? > > Thanks, > Harsha > > On Wed, Apr 17, 2019, at 2:29 PM, Ying Zheng wrote: > > I have updated the config description in the KIP, made the example more > > clear > > > > The proposed change allows setting different min versions for different > > APIs, and the ApiVersionRequest change is already in the KIP. > > > > On Fri, Apr 12, 2019 at 8:22 PM Harsha wrote: > > > > > Hi Ismael, > > > I meant to say blocking clients based on their API version > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L48 > > > But If I understand what you are saying, since each client release can > > > support different versions for each of fetch, produce, offset commit etc.. > > > and it's harder to block just based on single min.api.version setting > > > across different clients. > > > The idea I had in my mind was to do this via ApiVersionRequest, when a > > > client makes api request to broker in response we return min and max > > > version supported for each Api. When min.api.version enabled on broker, it > > > returns the maxVersion it supports for each of the requests in that > > > release > > > as min versions to the clients. > > > > > > Example: > > > Kafka 1.1.1 broker and min.api.verson set to > > > https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/api/ApiVersion.scala#L79 > > > (KAFKA_1_1_IV0) and client makes a ApiVersionsRequest and in response for > > > example produce request > > > > > > https://github.com/apache/kafka/blob/1.1/clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java#L112 > > > Instead of returning all of the supported versions it will return > > > PRODUCE_REQUEST_V5 as the only supported version. > > > > > > Irrespective of the above approach I understand your point still stands > > > which is sarama might not choose to implement all the higher version > > > protocols for Kafka 1.1 release and they might introduce higher version of > > > produce request in a subsequent minor release and it will be harder for > > > users to figure out which release of sarama client they can use. > > > > > > > > > Ying, if you have a different apporach which might address this issue > > > please add. > > > > > > > > > Thanks, > > > Harsha > > > > > > On Fri, Apr 12, 2019, at 7:23 PM, Ismael Juma wrote: > > > > Hi Harsha, > > > > > > > > There is no such thing as 1.1 protocol. I encourage you to describe an > > > > example config that achieves what you are suggesting here. It's pretty > > > > complicated because the versions are per API and each client evolves > > > > independently. > > > > > > > > Ismael > > > > > > > > On Sat, Apr 13, 2019 at 4:09 AM Harsha wrote: > > > > > > > > > Hi, > > > > > > > > > > "Relying on min.version seems like a pretty clunky way to achieve the > > > above > > > > > > list. The challenge is that it's pretty difficult to do it in a way > > > that > > > > > > works for clients across languages. They each add support for new > > > > > protocol > > > > > > versions independently (it could even happen in a bug fix release). > > > So, > > > > > if > > > > > > you tried to block Sarama in #2, you may block Java clients too." > > > > > > > > > > That's the intended effect, right? if you as the admin/operator > > > > > configures the broker to have min.api.version to be 1.1 > > > > > it should block java , sarama clients etc.. which are below the 1.1 > > > > > protocol. As mentioned this is not just related to log.format upgrade > > > > > problem but in general a forcing cause to get the users to upgrade > > > their > > > > > client version in a multi-tenant environment. > > > > > > > > > > "
Re: [VOTE] KIP-433: Block old clients on brokers
Thanks Ying for updating the KIP. Hi Ismael, Given min.api.version allows admin/users to specifiy min.version for each request this should address your concerns right? Thanks, Harsha On Wed, Apr 17, 2019, at 2:29 PM, Ying Zheng wrote: > I have updated the config description in the KIP, made the example more > clear > > The proposed change allows setting different min versions for different > APIs, and the ApiVersionRequest change is already in the KIP. > > On Fri, Apr 12, 2019 at 8:22 PM Harsha wrote: > > > Hi Ismael, > > I meant to say blocking clients based on their API version > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L48 > > But If I understand what you are saying, since each client release can > > support different versions for each of fetch, produce, offset commit etc.. > > and it's harder to block just based on single min.api.version setting > > across different clients. > > The idea I had in my mind was to do this via ApiVersionRequest, when a > > client makes api request to broker in response we return min and max > > version supported for each Api. When min.api.version enabled on broker, it > > returns the maxVersion it supports for each of the requests in that release > > as min versions to the clients. > > > > Example: > > Kafka 1.1.1 broker and min.api.verson set to > > https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/api/ApiVersion.scala#L79 > > (KAFKA_1_1_IV0) and client makes a ApiVersionsRequest and in response for > > example produce request > > > > https://github.com/apache/kafka/blob/1.1/clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java#L112 > > Instead of returning all of the supported versions it will return > > PRODUCE_REQUEST_V5 as the only supported version. > > > > Irrespective of the above approach I understand your point still stands > > which is sarama might not choose to implement all the higher version > > protocols for Kafka 1.1 release and they might introduce higher version of > > produce request in a subsequent minor release and it will be harder for > > users to figure out which release of sarama client they can use. > > > > > > Ying, if you have a different apporach which might address this issue > > please add. > > > > > > Thanks, > > Harsha > > > > On Fri, Apr 12, 2019, at 7:23 PM, Ismael Juma wrote: > > > Hi Harsha, > > > > > > There is no such thing as 1.1 protocol. I encourage you to describe an > > > example config that achieves what you are suggesting here. It's pretty > > > complicated because the versions are per API and each client evolves > > > independently. > > > > > > Ismael > > > > > > On Sat, Apr 13, 2019 at 4:09 AM Harsha wrote: > > > > > > > Hi, > > > > > > > > "Relying on min.version seems like a pretty clunky way to achieve the > > above > > > > > list. The challenge is that it's pretty difficult to do it in a way > > that > > > > > works for clients across languages. They each add support for new > > > > protocol > > > > > versions independently (it could even happen in a bug fix release). > > So, > > > > if > > > > > you tried to block Sarama in #2, you may block Java clients too." > > > > > > > > That's the intended effect, right? if you as the admin/operator > > > > configures the broker to have min.api.version to be 1.1 > > > > it should block java , sarama clients etc.. which are below the 1.1 > > > > protocol. As mentioned this is not just related to log.format upgrade > > > > problem but in general a forcing cause to get the users to upgrade > > their > > > > client version in a multi-tenant environment. > > > > > > > > "> For #3, it seems simplest to have a config that requires clients to > > > > support > > > > > a given message format version (or higher). For #2, it seems like > > you'd > > > > > want clients to advertise their versions. That would be useful for > > > > multiple > > > > > reasons." > > > > This kip offers the ability to block clients based on the protocol they > > > > support. This should be independent of the message format upgrade. Not > > all > > > > of the features or bugs are dependent on a message format and having a > > > > message format depe
Re: [VOTE] KIP-433: Block old clients on brokers
Hi Ismael, I meant to say blocking clients based on their API version https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L48 But If I understand what you are saying, since each client release can support different versions for each of fetch, produce, offset commit etc.. and it's harder to block just based on single min.api.version setting across different clients. The idea I had in my mind was to do this via ApiVersionRequest, when a client makes api request to broker in response we return min and max version supported for each Api. When min.api.version enabled on broker, it returns the maxVersion it supports for each of the requests in that release as min versions to the clients. Example: Kafka 1.1.1 broker and min.api.verson set to https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/api/ApiVersion.scala#L79 (KAFKA_1_1_IV0) and client makes a ApiVersionsRequest and in response for example produce request https://github.com/apache/kafka/blob/1.1/clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java#L112 Instead of returning all of the supported versions it will return PRODUCE_REQUEST_V5 as the only supported version. Irrespective of the above approach I understand your point still stands which is sarama might not choose to implement all the higher version protocols for Kafka 1.1 release and they might introduce higher version of produce request in a subsequent minor release and it will be harder for users to figure out which release of sarama client they can use. Ying, if you have a different apporach which might address this issue please add. Thanks, Harsha On Fri, Apr 12, 2019, at 7:23 PM, Ismael Juma wrote: > Hi Harsha, > > There is no such thing as 1.1 protocol. I encourage you to describe an > example config that achieves what you are suggesting here. It's pretty > complicated because the versions are per API and each client evolves > independently. > > Ismael > > On Sat, Apr 13, 2019 at 4:09 AM Harsha wrote: > > > Hi, > > > > "Relying on min.version seems like a pretty clunky way to achieve the above > > > list. The challenge is that it's pretty difficult to do it in a way that > > > works for clients across languages. They each add support for new > > protocol > > > versions independently (it could even happen in a bug fix release). So, > > if > > > you tried to block Sarama in #2, you may block Java clients too." > > > > That's the intended effect, right? if you as the admin/operator > > configures the broker to have min.api.version to be 1.1 > > it should block java , sarama clients etc.. which are below the 1.1 > > protocol. As mentioned this is not just related to log.format upgrade > > problem but in general a forcing cause to get the users to upgrade their > > client version in a multi-tenant environment. > > > > "> For #3, it seems simplest to have a config that requires clients to > > support > > > a given message format version (or higher). For #2, it seems like you'd > > > want clients to advertise their versions. That would be useful for > > multiple > > > reasons." > > This kip offers the ability to block clients based on the protocol they > > support. This should be independent of the message format upgrade. Not all > > of the features or bugs are dependent on a message format and having a > > message format dependency to block clients means we have to upgrade to > > message.format and we cannot just say we've 1.1 brokers with 0.8.2 message > > format and now we want to block all 0.8.x clients. > > > > min.api.version helps at the cluster level to say that all users required > > to upgrade clients to the at minimum need to speak the min.api.version and > > not tie to message.format because not all cases one wants to upgrade the > > message format and block the old clients. > > > > > > To Gwen's point, I think we should also return in the error message that > > the broker only supports min.api.version and above. So that users can see a > > clear message and upgrade to a newer version. > > > > > > Thanks, > > Harsha > > > > > > On Fri, Apr 12, 2019, at 12:19 PM, Ismael Juma wrote: > > > Hi Ying, > > > > > > The actual reasons are important so that people can evaluate the KIP (and > > > vote). :) Thanks for providing a few more: > > > > > > (1) force users to check pointing in Kafka instead of zookeeper > > > (2) forbid an old go (sarama) client library which is known to have some > > > serious bugs > > > (3) force kafka 1.x clients with
Re: [VOTE] KIP-433: Block old clients on brokers
Hi, "Relying on min.version seems like a pretty clunky way to achieve the above > list. The challenge is that it's pretty difficult to do it in a way that > works for clients across languages. They each add support for new protocol > versions independently (it could even happen in a bug fix release). So, if > you tried to block Sarama in #2, you may block Java clients too." That's the intended effect, right? if you as the admin/operator configures the broker to have min.api.version to be 1.1 it should block java , sarama clients etc.. which are below the 1.1 protocol. As mentioned this is not just related to log.format upgrade problem but in general a forcing cause to get the users to upgrade their client version in a multi-tenant environment. "> For #3, it seems simplest to have a config that requires clients to support > a given message format version (or higher). For #2, it seems like you'd > want clients to advertise their versions. That would be useful for multiple > reasons." This kip offers the ability to block clients based on the protocol they support. This should be independent of the message format upgrade. Not all of the features or bugs are dependent on a message format and having a message format dependency to block clients means we have to upgrade to message.format and we cannot just say we've 1.1 brokers with 0.8.2 message format and now we want to block all 0.8.x clients. min.api.version helps at the cluster level to say that all users required to upgrade clients to the at minimum need to speak the min.api.version and not tie to message.format because not all cases one wants to upgrade the message format and block the old clients. To Gwen's point, I think we should also return in the error message that the broker only supports min.api.version and above. So that users can see a clear message and upgrade to a newer version. Thanks, Harsha On Fri, Apr 12, 2019, at 12:19 PM, Ismael Juma wrote: > Hi Ying, > > The actual reasons are important so that people can evaluate the KIP (and > vote). :) Thanks for providing a few more: > > (1) force users to check pointing in Kafka instead of zookeeper > (2) forbid an old go (sarama) client library which is known to have some > serious bugs > (3) force kafka 1.x clients with the ability to roll back if there's an > issue (unlike a message format upgrade) > > Relying on min.version seems like a pretty clunky way to achieve the above > list. The challenge is that it's pretty difficult to do it in a way that > works for clients across languages. They each add support for new protocol > versions independently (it could even happen in a bug fix release). So, if > you tried to block Sarama in #2, you may block Java clients too. > > For #3, it seems simplest to have a config that requires clients to support > a given message format version (or higher). For #2, it seems like you'd > want clients to advertise their versions. That would be useful for multiple > reasons. > > Ismael > > On Fri, Apr 12, 2019 at 8:42 PM Ying Zheng wrote: > > > Hi Ismael, > > > > Those are just examples. I think the administrators should be able to block > > certain client libraries for whatever reason. Some other possible reasons > > include, force users to check pointing in Kafka instead of zookeeper, > > forbid an old go (sarama) client library which is known to have some > > serious bugs. > > > > message.downconversion.enable does not solve our problems. We are now > > planning to upgrade to message format V3, and force users to upgrade to > > Kafka 1.x clients. With the proposed min.api.version setting, in case of > > there is anything wrong, we can roll back the setting. If we upgrade the > > file format, there is no way to rollback (Kafka doesn't support downgrading > > message format). > > > > On Thu, Apr 11, 2019 at 7:05 PM Ismael Juma wrote: > > > > > Hi Ying, > > > > > > It looks to me that all the examples given in the KIP can be handled with > > > the existing "message.downconversion.enable" config and by configuring > > the > > > message format to be the latest: > > > > > > 1. Kafka 8 / 9 / 10 consumer hangs when the message contains message > > header > > > > ( KAFKA-6739 - Down-conversion fails for records with headers > > RESOLVED ) > > > > 2. LZ4 is not correctly handled in Kafka 8 and Kafka 9 ( KAFKA-3160 - > > > > Kafka LZ4 framing code miscalculates header checksum RESOLVED ) > > > > 3. Performance penalty of converting message format from V3 to V1 or V2 > > > > for the old consumers (KIP-31 - Move to relative offsets in compressed > > &g
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
Thanks, Ron. Updating the KIP. will add answers here as well 1) If the cold storage technology can be cross-region, is there a possibility for a disaster recovery Kafka cluster to share the messages in cold storage? My guess is the answer is no, and messages replicated to the D/R cluster have to be migrated to cold storage from there independently. (The same cross-region cold storage medium could be used, but every message would appear there twice). If I understand the question correctly, what you are saying is Kafka A cluster (active) shipping logs to remote storage which cross-region replication and another Kafka Cluster B (Passive) will it be able to use the remote storage copied logs directly. For the initial version my answer is No. We can handle this in subsequent changes after this one. 2) Can/should external (non-Kafka) tools have direct access to the messages in cold storage. I think this might have been addressed when someone asked about ACLs, and I believe the answer is "no" -- if some external tool needs to operate on that data then that external tool should read that data by acting as a Kafka consumer. Again, just asking to get the answer clearly documented in case it is unclear. The answer is No. All tools/clients must go through broker APIs to access any data (local or remote). Only Kafka broker user will have access to remote storage logs and Security/ACLs will work the way it does today. Tools/Clients going directly to the remote storage might help in terms of efficiency but this requires Protocol changes and some way of syncing ACLs in Kafka to the Remote storage. Thanks, Harsha On Mon, Apr 8, 2019, at 8:48 AM, Ron Dagostino wrote: > Hi Harsha. A couple of questions. I think I know the answers, but it > would be good to see them explicitly documented. > > 1) If the cold storage technology can be cross-region, is there a > possibility for a disaster recovery Kafka cluster to share the messages in > cold storage? My guess is the answer is no, and messages replicated to the > D/R cluster have to be migrated to cold storage from there independently. > (The same cross-region cold storage medium could be used, but every message > would appear there twice). > > 2) Can/should external (non-Kafka) tools have direct access to the messages > in cold storage. I think this might have been addressed when someone asked > about ACLs, and I believe the answer is "no" -- if some external tool needs > to operate on that data then that external tool should read that data by > acting as a Kafka consumer. Again, just asking to get the answer clearly > documented in case it is unclear. > > Ron > > > On Thu, Apr 4, 2019 at 12:53 AM Harsha wrote: > > > Hi Viktor, > > > > > > "Now, will the consumer be able to consume a remote segment if: > > - the remote segment is stored in the remote storage, BUT > > - the leader broker failed right after this AND > > - the follower which is to become a leader didn't scan yet for a new > > segment?" > > > > If I understand correctly, after a local log segment copied to remote and > > leader is failed to write the index files and leadership changed to a > > follower. In this case we consider the log segment copy failed and newly > > elected leader will start copying the data from last the known offset in > > the remote to copy. Consumers who are looking for the offset which might > > be in the failed copy log segment will continue to be read the data from > > local disk since the local log segment will only be deleted once a > > successful copy of the log segment. > > > > "As a follow-up question, what are your experiences, does a failover in a > > broker causes bigger than usual churn in the consumers? (I'm thinking about > > the time required to rebuild remote index files.)" > > > > Rebuild remote index files will only happen in case of remote storage > > missing all the copied index files. Fail-over will not trigger this > > rebuild. > > > > > > Hi Ryan, > > > > "Harsha, can you comment on this alternative approach: instead of fetching > > directly from remote storage via a new API, implement something like > > paging, where segments are paged-in and out of cold storage based on access > > frequency/recency? For example, when a remote segment is accessed, it could > > be first fetched to disk and then read from there. I suppose this would > > require less code changes, or at least less API changes." > > > > Copying whole log segment from remote is inefficient. When tiered storage > > is enabled users might prefer hardware with smaller disks and having to > > copy the log segment to lo
Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection
Looks like the KIP is passed with 3 binding votes. From Matthias, Bill Bejeck and myself you got 3 binding votes. You can do the full tally of the votes and send out a close of vote thread. Thanks, Harsha On Thu, Apr 4, 2019, at 12:24 PM, M. Manna wrote: > Hello, > > Trying to revive this thread again. Would anyone be interested in having > this KiP through > > > Thanks, > > On Fri, 25 Jan 2019 at 16:44, M. Manna wrote: > > > Hello, > > > > I am trying to revive this thread. I only got 1 binding vote so far. > > > > Please feel free to revisit and comment here. > > > > Thanks, > > > > On Thu, 25 Oct 2018 at 00:15, M. Manna wrote: > > > >> Hey IJ, > >> > >> Thanks for your interest in the KIP. > >> > >> My point was simply that the round-robin should happen even if the key is > >> not null. As for the importance of key in our case, we treat the key as > >> metadata. Each key is composed of certain info which are parsed by our > >> consumer thread. We will then determine whether it's an actionable message > >> (e.g. process it), or a loopback(ignore it). You could argue, "Why not > >> append this metadata with the record and parse it there?". But that means > >> the following: > >> > >> 1) I'm always passing null key to achieve this - I would like to pass > >> Null/Not-Null/Other key i.e. flexibility > >> 2) Suppose the message size is 99 KB and and max message bytes allowed is > >> 100K. Now prefixing metadata with message results into the actual message > >> being 101K. This will fail at producer level and cause a retry/log this in > >> our DB for future pickup. > >> > >> To avoid all these, we are simply proposing this new partitioner class. > >> but all Kafka new releases will still have DefaultPartitioner as default, > >> unless they change the prop file to use our new class. > >> > >> Regards, > >> > >> On Sun, 21 Oct 2018 at 04:05, Ismael Juma wrote: > >> > >>> Thanks for the KIP. Can you please elaborate on the need for the key in > >>> this case? The KIP simply states that the key is needed for metadata, but > >>> doesn't give any more details. > >>> > >>> Ismael > >>> > >>> On Tue, Sep 4, 2018 at 3:39 AM M. Manna wrote: > >>> > >>> > Hello, > >>> > > >>> > I have made necessary changes as per the original discussion thread, > >>> and > >>> > would like to put it for votes. > >>> > > >>> > Thank you very much for your suggestion and guidance so far. > >>> > > >>> > Regards, > >>> > > >>> > >> >
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
Hi Viktor, "Now, will the consumer be able to consume a remote segment if: - the remote segment is stored in the remote storage, BUT - the leader broker failed right after this AND - the follower which is to become a leader didn't scan yet for a new segment?" If I understand correctly, after a local log segment copied to remote and leader is failed to write the index files and leadership changed to a follower. In this case we consider the log segment copy failed and newly elected leader will start copying the data from last the known offset in the remote to copy. Consumers who are looking for the offset which might be in the failed copy log segment will continue to be read the data from local disk since the local log segment will only be deleted once a successful copy of the log segment. "As a follow-up question, what are your experiences, does a failover in a broker causes bigger than usual churn in the consumers? (I'm thinking about the time required to rebuild remote index files.)" Rebuild remote index files will only happen in case of remote storage missing all the copied index files. Fail-over will not trigger this rebuild. Hi Ryan, "Harsha, can you comment on this alternative approach: instead of fetching directly from remote storage via a new API, implement something like paging, where segments are paged-in and out of cold storage based on access frequency/recency? For example, when a remote segment is accessed, it could be first fetched to disk and then read from there. I suppose this would require less code changes, or at least less API changes." Copying whole log segment from remote is inefficient. When tiered storage is enabled users might prefer hardware with smaller disks and having to copy the log segment to local disk again , especially incase of multiple consumers on multiple topics triggering this might negatively affect the available local storage. What we proposed in the KIP doesn't affect the existing APIs and we didn't call for any API changes. "And related to paging, does the proposal address what happens when a broker runs out of HDD space? Maybe we should have a way to configure a max number of segments or bytes stored on each broker, after which older or least-recently-used segments are kicked out, even if they aren't expired per the retention policy? Otherwise, I suppose tiered storage requires some babysitting to ensure that brokers don't run out of local storage, despite having access to potentially unbounded cold storage." Existing Kafka behavior will not change with addition of tiered storage and enabling it also will not change behavior. Just like today it's up to the operator to make sure the HD space is monitored and take necessary actions to mitigate that before it becomes fatal failure for broker. We don't stop users to configure the retention period to infinite and they can easily run out of the space. These are not the alternatives considered as they are not efficient copy in out of local disk , hence the reason we didn't add to alternatives considered :). Thanks, Harsha On Wed, Apr 3, 2019, at 7:51 AM, Ryanne Dolan wrote: > Harsha, can you comment on this alternative approach: instead of fetching > directly from remote storage via a new API, implement something like > paging, where segments are paged-in and out of cold storage based on access > frequency/recency? For example, when a remote segment is accessed, it could > be first fetched to disk and then read from there. I suppose this would > require less code changes, or at least less API changes. > > And related to paging, does the proposal address what happens when a broker > runs out of HDD space? Maybe we should have a way to configure a max number > of segments or bytes stored on each broker, after which older or > least-recently-used segments are kicked out, even if they aren't expired > per the retention policy? Otherwise, I suppose tiered storage requires some > babysitting to ensure that brokers don't run out of local storage, despite > having access to potentially unbounded cold storage. > > Just some things to add to Alternatives Considered :) > > Ryanne > > On Wed, Apr 3, 2019 at 8:21 AM Viktor Somogyi-Vass > wrote: > > > Hi Harsha, > > > > Thanks for the answer, makes sense. > > In the meantime one edge case popped up in my mind but first let me > > summarize what I understand if I interpret your KIP correctly. > > > > So basically whenever the leader RSM copies over a segment to the remote > > storage, the leader RLM will append an entry to its remote index files with > > the remote position. After this LogManager can delete the local segment. > > Parallel to this RLM followers are periodically scanning the remote storage > > for files and if they find a new one they update their
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
re extending the current index mechanism to find a offset and its message to find a file in remote storage for a givent offset. This will be optimal way finding for a given offset which remote segment might be serving compare to storing all of this data into internal topic. "To add to Eric's question/confusion about where logic lives (RLM vs. RSM), I think it would be helpful to explicitly identify in the KIP that the RLM delegates to the RSM since the RSM is part of the public API and is the pluggable piece. For example, instead of saying "RLM will ship the log segment files that are older than a configurable time to remote storage" I think it would be better to say "RLM identifies log segment files that are older than a configurable time and delegates to the configured RSM to ship them to remote storage" (or something like that -- just make it clear that the RLM is delegating to the configured RSM)." Thanks. I agree with you. I'll update the KIP. Hi Ambud, Thanks for the comments. "1. Wouldn't implicit checking for old offsets in remote location if not found locally on the leader i.e. do we really need remote index files? Since the storage path for a given topic would presumably be constant across all the brokers, the remote topic-partition path could simply be checked to see if there are any segment file names that would meet the offset requirements for a Consumer Fetch Request. RSM implementations could optionally cache this information." By storing the remote index files locally , it will be faster for us to determine for a requested offset which file might contain the data. This will help us resolve the remote file quickly and return the response. Instead of making a call to remote tier for index look up. Given index files are smaller , it won't be much hit to the storage space. "2. Would it make sense to create an internal compacted Kafka topic to publish & record remote segment information? This would enable the followers to get updates about new segments rather than running list() operations on remote storage to detect new segments which may be expensive." I think Ron also alluding to this. We thought shipping remote index files to remote storage files and let the follower's RLM picking that up makes it easy to have the current replication protocol without any changes. So we don't determine if a follower is in ISR or not based on another topic's replication. We will run small tests and determine if use of topic is better for this. Thanks for the suggestion. 3. For RLM to scan local segment rotations are you thinking of leveraging java.nio.file.WatchService or simply running listFiles() on a periodic basis? Since WatchService implementation is heavily OS dependent it might create some complications around missing FS Events. Ideally we want to introduce file events like you suggested. For POC work we are using just listFiles(). Also copying these files to remote can be slower and we will not delete the files from local disk until the segment is copied and any requests to the data in these files will be served from local disk. So I don't think we need to be aggressive and optimize the this copy segment to remote path. Hi Viktor, Thanks for the comments. "I have a rather technical question to this. How do you plan to package this extension? Does this mean that Kafka will depend on HDFS? I think it'd be nice to somehow separate this off to a different package in the project so that it could be built and released separately from the main Kafka packages." We would like all of this code to be part of Apache Kafka . In early days of Kafka, there is external module which used to contain kafka to hdfs copy tools and dependencies. We would like to have RLM (class implementation) and RSM(interface) to be in core and as you suggested, implementation of RSM could be in another package so that the dependencies of RSM won't come into Kafka's classpath unless someone explicity configures them. Thanks, Harsha On Mon, Apr 1, 2019, at 1:02 AM, Viktor Somogyi-Vass wrote: > Hey Harsha, > > I have a rather technical question to this. How do you plan to package this > extension? Does this mean that Kafka will depend on HDFS? > I think it'd be nice to somehow separate this off to a different package in > the project so that it could be built and released separately from the main > Kafka packages. > This decoupling would be useful when direct dependency on HDFS (or other > implementations) is not needed and would also encourage decoupling for > other storage implementations. > > Best, > Viktor > > On Mon, Apr 1, 2019 at 3:44 AM Ambud Sharma wrote: > > > Hi Harsha, > > > > Thank you for proposing this KIP. We are looking forward to this feature as > > well. > > > > A few questions around the design & implem
Re: [DISCUSS] KIP-405: Kafka Tiered Storage
Hi All, Thanks for your initial feedback. We updated the KIP. Please take a look and let us know if you have any questions. https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage Thanks, Harsha On Wed, Feb 6, 2019, at 10:30 AM, Harsha wrote: > Thanks Eno, Adam & Satish for you review and questions. I'll address > these in KIP and update the thread here. > > Thanks, > Harsha > > On Wed, Feb 6, 2019, at 7:09 AM, Satish Duggana wrote: > > Thanks, Harsha for the KIP. It is a good start for tiered storage in > > Kafka. I have a few comments/questions. > > > > It may be good to have a configuration to keep the number of local > > segments instead of keeping only the active segment. This config can > > be exposed at cluster and topic levels with default value as 1. In > > some use cases, few consumers may lag over one segment, it will be > > better to serve from local storage instead of remote storage. > > > > It may be better to keep “remote.log.storage.enable” and respective > > configuration at topic level along with cluster level. It will be > > helpful in environments where few topics are configured with > > local-storage and other topics are configured with remote storage. > > > > Each topic-partition leader pushes its log segments with respective > > index files to remote whenever active log rolls over, it updates the > > remote log index file for the respective remote log segment. The > > second option is to add offset index files also for each segment. It > > can serve consumer fetch requests for old segments from local log > > segment instead of serving directly from the remote log which may > > cause high latencies. There can be different strategies in when the > > remote segment is copied to a local segment. > > > > What is “remote.log.manager.scheduler.interval.ms” config about? > > > > How do followers sync RemoteLogSegmentIndex files? Do they request > > from leader replica? This looks to be important as the failed over > > leader should have RemoteLogSegmentIndex updated and ready to avoid > > high latencies in serving old data stored in remote logs. > > > > Thanks, > > Satish. > > > > On Tue, Feb 5, 2019 at 10:42 PM Ryanne Dolan wrote: > > > > > > Thanks Harsha, makes sense. > > > > > > Ryanne > > > > > > On Mon, Feb 4, 2019 at 5:53 PM Harsha Chintalapani > > > wrote: > > > > > > > "I think you are saying that this enables additional (potentially > > > > cheaper) > > > > storage options without *requiring* an existing ETL pipeline. “ > > > > Yes. > > > > > > > > " But it's not really a replacement for the sort of pipelines people > > > > build > > > > with Connect, Gobblin etc.” > > > > > > > > It is not. But also making an assumption that everyone runs these > > > > pipelines for storing raw Kafka data into HDFS or S3 is also wrong > > > > assumption. > > > > The aim of this KIP is to provide tiered storage as whole package not > > > > asking users to ship the data on their own using existing ETL, which > > > > means > > > > running a consumer and maintaining those pipelines. > > > > > > > > " My point was that, if you are already offloading records in an ETL > > > > pipeline, why do you need a new pipeline built into the broker to ship > > > > the > > > > same data to the same place?” > > > > > > > > As you said its ETL pipeline, which means users of these pipelines are > > > > reading the data from broker and transforming its state and storing it > > > > somewhere. > > > > The point of this KIP is store log segments as it is without changing > > > > their structure so that we can use the existing offset mechanisms to > > > > look > > > > it up when the consumer needs to read old data. When you do load it via > > > > your existing pipelines you are reading the topic as a whole , which > > > > doesn’t guarantee that you’ll produce this data back into HDFS in S3 in > > > > the > > > > same order and who is going to generate the Index files again. > > > > > > > > > > > > "So you'd end up with one of 1)cold segments are only useful to Kafka; > > > > 2) > > > > you have the same data written to HDFS/etc twice, once for Kafka and > > > > once > > > &
Re: [VOTE] 2.2.0 RC2
+1 (non-bidning) - Download artifacts, setup 3 node cluster - Ran producer/consumer clients Thanks, Harsha On Thu, Mar 21, 2019, at 5:54 AM, Andrew Schofield wrote: > +1 (non-binding) > > - Downloaded the artifacts > - Ran Kafka Connect connectors > > Thanks, > Andrew Schofield > IBM Event Streams > > On 19/03/2019, 19:13, "Manikumar" wrote: > > +1 (non-binding) > > - Verified the artifacts, build from src, ran tests > - Verified the quickstart, ran producer/consumer performance tests. > > Thanks for running release!. > > Thanks, > Manikumar > > On Wed, Mar 20, 2019 at 12:19 AM David Arthur > wrote: > > > +1 > > > > Validated signatures, and ran through quick-start. > > > > Thanks! > > > > On Mon, Mar 18, 2019 at 4:00 AM Jakub Scholz > wrote: > > > > > +1 (non-binding). I used the staged binaries and run some of my > tests > > > against them. All seems to look good to me. > > > > > > On Sat, Mar 9, 2019 at 11:56 PM Matthias J. Sax > > > > wrote: > > > > > > > Hello Kafka users, developers and client-developers, > > > > > > > > This is the third candidate for release of Apache Kafka 2.2.0. > > > > > > > > - Added SSL support for custom principal name > > > > - Allow SASL connections to periodically re-authenticate > > > > - Command line tool bin/kafka-topics.sh adds AdminClient > support > > > > - Improved consumer group management > > > >- default group.id is `null` instead of empty string > > > > - API improvement > > > >- Producer: introduce close(Duration) > > > >- AdminClient: introduce close(Duration) > > > >- Kafka Streams: new flatTransform() operator in Streams > DSL > > > >- KafkaStreams (and other classed) now implement > AutoClosable to > > > > support try-with-resource > > > >- New Serdes and default method implementations > > > > - Kafka Streams exposed internal client.id via ThreadMetadata > > > > - Metric improvements: All `-min`, `-avg` and `-max` > metrics will now > > > > output `NaN` as default value > > > > Release notes for the 2.2.0 release: > > > > > https://eur02.safelinks.protection.outlook.com/?url=https:%2F%2Fhome.apache.org%2F~mjsax%2Fkafka-2.2.0-rc2%2FRELEASE_NOTES.htmldata=02%7C01%7C%7Cbc5822a806a749b0638208d6ac9ef756%7C84df9e7fe9f640afb435%7C1%7C0%7C636886196079314852sdata=zBUbQlQiAuGZzs33TUPUqsuC8IpPavg2lT3yPFO%2F3nA%3Dreserved=0 > > > > > > > > *** Please download, test, and vote by Thursday, March 14, > 9am PST. > > > > > > > > Kafka's KEYS file containing PGP keys we use to sign the > release: > > > > > https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkafka.apache.org%2FKEYSdata=02%7C01%7C%7Cbc5822a806a749b0638208d6ac9ef756%7C84df9e7fe9f640afb435%7C1%7C0%7C636886196079314852sdata=g1Gg%2BoIRgpKUum5%2Bmi2plT1qIfH9d2aZkdK9jw7DLxM%3Dreserved=0 > > > > > > > > * Release artifacts to be voted upon (source and binary): > > > > > https://eur02.safelinks.protection.outlook.com/?url=https:%2F%2Fhome.apache.org%2F~mjsax%2Fkafka-2.2.0-rc2%2Fdata=02%7C01%7C%7Cbc5822a806a749b0638208d6ac9ef756%7C84df9e7fe9f640afb435%7C1%7C0%7C636886196079324862sdata=dUZrMCGvR4ki8XS%2B9dEDQ5Bavv4A4xq86CtcXQ6tnFs%3Dreserved=0 > > > > > > > > * Maven artifacts to be voted upon: > > > > > https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fkafka%2Fdata=02%7C01%7C%7Cbc5822a806a749b0638208d6ac9ef756%7C84df9e7fe9f640afb435%7C1%7C0%7C636886196079324862sdata=sCoRIXmcRQd473bRqwFgQaSm2XI%2BBqHw%2FbiddQd4hnE%3Dreserved=0 > > > > > > > > * Javadoc: > > > > > https://eur02.safelinks.protection.outlook.com/?url=https:%2F%2Fhome.apache.org%2F~mjsax%2Fkafka-2.2.0-rc2%2Fjavadoc%2Fdata=02%7C01%7C%7Cbc5822a806a749b0638208d6ac9ef756%7C84df9e7fe9f640afb435%7C1%7C0%7C636886196079324862sdata=iK4WEFuaK0lCySWROi7BbBv%2Bpg8h%2B9umbVNA7I1rqxc%3Dreserved=0 > > > > > > > > * Tag to be voted upon (off 2.2 branch) is the 2.2.0 tag: > > > > > https://eur02.safelinks
Re: [VOTE] KIP-427: Add AtMinIsr topic partition category (new metric & TopicCommand option)
+1 (binding) -Harsha On Thu, Mar 7, 2019, at 6:48 PM, hacker win7 wrote: > +1 (non-binding) > > > On Mar 8, 2019, at 02:32, Stanislav Kozlovski > > wrote: > > > > Thanks for the KIP, Kevin! This change will be a good improvement to > > Kafka's observability story > > > > +1 (non-binding) > > > > On Thu, Mar 7, 2019 at 4:49 AM Vahid Hashemian > > wrote: > > > >> Thanks for the KIP Kevin. > >> > >> +1 (binding) > >> > >> --Vahid > >> > >> On Wed, Mar 6, 2019 at 8:39 PM Dongjin Lee wrote: > >> > >>> +1 (non-binding) > >>> > >>> On Wed, Mar 6, 2019, 3:14 AM Dong Lin wrote: > >>> > >>>> Hey Kevin, > >>>> > >>>> Thanks for the KIP! > >>>> > >>>> +1 (binding) > >>>> > >>>> Thanks, > >>>> Dong > >>>> > >>>> On Tue, Mar 5, 2019 at 9:38 AM Kevin Lu wrote: > >>>> > >>>>> Hi All, > >>>>> > >>>>> I would like to start the vote thread for KIP-427: Add AtMinIsr topic > >>>>> partition category (new metric & TopicCommand option). > >>>>> > >>>>> > >>>> > >>> > >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103089398 > >>>>> > >>>>> Thanks! > >>>>> > >>>>> Regards, > >>>>> Kevin > >>>>> > >>>> > >>> > >> > > > > > > -- > > Best, > > Stanislav > >
Re: [VOTE] KIP-436 Add a metric indicating start time
+1 (binding) Thanks, Harsha On Fri, Mar 8, 2019, at 2:55 AM, Dongjin Lee wrote: > +1 (non binding) > > 2 bindings, 3 non-bindings until now. (Colin, Manikumar / Satish, Mickael, > Dongjin) > > On Fri, Mar 8, 2019 at 7:44 PM Mickael Maison > wrote: > > > +1 (non binding) > > Thanks > > > > On Fri, Mar 8, 2019 at 6:39 AM Satish Duggana > > wrote: > > > > > > Thanks for the KIP, > > > +1 (non-binding) > > > > > > ~Satish. > > > > > > On Thu, Mar 7, 2019 at 11:58 PM Manikumar > > wrote: > > > > > > > +1 (binding). > > > > > > > > Thanks for the KIP. > > > > > > > > Thanks, > > > > Manikumar > > > > > > > > > > > > On Thu, Mar 7, 2019 at 11:52 PM Colin McCabe > > wrote: > > > > > > > > > +1 (binding). > > > > > > > > > > Thanks, Stanislav. > > > > > > > > > > best, > > > > > Colin > > > > > > > > > > On Tue, Mar 5, 2019, at 05:23, Stanislav Kozlovski wrote: > > > > > > Hey everybody, > > > > > > > > > > > > I'd like to start a vote thread about the lightweight KIP-436 > > > > > > KIP: KIP-436 > > > > > > < > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-436%3A+Add+a+metric+indicating+start+time > > > > > > > > > > > > JIRA: KAFKA-7992 <https://issues.apache.org/jira/browse/KAFKA-7992 > > > > > > > > > Pull Request: 6318 <https://github.com/apache/kafka/pull/6318> > > > > > > > > > > > > -- > > > > > > Best, > > > > > > Stanislav > > > > > > > > > > > > > > > > > > > > -- > *Dongjin Lee* > > *A hitchhiker in the mathematical world.* > *github: <http://goog_969573159/>github.com/dongjinleekr > <https://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: speakerdeck.com/dongjin > <https://speakerdeck.com/dongjin>* >
Re: [DISCUSS] KIP-426: Persist Broker Id to Zookeeper
Hi Eno, A control plane needs to do this today because Kafka doesn't provide such mapping. I am not sure why we want every control plane figure this out and rather let this mapping which exists today in Kafka at node level on disk be at a global level in zookeeper. If we implement this, any control plane will be much simpler and not all the different environments need to understand and re-implement this broker.id mapping. I don't understand duplication either, which control-pane we are talking about? Irrespective which control pane a user ends up using I want to understand the concerns about a broker.id host mapping being available in zookeeper. Broker id belongs to Kafka and not in control pane. Thanks, Harsha On Sat, Mar 2, 2019, at 3:50 AM, Eno Thereska wrote: > Hi Harsha, Li Kan, > > What Colin mentioned is what I see in practice as well (at AWS and our > clusters). A control plane management tool decides the mapping > hostname-broker ID and can change it as it sees fit as brokers fail and new > ones are brought in. That control plane usually already has a database of > sorts that keeps track of existing broker IDs. So this work would duplicate > what that control plane already does. It could also lead to extra work if > that control plane decides to do something different that what the mapping > in Zookeeper has. > > At a minimum I'd like to see the motivation expanded and a description of > how the current cluster is managed that Li Kan has in mind. > > Thanks > Eno > > On Sat, Mar 2, 2019 at 1:43 AM Harsha wrote: > > > Hi, > > Cluster management tools are more generic and they are not aware of > > Kafka specific configs like broker.id. > > Even if they are aware of broker.id's , they will be lost when a disk is > > lost. > > Irrespective of these use cases, let's look at the problem in > > isolation. > > 1. disks are the most common failure case in Kafka clusters > > 2. We are storing auto-generated broker.id on disks hence we lose this > > broker.id mapping when disks fail. > > 3. If we keep the previously generated broker.id mapping along with host > > on zookeeper it's easier to retrieve that mapping on a new host. This would > > reduce the reassignment step and allow us to just copy the data and start > > the new node with the previous broker.id > > which is what the KIP is proposing. > > I want to understand what are your concerns in moving this mapping which > > already exists on disk to zookeeper? > > > > Thanks, > > Harsha > > > > On Fri, Mar 1, 2019, at 11:11 AM, Colin McCabe wrote: > > > On Wed, Feb 27, 2019, at 14:12, Harsha wrote: > > > > Hi Colin, > > > > What we want to is to preserve the broker.id so that we > > > > can do an offline rebuild of a broker. In our cases going through > > > > online Kafka replication to bring up, a failed node will put producer > > > > latencies at risk given the new broker will put all the other leaders > > > > busy with its replication requests. For an offline rebuild, we do not > > > > need to do rebalance as long as we can recover the broker.id > > > > Overall, irrespective of this use case we still want an > > > > ability to retrieve a broker.id for an existing host. This will make > > > > swapping in new hosts with failed hosts by keeping the existing > > > > hostname easier. > > > > > > Thanks for the explanation. Shouldn't this should be handled by the > > > cluster management tool, though? Kafka doesn't include a mechanism for > > > re-creating nodes that failed. That's up to kubernetes, or ansible, or > > > whatever cluster provisioning framework you have in place. This feels > > > like the same kind of thing: managing how the cluster is provisioned. > > > > > > best, > > > Colin > > > > > > > > > > > Thanks, > > > > Harsha > > > > On Wed, Feb 27, 2019, at 11:53 AM, Colin McCabe wrote: > > > > > Hi Li, > > > > > > > > > > > The mechanism simplifies deployment because the same > > configuration can be > > > > > > used across all brokers, however, in a large system where disk > > failure is > > > > > > a norm, the meta file could often get lost, causing a new broker > > id being > > > > > > allocated. This is problematic because new broker id has no > > partition > > > > > > assigned to it so it can’t do anything, while partitions assigned > > to the > > > >
Re: [DISCUSS] KIP-426: Persist Broker Id to Zookeeper
Hi, Cluster management tools are more generic and they are not aware of Kafka specific configs like broker.id. Even if they are aware of broker.id's , they will be lost when a disk is lost. Irrespective of these use cases, let's look at the problem in isolation. 1. disks are the most common failure case in Kafka clusters 2. We are storing auto-generated broker.id on disks hence we lose this broker.id mapping when disks fail. 3. If we keep the previously generated broker.id mapping along with host on zookeeper it's easier to retrieve that mapping on a new host. This would reduce the reassignment step and allow us to just copy the data and start the new node with the previous broker.id which is what the KIP is proposing. I want to understand what are your concerns in moving this mapping which already exists on disk to zookeeper? Thanks, Harsha On Fri, Mar 1, 2019, at 11:11 AM, Colin McCabe wrote: > On Wed, Feb 27, 2019, at 14:12, Harsha wrote: > > Hi Colin, > > What we want to is to preserve the broker.id so that we > > can do an offline rebuild of a broker. In our cases going through > > online Kafka replication to bring up, a failed node will put producer > > latencies at risk given the new broker will put all the other leaders > > busy with its replication requests. For an offline rebuild, we do not > > need to do rebalance as long as we can recover the broker.id > > Overall, irrespective of this use case we still want an > > ability to retrieve a broker.id for an existing host. This will make > > swapping in new hosts with failed hosts by keeping the existing > > hostname easier. > > Thanks for the explanation. Shouldn't this should be handled by the > cluster management tool, though? Kafka doesn't include a mechanism for > re-creating nodes that failed. That's up to kubernetes, or ansible, or > whatever cluster provisioning framework you have in place. This feels > like the same kind of thing: managing how the cluster is provisioned. > > best, > Colin > > > > > Thanks, > > Harsha > > On Wed, Feb 27, 2019, at 11:53 AM, Colin McCabe wrote: > > > Hi Li, > > > > > > > The mechanism simplifies deployment because the same configuration can > > > be > > > > used across all brokers, however, in a large system where disk failure > > > is > > > > a norm, the meta file could often get lost, causing a new broker id > > > being > > > > allocated. This is problematic because new broker id has no partition > > > > assigned to it so it can’t do anything, while partitions assigned to > > > the > > > > old one lose one replica > > > > > > If all of the disks have failed, then the partitions will lose their > > > replicas no matter what, right? If any of the disks is still around, > > > then there will be a meta file on the disk which contains the previous > > > broker ID. So I'm not sure that we need to change anything here. > > > > > > best, > > > Colin > > > > > > > > > On Tue, Feb 5, 2019, at 14:38, Li Kan wrote: > > > > Hi, I have KIP-426, which is a small change on automatically determining > > > > broker id when starting up. I am new to Kafka so there are a bunch of > > > > design trade-offs that I might be missing or hard to decide, so I'd > > > > like to > > > > get some suggestions on it. I'd expect (and open) to modify (or even > > > > totally rewrite) the KIP based on suggestions. Thanks. > > > > > > > > -- > > > > Best, > > > > Kan > > > > > > > > > >
Re: [DISCUSS] KIP-427: Add AtMinIsr topic partition category (new metric & TopicCommand option)
Hi Dong, I think AtMinIsr is still valuable to indicate cluster is at a critical state and something needs to be done asap to restore. To your example " let's say min_isr = 1 and replica_set_size = 3, it is > still possible that planned maintenance (e.g. one broker restart + > partition reassignment) can cause isr size drop to 1. Since AtMinIsr can > also cause fault positive (i.e. the fact that AtMinIsr > 0 does not > necessarily need attention from user), " One broker restart shouldn't cause ISR to drop to 1 from 3 unless 2 partitions are co-located on the same broker. This is still a valuable indicator to the admins that the partition assignment needs to be moved. In our case, we run 4 replicas for critical topics with min.isr = 2 . URPs are not really good indicator to take immediate action if one of the replicas is down. If 2 replicas are down and we are at 2 alive replicas this is stop everything to restore the cluster to a good state. Thanks, Harsha On Wed, Feb 27, 2019, at 11:17 PM, Dong Lin wrote: > Hey Kevin, > > Thanks for the update. > > The KIP suggests that AtMinIsr is better than UnderReplicatedPartition as > indicator for alerting. However, in most case where min_isr = > replica_set_size - 1, these two metrics are exactly the same, where planned > maintenance can easily cause positive AtMinIsr value. In the other > scenario, for example let's say min_isr = 1 and replica_set_size = 3, it is > still possible that planned maintenance (e.g. one broker restart + > partition reassignment) can cause isr size drop to 1. Since AtMinIsr can > also cause fault positive (i.e. the fact that AtMinIsr > 0 does not > necessarily need attention from user), I am not sure it is worth to add > this metric. > > In the Usage section, it is mentioned that user needs to manually check > whether there is ongoing maintenance after AtMinIsr is triggered. Could you > explain how is this different from the current way where we use > UnderReplicatedPartition to trigger alert? More specifically, can we just > replace AtMinIsr with UnderReplicatedPartition in the Usage section? > > Thanks, > Dong > > > On Tue, Feb 26, 2019 at 6:49 PM Kevin Lu wrote: > > > Hi Dong! > > > > Thanks for the feedback! > > > > You bring up a good point in that the AtMinIsr metric cannot be used to > > identify failure in the mentioned scenarios. I admit the motivation section > > placed too much emphasis on "identifying failure". > > > > I have modified the KIP to reflect the implementation as the AtMinIsr > > metric is intended to serve as a warning as one more failure to a partition > > AtMinIsr will cause producers with acks=ALL configured to fail. It has an > > additional benefit when minIsr=1 as it will warn us that the entire > > partition is at risk of going offline, but that is more of a side effect > > that only applies in that scenario (minIsr=1). > > > > Regards, > > Kevin > > > > On Tue, Feb 26, 2019 at 5:11 PM Dong Lin wrote: > > > > > Hey Kevin, > > > > > > Thanks for the proposal! > > > > > > It seems that the proposed implementation does not match the motivation. > > > The motivation suggests that the operator wants to tell the planned > > > maintenance (e.g. broker restart) from unplanned failure (e.g. network > > > failure). But the use of the metric AtMinIsr does not really > > differentiate > > > between these causes of the reduced number of ISR. For example, an > > > unplanned failure can cause ISR to drop from 3 to 2 but it can still be > > > higher than the minIsr (say 1). And a planned maintenance can cause ISR > > to > > > drop from 3 to 2, which trigger the AtMinIsr metric if minIsr=2. Can you > > > update the design doc to fix or explain this issue? > > > > > > Thanks, > > > Dong > > > > > > On Tue, Feb 12, 2019 at 9:02 AM Kevin Lu wrote: > > > > > > > Hi All, > > > > > > > > Getting the discussion thread started for KIP-427 in case anyone is > > free > > > > right now. > > > > > > > > I’d like to propose a new category of topic partitions *AtMinIsr* which > > > are > > > > partitions that only have the minimum number of in sync replicas left > > in > > > > the ISR set (as configured by min.insync.replicas). > > > > > > > > This would add two new metrics *ReplicaManager.AtMinIsrPartitionCount > > *& > > > > *Partition.AtMinIsr*, and a new TopicCommand option* > > > > --at-min-isr-partitions* to help in monitoring and alerting. > > > > > > > > KIP link: KIP-427: Add AtMinIsr topic partition category (new metric & > > > > TopicCommand option) > > > > < > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103089398 > > > > > > > > > > > > > Please take a look and let me know what you think. > > > > > > > > Regards, > > > > Kevin > > > > > > > > > >
Re: [DISCUSS] KIP-433: Provide client API version to authorizer
HI Colin, Overlooked the IDEMPOTENT_WRITE ACL. This along with client.min.version should solve the cases proposed in the KIP. Can we turn this KIP into adding min.client.version config to broker and it could be part of the dynamic config . Thanks, Harsha On Wed, Feb 27, 2019, at 12:17 PM, Colin McCabe wrote: > On Tue, Feb 26, 2019, at 16:33, Harsha wrote: > > Hi Colin, > > > > "> I think Ismael and Gwen here bring up a good point. The version of the > > > request is a technical detail that isn't really related to > > > authorization. There are a lot of other technical details like this > > > like the size of the request, the protocol it came in on, etc. None of > > > them are passed to the authorizer-- they all have configuration knobs > > > to control how we handle them. If we add this technical detail, > > > logically we'll have to start adding all the others, and the authorizer > > > API will get really bloated. It's better to keep it focused on > > > authorization, I think." > > > > probably my previous email is not clear but I am agreeing with Gwen's > > point. > > I am not in favor of extending authorizer to support this. > > > > > > "> Another thing to consider is that if we add a new broker configuration > > > that lets us set a minimum client version which is allowed, that could > > > be useful to other users as well. On the other hand, most users are > > > not likely to write a custom authorizer to try to take advantage of > > > version information being passed to the authorizer. So, I think using> a > > > configuration is clearly the better way to go here. Perhaps it can > > > be a KIP-226 dynamic configuration to make this easier to deploy?" > > > > Although minimum client version might help to a certain extent there > > are other cases where we want users to not start using transactions for > > example. My proposal in the previous thread was to introduce another > > module/interface, let's say > > "SupportedAPIs" which will take in dynamic configuration to check which > > APIs are allowed. > > It can throw UnsupportedException just like we are throwing > > Authorization Exception. > > Hi Harsha, > > We can already prevent people from using transactions using ACLs, > right? That's what the IDEMPOTENT_WRITE ACL was added for. > > In general, I think users should be able to think of ACLs in terms of > "what can I do" rather than "how is it implemented." For example, > maybe some day we will replace FetchRequest with GetStuffRequest. But > users who have READ permission on a topic shouldn't have to change > anything. So I think the Authorizer interface should not be aware of > individual RPC types or message versions. > > best, > Colin > > > > > > > > Thanks, > > Harsha > > > > > > n Tue, Feb 26, 2019, at 10:04 AM, Colin McCabe wrote: > > > Hi Harsha, > > > > > > I think Ismael and Gwen here bring up a good point. The version of the > > > request is a technical detail that isn't really related to > > > authorization. There are a lot of other technical details like this > > > like the size of the request, the protocol it came in on, etc. None of > > > them are passed to the authorizer-- they all have configuration knobs > > > to control how we handle them. If we add this technical detail, > > > logically we'll have to start adding all the others, and the authorizer > > > API will get really bloated. It's better to keep it focused on > > > authorization, I think. > > > > > > Another thing to consider is that if we add a new broker configuration > > > that lets us set a minimum client version which is allowed, that could > > > be useful to other users as well. On the other hand, most users are > > > not likely to write a custom authorizer to try to take advantage of > > > version information being passed to the authorizer. So, I think using > > > a configuration is clearly the better way to go here. Perhaps it can > > > be a KIP-226 dynamic configuration to make this easier to deploy? > > > > > > cheers, > > > Colin > > > > > > > > > On Mon, Feb 25, 2019, at 15:43, Harsha wrote: > > > > Hi Ying, > > > > I think the question is can we add a module in the core which > > > > can take up the dynamic config and does a block certain APIs. Thi
