[jira] [Updated] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-3729: --- Description: KIP WIP: [https://cwiki.apache.org/confluence/display/KAFKA/KIP-463%3A+Auto-configure+non-default+Serdes+passed+alongside+the+TopologyBuilder] >From Guozhang Wang: "Only default serdes provided through configs are auto-configured today. But we could auto-configure other serdes passed alongside the topology builder as well." After the PR was merged, we realized that the current approach to implement this features is actually not backward compatible. Thus, we need to revert the commit for now to not break backward compatibility in 2.3 release. After some more thinking, it seems that this feature is actually more complicated to get right as it seem on the surface and hence it would required a proper KIP. The following issues are identified: * in the new code, configure() would be called twice, one in user code (if people don't rewrite existing applications) and later via Kafka Streams – the second call could "reconfigure" the Serde and overwrite the correct configuration from the first call done by the user * if there are multiple Serdes using the same configuration parameters names, it's only possible to specify this parameter name once in the global StreamsConfig; hence, it's not possible for users to configure both Serdes differently * basically, the global StreamsConfig needs to contain all configuration parameters over all used Serdes to make a potential second call to `configure()` idempotant To address the issues, some ideas would be: * pass in the configuration via the constructor and deprecate `configure()` method * add a new method `isConfigured()` that would allow to skip the second configuration call within Kafka Streams runtime There might be other ways to address this, and the different options should be discussed on the KIP. was: >From Guozhang Wang: "Only default serdes provided through configs are auto-configured today. But we could auto-configure other serdes passed alongside the topology builder as well." After the PR was merged, we realized that the current approach to implement this features is actually not backward compatible. Thus, we need to revert the commit for now to not break backward compatibility in 2.3 release. After some more thinking, it seems that this feature is actually more complicated to get right as it seem on the surface and hence it would required a proper KIP. The following issues are identified: * in the new code, configure() would be called twice, one in user code (if people don't rewrite existing applications) and later via Kafka Streams – the second call could "reconfigure" the Serde and overwrite the correct configuration from the first call done by the user * if there are multiple Serdes using the same configuration parameters names, it's only possible to specify this parameter name once in the global StreamsConfig; hence, it's not possible for users to configure both Serdes differently * basically, the global StreamsConfig needs to contain all configuration parameters over all used Serdes to make a potential second call to `configure()` idempotant To address the issues, some ideas would be: * pass in the configuration via the constructor and deprecate `configure()` method * add a new method `isConfigured()` that would allow to skip the second configuration call within Kafka Streams runtime There might be other ways to address this, and the different options should be discussed on the KIP. > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Priority: Major > Labels: api, needs-dicussion, needs-kip, newbie > Attachments: 3729.txt, 3729.v6.txt > > > KIP WIP: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-463%3A+Auto-configure+non-default+Serdes+passed+alongside+the+TopologyBuilder] > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." > After the PR was merged, we realized that the current approach to implement > this features is actually not backward compatible. Thus, we need to revert > the commit for now to not break backward compatibility in 2.3 release. After > some more thinking, it seems that this feature is actually more complicated > to get right as it seem on the surface and hence it would required a proper > KIP. > The following issues are identified: > * in
[jira] [Updated] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Yu updated KAFKA-3729: -- Labels: api needs-kip newbie (was: api newbie) > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Assignee: Ted Yu >Priority: Major > Labels: api, needs-kip, newbie > Attachments: 3729.txt, 3729.v6.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." > After the PR was merged, we realized that the current approach to implement > this features is actually not backward compatible. Thus, we need to revert > the commit for now to not break backward compatibility in 2.3 release. After > some more thinking, it seems that this feature is actually more complicated > to get right as it seem on the surface and hence it would required a proper > KIP. > The following issues are identified: > * in the new code, configure() would be called twice, one in user code (if > people don't rewrite existing applications) and later via Kafka Streams – the > second call could "reconfigure" the Serde and overwrite the correct > configuration from the first call done by the user > * if there are multiple Serdes using the same configuration parameters > names, it's only possible to specify this parameter name once in the global > StreamsConfig; hence, it's not possible for users to configure both Serdes > differently > * basically, the global StreamsConfig needs to contain all configuration > parameters over all used Serdes to make a potential second call to > `configure()` idempotant > To address the issues, some ideas would be: > * pass in the configuration via the constructor and deprecate `configure()` > method > * add a new method `isConfigured()` that would allow to skip the second > configuration call within Kafka Streams runtime > There might be other ways to address this, and the different options should > be discussed on the KIP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Yu updated KAFKA-3729: -- Labels: api needs-dicussion needs-kip newbie (was: api needs-kip newbie) > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Assignee: Ted Yu >Priority: Major > Labels: api, needs-dicussion, needs-kip, newbie > Attachments: 3729.txt, 3729.v6.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." > After the PR was merged, we realized that the current approach to implement > this features is actually not backward compatible. Thus, we need to revert > the commit for now to not break backward compatibility in 2.3 release. After > some more thinking, it seems that this feature is actually more complicated > to get right as it seem on the surface and hence it would required a proper > KIP. > The following issues are identified: > * in the new code, configure() would be called twice, one in user code (if > people don't rewrite existing applications) and later via Kafka Streams – the > second call could "reconfigure" the Serde and overwrite the correct > configuration from the first call done by the user > * if there are multiple Serdes using the same configuration parameters > names, it's only possible to specify this parameter name once in the global > StreamsConfig; hence, it's not possible for users to configure both Serdes > differently > * basically, the global StreamsConfig needs to contain all configuration > parameters over all used Serdes to make a potential second call to > `configure()` idempotant > To address the issues, some ideas would be: > * pass in the configuration via the constructor and deprecate `configure()` > method > * add a new method `isConfigured()` that would allow to skip the second > configuration call within Kafka Streams runtime > There might be other ways to address this, and the different options should > be discussed on the KIP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-3729: --- Fix Version/s: (was: 2.3.0) > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Assignee: Ted Yu >Priority: Major > Labels: api, newbie > Attachments: 3729.txt, 3729.v6.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." > After the PR was merged, we realized that the current approach to implement > this features is actually not backward compatible. Thus, we need to revert > the commit for now to not break backward compatibility in 2.3 release. After > some more thinking, it seems that this feature is actually more complicated > to get right as it seem on the surface and hence it would required a proper > KIP. > The following issues are identified: > * in the new code, configure() would be called twice, one in user code (if > people don't rewrite existing applications) and later via Kafka Streams – the > second call could "reconfigure" the Serde and overwrite the correct > configuration from the first call done by the user > * if there are multiple Serdes using the same configuration parameters > names, it's only possible to specify this parameter name once in the global > StreamsConfig; hence, it's not possible for users to configure both Serdes > differently > * basically, the global StreamsConfig needs to contain all configuration > parameters over all used Serdes to make a potential second call to > `configure()` idempotant > To address the issues, some ideas would be: > * pass in the configuration via the constructor and deprecate `configure()` > method > * add a new method `isConfigured()` that would allow to skip the second > configuration call within Kafka Streams runtime > There might be other ways to address this, and the different options should > be discussed on the KIP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-3729: --- Description: >From Guozhang Wang: "Only default serdes provided through configs are auto-configured today. But we could auto-configure other serdes passed alongside the topology builder as well." After the PR was merged, we realized that the current approach to implement this features is actually not backward compatible. Thus, we need to revert the commit for now to not break backward compatibility in 2.3 release. After some more thinking, it seems that this feature is actually more complicated to get right as it seem on the surface and hence it would required a proper KIP. The following issues are identified: * in the new code, configure() would be called twice, one in user code (if people don't rewrite existing applications) and later via Kafka Streams – the second call could "reconfigure" the Serde and overwrite the correct configuration from the first call done by the user * if there are multiple Serdes using the same configuration parameters names, it's only possible to specify this parameter name once in the global StreamsConfig; hence, it's not possible for users to configure both Serdes differently * basically, the global StreamsConfig needs to contain all configuration parameters over all used Serdes to make a potential second call to `configure()` idempotant To address the issues, some ideas would be: * pass in the configuration via the constructor and deprecate `configure()` method * add a new method `isConfigured()` that would allow to skip the second configuration call within Kafka Streams runtime There might be other ways to address this, and the different options should be discussed on the KIP. was: >From Guozhang Wang: "Only default serdes provided through configs are auto-configured today. But we could auto-configure other serdes passed alongside the topology builder as well." > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Assignee: Ted Yu >Priority: Major > Labels: api, newbie > Fix For: 2.3.0 > > Attachments: 3729.txt, 3729.v6.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." > After the PR was merged, we realized that the current approach to implement > this features is actually not backward compatible. Thus, we need to revert > the commit for now to not break backward compatibility in 2.3 release. After > some more thinking, it seems that this feature is actually more complicated > to get right as it seem on the surface and hence it would required a proper > KIP. > The following issues are identified: > * in the new code, configure() would be called twice, one in user code (if > people don't rewrite existing applications) and later via Kafka Streams – the > second call could "reconfigure" the Serde and overwrite the correct > configuration from the first call done by the user > * if there are multiple Serdes using the same configuration parameters > names, it's only possible to specify this parameter name once in the global > StreamsConfig; hence, it's not possible for users to configure both Serdes > differently > * basically, the global StreamsConfig needs to contain all configuration > parameters over all used Serdes to make a potential second call to > `configure()` idempotant > To address the issues, some ideas would be: > * pass in the configuration via the constructor and deprecate `configure()` > method > * add a new method `isConfigured()` that would allow to skip the second > configuration call within Kafka Streams runtime > There might be other ways to address this, and the different options should > be discussed on the KIP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KAFKA-3729: -- Attachment: (was: 3729.v6.txt) > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Assignee: Ted Yu >Priority: Major > Labels: api, newbie > Attachments: 3729.txt, 3729.v6.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KAFKA-3729: -- Attachment: 3729.v6.txt > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Assignee: Ted Yu >Priority: Major > Labels: api, newbie > Attachments: 3729.txt, 3729.v6.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KAFKA-3729: -- Attachment: 3729.v6.txt > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Assignee: Ted Yu >Priority: Major > Labels: api, newbie > Attachments: 3729.txt, 3729.v6.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KAFKA-3729: -- Attachment: 3729.txt > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Priority: Major > Labels: api, newbie > Attachments: 3729.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
