[jira] [Updated] (KAFKA-6555) Making state store queryable during restoration
[ https://issues.apache.org/jira/browse/KAFKA-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-6555: --- Flags: (was: Important) > Making state store queryable during restoration > --- > > Key: KAFKA-6555 > URL: https://issues.apache.org/jira/browse/KAFKA-6555 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Ashish Surana >Assignee: Ashish Surana >Priority: Major > > State store in Kafka streams are currently only queryable when StreamTask is > in RUNNING state. The idea is to make it queryable even in the RESTORATION > (PARTITION_ASSIGNED) state as the time spend on restoration can be huge and > making the data inaccessible during this time could be downtime not suitable > for many applications. > When the active partition goes down then one of the following occurs: > # One of the standby replica partition gets promoted to active: Replica task > has to restore the remaining state from the changelog topic before it can > become RUNNING. The time taken for this depends on how much the replica is > lagging behind. During this restoration time the state store for that > partition is currently not queryable resulting in the partition downtime. We > can make the state store partition queryable for the data already present in > the state store. > # When there is no replica or standby task, then active task will be started > in one of the existing node. That node has to build the entire state from the > changelog topic which can take lot of time depending on how big is the > changelog topic, and keeping state store not queryable during this time is > the downtime for the parition. > It's very important improvement as it could simply improve the availability > of microservices developed using kafka streams. > I am working on a patch for this change. Any feedback or comments are welcome. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6555) Making state store queryable during restoration
[ https://issues.apache.org/jira/browse/KAFKA-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Surana updated KAFKA-6555: - Description: State store in Kafka streams are currently only queryable when StreamTask is in RUNNING state. The idea is to make it queryable even in the RESTORATION (PARTITION_ASSIGNED) state as the time spend on restoration can be huge and making the data inaccessible during this time could be downtime not suitable for many applications. When the active partition goes down then one of the following occurs: # One of the standby replica partition gets promoted to active: Replica task has to restore the remaining state from the changelog topic before it can become RUNNING. The time taken for this depends on how much the replica is lagging behind. During this restoration time the state store for that partition is currently not queryable resulting in the partition downtime. We can make the state store partition queryable for the data already present in the state store. # When there is no replica or standby task, then active task will be started in one of the existing node. That node has to build the entire state from the changelog topic which can take lot of time depending on how big is the changelog topic, and keeping state store not queryable during this time is the downtime for the parition. It's very important improvement as it could simply improve the availability of microservices developed using kafka streams. I am working on a patch for this change. Any feedback or comments are welcome. was: State store in Kafka streams are currently only queryable when StreamTask is in RUNNING state. The idea is to make it queryable even in the RESTORATION (PARTITION_ASSIGNED) state as the time spend on restoration can be huge and making the data inaccessible during this time could be downtime not suitable for many applications. When the active partition goes down then one of the following occurs: # One of the standby replica partition gets promoted to active: Replica task has to restore the remaining state from the changelog topic before it can become RUNNING. The time taken for this depends on how much the replica is lagging behind. During this restoration time the state store for that partition is currently not queryable giving making the partition down. We can make the state store partition queryable for the data already present in the state store. # When there is no replica or standby task, then active task will be started in one of the existing node. That node has to build the entire state from the changelog topic which can take lot of time depending on how big is the changelog topic, and keeping state store not queryable during this time is the downtime for the parition. It's very important improvement as it could simply improve the availability of microservices developed using kafka streams. I am working on a patch for this change. Any feedback or comments are welcome. > Making state store queryable during restoration > --- > > Key: KAFKA-6555 > URL: https://issues.apache.org/jira/browse/KAFKA-6555 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Ashish Surana >Priority: Major > > State store in Kafka streams are currently only queryable when StreamTask is > in RUNNING state. The idea is to make it queryable even in the RESTORATION > (PARTITION_ASSIGNED) state as the time spend on restoration can be huge and > making the data inaccessible during this time could be downtime not suitable > for many applications. > When the active partition goes down then one of the following occurs: > # One of the standby replica partition gets promoted to active: Replica task > has to restore the remaining state from the changelog topic before it can > become RUNNING. The time taken for this depends on how much the replica is > lagging behind. During this restoration time the state store for that > partition is currently not queryable resulting in the partition downtime. We > can make the state store partition queryable for the data already present in > the state store. > # When there is no replica or standby task, then active task will be started > in one of the existing node. That node has to build the entire state from the > changelog topic which can take lot of time depending on how big is the > changelog topic, and keeping state store not queryable during this time is > the downtime for the parition. > It's very important improvement as it could simply improve the availability > of microservices developed using kafka streams. > I am working on a patch for this change. Any feedback or comments are welcome. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6555) Making state store queryable during restoration
[ https://issues.apache.org/jira/browse/KAFKA-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Surana updated KAFKA-6555: - Description: State store in Kafka streams are currently only queryable when StreamTask is in RUNNING state. The idea is to make it queryable even in the RESTORATION (PARTITION_ASSIGNED) state as the time spend on restoration can be huge and making the data inaccessible during this time could be downtime not suitable for many applications. When the active partition goes down then one of the following occurs: # One of the standby replica partition gets promoted to active: Replica task has to restore the remaining state from the changelog topic before it can become RUNNING. The time taken for this depends on how much the replica is lagging behind. During this restoration time the state store for that partition is currently not queryable giving making the partition down. We can make the state store partition queryable for the data already present in the state store. # When there is no replica or standby task, then active task will be started in one of the existing node. That node has to build the entire state from the changelog topic which can take lot of time depending on how big is the changelog topic, and keeping state store not queryable during this time is the downtime for the parition. It's very important improvement as it could simply improve the availability of microservices developed using kafka streams. I am working on a patch for this change. Any feedback or comments are welcome. was: State store in Kafka streams are currently only queryable when StreamTask is in RUNNING state. The idea is to make it queryable even in the RESTORATION (PARTITION_ASSIGNED) state as the time spend on restoration can be huge and making the data inaccessible during this time could be downtime not suitable for many applications. When the active partition goes down then one of the following occurs: # One of the standby replica partition gets promoted to active: Replica task has to restore the remaining state from the changelog topic before it can become RUNNING. The time taken for this depends on how much the replica is lagging behind. During this restoration time the state store for that partition is currently not queryable giving making the partition down. We can make the state store partition queryable for the data already present in the state store. # When there is no replica or standby task, then active task will be started in one of the existing node. That node has to build the entire state from the changelog topic which can take lot of time depending on how big is the changelog topic, and keeping state store not queryable during this time is the downtime for the parition. It's very important improvement as it could simply improve the availability of microservices developed using kafka streams. Any feedback or comments are welcome. > Making state store queryable during restoration > --- > > Key: KAFKA-6555 > URL: https://issues.apache.org/jira/browse/KAFKA-6555 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Ashish Surana >Priority: Major > > State store in Kafka streams are currently only queryable when StreamTask is > in RUNNING state. The idea is to make it queryable even in the RESTORATION > (PARTITION_ASSIGNED) state as the time spend on restoration can be huge and > making the data inaccessible during this time could be downtime not suitable > for many applications. > When the active partition goes down then one of the following occurs: > # One of the standby replica partition gets promoted to active: Replica task > has to restore the remaining state from the changelog topic before it can > become RUNNING. The time taken for this depends on how much the replica is > lagging behind. During this restoration time the state store for that > partition is currently not queryable giving making the partition down. We can > make the state store partition queryable for the data already present in the > state store. > # When there is no replica or standby task, then active task will be started > in one of the existing node. That node has to build the entire state from the > changelog topic which can take lot of time depending on how big is the > changelog topic, and keeping state store not queryable during this time is > the downtime for the parition. > It's very important improvement as it could simply improve the availability > of microservices developed using kafka streams. > I am working on a patch for this change. Any feedback or comments are welcome. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6555) Making state store queryable during restoration
[ https://issues.apache.org/jira/browse/KAFKA-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Surana updated KAFKA-6555: - Description: State store in Kafka streams are currently only queryable when StreamTask is in RUNNING state. The idea is to make it queryable even in the RESTORATION (PARTITION_ASSIGNED) state as the time spend on restoration can be huge and making the data inaccessible during this time could be downtime not suitable for many applications. When the active partition goes down then one of the following occurs: # One of the standby replica partition gets promoted to active: Replica task has to restore the remaining state from the changelog topic before it can become RUNNING. The time taken for this depends on how much the replica is lagging behind. During this restoration time the state store for that partition is currently not queryable giving making the partition down. We can make the state store partition queryable for the data already present in the state store. # When there is no replica or standby task, then active task will be started in one of the existing node. That node has to build the entire state from the changelog topic which can take lot of time depending on how big is the changelog topic, and keeping state store not queryable during this time is the downtime for the parition. It's very important improvement as it could simply improve the availability of microservices developed using kafka streams. Any feedback or comments are welcome. was: State store in Kafka streams are currently only queryable when StreamTask is in RUNNING state. The idea is to make it queryable even in the RESTORATION (PARTITION_ASSIGNED) state as the time spend on restoration can be huge and making the data inaccessible during this time could be downtime not suitable for many applications. When the active partition goes down then one of the following occurs: # One of the standby replica partition gets promoted to active: Replica task has to restore the remaining state from the changelog topic before it can become RUNNING. The time taken for this depends on how much the replica is lagging behind. During this restoration time the state store for that partition is currently not queryable giving making the partition down. We can make the state store partition queryable for the data already present in the state store. # When there is no replica or standby task, then active task will be started in one of the existing node. That node has to build the entire state from the changelog topic which can take lot of time depending on how big is the changelog topic, and keeping state store not queryable during this time is the downtime for the parition. It's very important improvement as it could simply improve the availability of microservices developed using kafka streams. Any feedback or comments are welcome. > Making state store queryable during restoration > --- > > Key: KAFKA-6555 > URL: https://issues.apache.org/jira/browse/KAFKA-6555 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Ashish Surana >Priority: Major > > State store in Kafka streams are currently only queryable when StreamTask is > in RUNNING state. The idea is to make it queryable even in the RESTORATION > (PARTITION_ASSIGNED) state as the time spend on restoration can be huge and > making the data inaccessible during this time could be downtime not suitable > for many applications. > When the active partition goes down then one of the following occurs: > # One of the standby replica partition gets promoted to active: Replica task > has to restore the remaining state from the changelog topic before it can > become RUNNING. The time taken for this depends on how much the replica is > lagging behind. During this restoration time the state store for that > partition is currently not queryable giving making the partition down. We can > make the state store partition queryable for the data already present in the > state store. > # When there is no replica or standby task, then active task will be started > in one of the existing node. That node has to build the entire state from the > changelog topic which can take lot of time depending on how big is the > changelog topic, and keeping state store not queryable during this time is > the downtime for the parition. > It's very important improvement as it could simply improve the availability > of microservices developed using kafka streams. Any feedback or comments are > welcome. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6555) Making state store queryable during restoration
[ https://issues.apache.org/jira/browse/KAFKA-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Surana updated KAFKA-6555: - Description: State store in Kafka streams are currently only queryable when StreamTask is in RUNNING state. The idea is to make it queryable even in the RESTORATION (PARTITION_ASSIGNED) state as the time spend on restoration can be huge and making the data inaccessible during this time could be downtime not suitable for many applications. When the active partition goes down then one of the following occurs: # One of the standby replica partition gets promoted to active: Replica task has to restore the remaining state from the changelog topic before it can become RUNNING. The time taken for this depends on how much the replica is lagging behind. During this restoration time the state store for that partition is currently not queryable giving making the partition down. We can make the state store partition queryable for the data already present in the state store. # When there is no replica or standby task, then active task will be started in one of the existing node. That node has to build the entire state from the changelog topic which can take lot of time depending on how big is the changelog topic, and keeping state store not queryable during this time is the downtime for the parition. It's very important improvement as it could simply improve the availability of microservices developed using kafka streams. Any feedback or comments are welcome. was: State store in Kafka streams are currently only queryable when StreamTask is in RUNNING state. The idea is to make it queryable even in the RESTORATION (PARTITION_ASSIGNED) state as the time spend on restoration can be huge and making the data inaccessible during this time could be downtime not suitable for many applications. When the active partition goes down then one of the following occurs: # One of the standby replica partition gets promoted to active: Replica restores the remaining state from the changelog topic. The time taken for this depends on how much the replica is lagging behind in consuming changelog topic. During this restoration time the state store for that partition is not queryable giving us the downtime for the entire parition. We can make it queryable for the data already present in the state store. # When there is no replica or standby task, then active task will be started in one of the existing node. That node has to build the entire state from the changelog topic which can take lot of time depending on how big is the changelog topic, and keeping state store not queryable during this time is the downtime for the parition. It's very important improvement as it could simply improve the availability of microservices developed using kafka streams. Any feedback or comments are welcome. > Making state store queryable during restoration > --- > > Key: KAFKA-6555 > URL: https://issues.apache.org/jira/browse/KAFKA-6555 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Ashish Surana >Priority: Major > > State store in Kafka streams are currently only queryable when StreamTask is > in RUNNING state. The idea is to make it queryable even in the RESTORATION > (PARTITION_ASSIGNED) state as the time spend on restoration can be huge and > making the data inaccessible during this time could be downtime not suitable > for many applications. > When the active partition goes down then one of the following occurs: > # One of the standby replica partition gets promoted to active: Replica task > has to restore the remaining state from the changelog topic before it can > become RUNNING. The time taken for this depends on how much the replica is > lagging behind. During this restoration time the state store for that > partition is currently not queryable giving making the partition down. We can > make the state store partition queryable for the data already present in the > state store. > # When there is no replica or standby task, then active task will be started > in one of the existing node. That node has to build the entire state from the > changelog topic which can take lot of time depending on how big is the > changelog topic, and keeping state store not queryable during this time is > the downtime for the parition. > > It's very important improvement as it could simply improve the availability > of microservices developed using kafka streams. Any feedback or comments are > welcome. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6555) Making state store queryable during restoration
[ https://issues.apache.org/jira/browse/KAFKA-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Surana updated KAFKA-6555: - Flags: Important > Making state store queryable during restoration > --- > > Key: KAFKA-6555 > URL: https://issues.apache.org/jira/browse/KAFKA-6555 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Ashish Surana >Priority: Major > > State store in Kafka streams are currently only queryable when StreamTask is > in RUNNING state. The idea is to make it queryable even in the RESTORATION > (PARTITION_ASSIGNED) state as the time spend on restoration can be huge and > making the data inaccessible during this time could be downtime not suitable > for many applications. > When the active partition goes down then one of the following occurs: > # One of the standby replica partition gets promoted to active: Replica > restores the remaining state from the changelog topic. The time taken for > this depends on how much the replica is lagging behind in consuming changelog > topic. During this restoration time the state store for that partition is not > queryable giving us the downtime for the entire parition. We can make it > queryable for the data already present in the state store. > # When there is no replica or standby task, then active task will be started > in one of the existing node. That node has to build the entire state from the > changelog topic which can take lot of time depending on how big is the > changelog topic, and keeping state store not queryable during this time is > the downtime for the parition. > > It's very important improvement as it could simply improve the availability > of microservices developed using kafka streams. Any feedback or comments are > welcome. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
