[jira] [Updated] (KAFKA-8395) Add an ability to backup log segment files on truncation
[ https://issues.apache.org/jira/browse/KAFKA-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe updated KAFKA-8395: --- Fix Version/s: (was: 2.2.2) (was: 2.3.0) > Add an ability to backup log segment files on truncation > > > Key: KAFKA-8395 > URL: https://issues.apache.org/jira/browse/KAFKA-8395 > Project: Kafka > Issue Type: New Feature > Components: core >Affects Versions: 2.2.0 >Reporter: Michael Axiak >Priority: Minor > Labels: needs-kip > > At HubSpot, we believe we hit a combination of bugs [1] [2], which may have > caused us to lose data. In this scenario, as part of metadata conflict > resolution a slowly starting up broker recovered an offset of zero and > truncated segment files. > As part of a belt-and-suspenders approach to reducing this risk in the > future, I propose adding the ability to rename/backup these files and > allowing kafka to move on. Note that this breaks the ordering guarantees, but > allows one to recover the data and decide later how to approach it. > This feature should be turned off by default but enabled with a configuration > option. > (A pull request is following soon on Github) > 1: https://issues.apache.org/jira/browse/KAFKA-2178 > 2: https://issues.apache.org/jira/browse/KAFKA-1120 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8395) Add an ability to backup log segment files on truncation
[ https://issues.apache.org/jira/browse/KAFKA-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe updated KAFKA-8395: --- Labels: needs-kip (was: ) > Add an ability to backup log segment files on truncation > > > Key: KAFKA-8395 > URL: https://issues.apache.org/jira/browse/KAFKA-8395 > Project: Kafka > Issue Type: New Feature > Components: core >Affects Versions: 2.2.0 >Reporter: Michael Axiak >Priority: Minor > Labels: needs-kip > Fix For: 2.3.0, 2.2.2 > > > At HubSpot, we believe we hit a combination of bugs [1] [2], which may have > caused us to lose data. In this scenario, as part of metadata conflict > resolution a slowly starting up broker recovered an offset of zero and > truncated segment files. > As part of a belt-and-suspenders approach to reducing this risk in the > future, I propose adding the ability to rename/backup these files and > allowing kafka to move on. Note that this breaks the ordering guarantees, but > allows one to recover the data and decide later how to approach it. > This feature should be turned off by default but enabled with a configuration > option. > (A pull request is following soon on Github) > 1: https://issues.apache.org/jira/browse/KAFKA-2178 > 2: https://issues.apache.org/jira/browse/KAFKA-1120 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8395) Add an ability to backup log segment files on truncation
[ https://issues.apache.org/jira/browse/KAFKA-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian updated KAFKA-8395: --- Fix Version/s: (was: 2.2.1) 2.2.2 2.3.0 > Add an ability to backup log segment files on truncation > > > Key: KAFKA-8395 > URL: https://issues.apache.org/jira/browse/KAFKA-8395 > Project: Kafka > Issue Type: New Feature > Components: core >Affects Versions: 2.2.0 >Reporter: Michael Axiak >Priority: Minor > Fix For: 2.3.0, 2.2.2 > > > At HubSpot, we believe we hit a combination of bugs [1] [2], which may have > caused us to lose data. In this scenario, as part of metadata conflict > resolution a slowly starting up broker recovered an offset of zero and > truncated segment files. > As part of a belt-and-suspenders approach to reducing this risk in the > future, I propose adding the ability to rename/backup these files and > allowing kafka to move on. Note that this breaks the ordering guarantees, but > allows one to recover the data and decide later how to approach it. > This feature should be turned off by default but enabled with a configuration > option. > (A pull request is following soon on Github) > 1: https://issues.apache.org/jira/browse/KAFKA-2178 > 2: https://issues.apache.org/jira/browse/KAFKA-1120 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8395) Add an ability to backup log segment files on truncation
[ https://issues.apache.org/jira/browse/KAFKA-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Axiak updated KAFKA-8395: - Flags: Patch > Add an ability to backup log segment files on truncation > > > Key: KAFKA-8395 > URL: https://issues.apache.org/jira/browse/KAFKA-8395 > Project: Kafka > Issue Type: New Feature > Components: core >Affects Versions: 2.2.0 >Reporter: Michael Axiak >Priority: Minor > Fix For: 2.2.1 > > > At HubSpot, we believe we hit a combination of bugs [1] [2], which may have > caused us to lose data. In this scenario, as part of metadata conflict > resolution a slowly starting up broker recovered an offset of zero and > truncated segment files. > As part of a belt-and-suspenders approach to reducing this risk in the > future, I propose adding the ability to rename/backup these files and > allowing kafka to move on. Note that this breaks the ordering guarantees, but > allows one to recover the data and decide later how to approach it. > This feature should be turned off by default but enabled with a configuration > option. > (A pull request is following soon on Github) > 1: https://issues.apache.org/jira/browse/KAFKA-2178 > 2: https://issues.apache.org/jira/browse/KAFKA-1120 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
