[jira] [Updated] (KAFKA-1933) Fine-grained locking in log append
[ https://issues.apache.org/jira/browse/KAFKA-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1933: --- Fix Version/s: (was: 0.8.3) > Fine-grained locking in log append > -- > > Key: KAFKA-1933 > URL: https://issues.apache.org/jira/browse/KAFKA-1933 > Project: Kafka > Issue Type: Improvement > Components: log >Reporter: Maxim Ivanov >Assignee: Maxim Ivanov >Priority: Minor > Attachments: KAFKA-1933.patch, KAFKA-1933_2015-02-09_12:27:06.patch > > > This patch adds finer locking when appending to log. It breaks > global append lock into 2 sequential and 1 parallel phase. > Basic idea is to allow every thread to "reserve" offsets in non > overlapping ranges, then do compression in parallel and then > "commit" write to log in the same order offsets where reserved. > Results on a server with 16 cores CPU available: > gzip: 564.0 sec -> 45.2 sec (12.4x speedup) > LZ4: 56.7 sec -> 9.9 sec (5.7x speedup) > Kafka was configured to run 16 IO threads, data was pushed using 32 netcat > instances pushing in parallel batches of 200 msg 6.2 kb each (3264 MB in > total) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1933) Fine-grained locking in log append
[ https://issues.apache.org/jira/browse/KAFKA-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-1933: - Status: In Progress (was: Patch Available) These results are very good. I'm scared of the code. The change is quite small, but scary. You mentioned there another version of the patch coming which is simpler? We made some improvements to compression performance recently which should help this problem in the absence of this patch, is it still needed? Fine-grained locking in log append -- Key: KAFKA-1933 URL: https://issues.apache.org/jira/browse/KAFKA-1933 Project: Kafka Issue Type: Improvement Components: log Reporter: Maxim Ivanov Assignee: Maxim Ivanov Priority: Minor Fix For: 0.8.3 Attachments: KAFKA-1933.patch, KAFKA-1933_2015-02-09_12:27:06.patch This patch adds finer locking when appending to log. It breaks global append lock into 2 sequential and 1 parallel phase. Basic idea is to allow every thread to reserve offsets in non overlapping ranges, then do compression in parallel and then commit write to log in the same order offsets where reserved. Results on a server with 16 cores CPU available: gzip: 564.0 sec - 45.2 sec (12.4x speedup) LZ4: 56.7 sec - 9.9 sec (5.7x speedup) Kafka was configured to run 16 IO threads, data was pushed using 32 netcat instances pushing in parallel batches of 200 msg 6.2 kb each (3264 MB in total) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1933) Fine-grained locking in log append
[ https://issues.apache.org/jira/browse/KAFKA-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1933: --- Assignee: Maxim Ivanov (was: Jay Kreps) Fine-grained locking in log append -- Key: KAFKA-1933 URL: https://issues.apache.org/jira/browse/KAFKA-1933 Project: Kafka Issue Type: Improvement Components: log Reporter: Maxim Ivanov Assignee: Maxim Ivanov Priority: Minor Fix For: 0.8.3 Attachments: KAFKA-1933.patch, KAFKA-1933_2015-02-09_12:27:06.patch This patch adds finer locking when appending to log. It breaks global append lock into 2 sequential and 1 parallel phase. Basic idea is to allow every thread to reserve offsets in non overlapping ranges, then do compression in parallel and then commit write to log in the same order offsets where reserved. Results on a server with 16 cores CPU available: gzip: 564.0 sec - 45.2 sec (12.4x speedup) LZ4: 56.7 sec - 9.9 sec (5.7x speedup) Kafka was configured to run 16 IO threads, data was pushed using 32 netcat instances pushing in parallel batches of 200 msg 6.2 kb each (3264 MB in total) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1933) Fine-grained locking in log append
[ https://issues.apache.org/jira/browse/KAFKA-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Ivanov updated KAFKA-1933: Fix Version/s: (was: 0.8.2.0) 0.8.3 Fine-grained locking in log append -- Key: KAFKA-1933 URL: https://issues.apache.org/jira/browse/KAFKA-1933 Project: Kafka Issue Type: Improvement Components: log Reporter: Maxim Ivanov Assignee: Jay Kreps Priority: Minor Fix For: 0.8.3 Attachments: KAFKA-1933.patch, KAFKA-1933_2015-02-09_12:27:06.patch This patch adds finer locking when appending to log. It breaks global append lock into 2 sequential and 1 parallel phase. Basic idea is to allow every thread to reserve offsets in non overlapping ranges, then do compression in parallel and then commit write to log in the same order offsets where reserved. Results on a server with 16 cores CPU available: gzip: 564.0 sec - 45.2 sec (12.4x speedup) LZ4: 56.7 sec - 9.9 sec (5.7x speedup) Kafka was configured to run 16 IO threads, data was pushed using 32 netcat instances pushing in parallel batches of 200 msg 6.2 kb each (3264 MB in total) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1933) Fine-grained locking in log append
[ https://issues.apache.org/jira/browse/KAFKA-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Ivanov updated KAFKA-1933: Description: This patch adds finer locking when appending to log. It breaks global append lock into 2 sequential and 1 parallel phase. Basic idea is to allow every thread to reserve offsets in non overlapping ranges, then do compression in parallel and then commit write to log in the same order offsets where reserved. Results on a server with 16 cores CPU available: gzip: 564.0 sec - 45.2 sec (12.4x speedup) LZ4: 56.7 sec - 9.9 sec (5.7x speedup) Kafka was configured to run 16 IO threads, data was pushed using 32 netcat instances pushing in parallel batches of 200 msg 6.2 kb each (3264 MB in total) was: This patch adds finer locking when appending to log. It breaks global append lock into 2 sequential and 1 parallel phase. Basic idea is to allow every thread to reserve offsets in non overlapping ranges, then do compression in parallel and then commit write to log in the same order offsets where reserved. On my Core i3 M370 @2.4Ghz (2 cores + HT) it resulted in following performance boost: LZ4: 7.2 sec - 4.2 sec Gzip: 62.3 sec - 26.9 sec Kafka was configured to run 4 IO threads, data was pushed using 5 netcat instances pushing in parallel batches of 200 msg 6.2 kb each (510 MB in total, 82180 messages in total) Fine-grained locking in log append -- Key: KAFKA-1933 URL: https://issues.apache.org/jira/browse/KAFKA-1933 Project: Kafka Issue Type: Improvement Components: log Reporter: Maxim Ivanov Assignee: Jay Kreps Priority: Minor Fix For: 0.8.2 Attachments: KAFKA-1933.patch, KAFKA-1933_2015-02-09_12:27:06.patch This patch adds finer locking when appending to log. It breaks global append lock into 2 sequential and 1 parallel phase. Basic idea is to allow every thread to reserve offsets in non overlapping ranges, then do compression in parallel and then commit write to log in the same order offsets where reserved. Results on a server with 16 cores CPU available: gzip: 564.0 sec - 45.2 sec (12.4x speedup) LZ4: 56.7 sec - 9.9 sec (5.7x speedup) Kafka was configured to run 16 IO threads, data was pushed using 32 netcat instances pushing in parallel batches of 200 msg 6.2 kb each (3264 MB in total) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1933) Fine-grained locking in log append
[ https://issues.apache.org/jira/browse/KAFKA-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Ivanov updated KAFKA-1933: Description: This patch adds finer locking when appending to log. It breaks global append lock into 2 sequential and 1 parallel phase. Basic idea is to allow every thread to reserve offsets in non overlapping ranges, then do compression in parallel and then commit write to log in the same order offsets where reserved. On my Core i3 M370 @2.4Ghz (2 cores + HT) it resulted in following performance boost: LZ4: 7.2 sec - 4.2 sec Gzip: 62.3 sec - 26.9 sec Kafka was configured to run 4 IO threads, data was pushed using 5 netcat instances pushing in parallel batches of 200 msg 6.2 kb each (510 MB in total, 82180 messages in total) was: This patch adds finer locking when appending to log. It breaks global append lock into 2 sequential and 1 parallel phase. Basic idea is to allow every thread to reserve offsets in non overlapping ranges, then do compression in parallel and then commit write to log in the same order offsets where reserved. On my Core i3 M370 @2.4Ghz (2 cores + HT) it resulted in following performance boost: LZ4: 7.2 sec - 3.9 sec Gzip: 62.3 sec - 24.8 sec Kafka was configured to run 4 IO threads, data was pushed using 5 netcat instances pushing in parallel batches of 200 msg 6.2 kb each (510 MB in total, 82180 messages in total) Fine-grained locking in log append -- Key: KAFKA-1933 URL: https://issues.apache.org/jira/browse/KAFKA-1933 Project: Kafka Issue Type: Improvement Components: log Reporter: Maxim Ivanov Assignee: Jay Kreps Priority: Minor Fix For: 0.8.2 Attachments: KAFKA-1933.patch, KAFKA-1933_2015-02-09_12:15:06.patch This patch adds finer locking when appending to log. It breaks global append lock into 2 sequential and 1 parallel phase. Basic idea is to allow every thread to reserve offsets in non overlapping ranges, then do compression in parallel and then commit write to log in the same order offsets where reserved. On my Core i3 M370 @2.4Ghz (2 cores + HT) it resulted in following performance boost: LZ4: 7.2 sec - 4.2 sec Gzip: 62.3 sec - 26.9 sec Kafka was configured to run 4 IO threads, data was pushed using 5 netcat instances pushing in parallel batches of 200 msg 6.2 kb each (510 MB in total, 82180 messages in total) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1933) Fine-grained locking in log append
[ https://issues.apache.org/jira/browse/KAFKA-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Ivanov updated KAFKA-1933: Attachment: (was: KAFKA-1933_2015-02-09_12:15:06.patch) Fine-grained locking in log append -- Key: KAFKA-1933 URL: https://issues.apache.org/jira/browse/KAFKA-1933 Project: Kafka Issue Type: Improvement Components: log Reporter: Maxim Ivanov Assignee: Jay Kreps Priority: Minor Fix For: 0.8.2 Attachments: KAFKA-1933.patch This patch adds finer locking when appending to log. It breaks global append lock into 2 sequential and 1 parallel phase. Basic idea is to allow every thread to reserve offsets in non overlapping ranges, then do compression in parallel and then commit write to log in the same order offsets where reserved. On my Core i3 M370 @2.4Ghz (2 cores + HT) it resulted in following performance boost: LZ4: 7.2 sec - 4.2 sec Gzip: 62.3 sec - 26.9 sec Kafka was configured to run 4 IO threads, data was pushed using 5 netcat instances pushing in parallel batches of 200 msg 6.2 kb each (510 MB in total, 82180 messages in total) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1933) Fine-grained locking in log append
[ https://issues.apache.org/jira/browse/KAFKA-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Ivanov updated KAFKA-1933: Attachment: KAFKA-1933_2015-02-09_12:15:06.patch Fine-grained locking in log append -- Key: KAFKA-1933 URL: https://issues.apache.org/jira/browse/KAFKA-1933 Project: Kafka Issue Type: Improvement Components: log Reporter: Maxim Ivanov Assignee: Jay Kreps Priority: Minor Fix For: 0.8.2 Attachments: KAFKA-1933.patch, KAFKA-1933_2015-02-09_12:15:06.patch This patch adds finer locking when appending to log. It breaks global append lock into 2 sequential and 1 parallel phase. Basic idea is to allow every thread to reserve offsets in non overlapping ranges, then do compression in parallel and then commit write to log in the same order offsets where reserved. On my Core i3 M370 @2.4Ghz (2 cores + HT) it resulted in following performance boost: LZ4: 7.2 sec - 3.9 sec Gzip: 62.3 sec - 24.8 sec Kafka was configured to run 4 IO threads, data was pushed using 5 netcat instances pushing in parallel batches of 200 msg 6.2 kb each (510 MB in total, 82180 messages in total) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1933) Fine-grained locking in log append
[ https://issues.apache.org/jira/browse/KAFKA-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Ivanov updated KAFKA-1933: Attachment: KAFKA-1933_2015-02-09_12:27:06.patch Fine-grained locking in log append -- Key: KAFKA-1933 URL: https://issues.apache.org/jira/browse/KAFKA-1933 Project: Kafka Issue Type: Improvement Components: log Reporter: Maxim Ivanov Assignee: Jay Kreps Priority: Minor Fix For: 0.8.2 Attachments: KAFKA-1933.patch, KAFKA-1933_2015-02-09_12:27:06.patch This patch adds finer locking when appending to log. It breaks global append lock into 2 sequential and 1 parallel phase. Basic idea is to allow every thread to reserve offsets in non overlapping ranges, then do compression in parallel and then commit write to log in the same order offsets where reserved. On my Core i3 M370 @2.4Ghz (2 cores + HT) it resulted in following performance boost: LZ4: 7.2 sec - 4.2 sec Gzip: 62.3 sec - 26.9 sec Kafka was configured to run 4 IO threads, data was pushed using 5 netcat instances pushing in parallel batches of 200 msg 6.2 kb each (510 MB in total, 82180 messages in total) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1933) Fine-grained locking in log append
[ https://issues.apache.org/jira/browse/KAFKA-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Ivanov updated KAFKA-1933: Attachment: KAFKA-1933.patch Fine-grained locking in log append -- Key: KAFKA-1933 URL: https://issues.apache.org/jira/browse/KAFKA-1933 Project: Kafka Issue Type: Improvement Components: log Reporter: Maxim Ivanov Assignee: Jay Kreps Priority: Minor Fix For: 0.8.2 Attachments: KAFKA-1933.patch This patch adds finer locking when appending to log. It breaks global append lock into 2 sequential and 1 parallel phase. Basic idea is to allow every thread to reserve offsets in non overlapping ranges, then do compression in parallel and then commit write to log in the same order offsets where reserved. On my Core i3 M370 @2.4Ghz (2 cores + HT) it resulted in following performance boost: LZ4: 7.2 sec - 3.9 sec Gzip: 62.3 sec - 24.8 sec Kafka was configured to run 4 IO threads, data was pushed using 5 netcat instances pushing in parallel batches of 200 msg 6.2 kb each (510 MB in total, 82180 messages in total) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
