[jira] [Assigned] (KAFKA-7817) Multiple Consumer Group Management with Regex
[ https://issues.apache.org/jira/browse/KAFKA-7817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Dunayevsky reassigned KAFKA-7817: -- Assignee: Alex Dunayevsky > Multiple Consumer Group Management with Regex > - > > Key: KAFKA-7817 > URL: https://issues.apache.org/jira/browse/KAFKA-7817 > Project: Kafka > Issue Type: New Feature > Components: tools >Affects Versions: 2.1.0 >Reporter: Alex Dunayevsky >Assignee: Alex Dunayevsky >Priority: Minor > > //TODO: > New feature: Provide ConsumerGroupCommand with ability to query/manage > multiple consumer groups using a single regex pattern. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7817) Multiple Consumer Group Management with Regex
[ https://issues.apache.org/jira/browse/KAFKA-7817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Dunayevsky updated KAFKA-7817: --- Description: //TODO: New feature: Provide ConsumerGroupCommand with ability to query/manage multiple consumer groups using a single regex pattern. was: New feature: Multiple Consumer Group Management with regular expressions (kafka-consumer-groups.sh). //TODO: Provide ConsumerGroupCommand with ability to query/manage multiple consumer groups using a single regex pattern. > Multiple Consumer Group Management with Regex > - > > Key: KAFKA-7817 > URL: https://issues.apache.org/jira/browse/KAFKA-7817 > Project: Kafka > Issue Type: New Feature > Components: tools >Affects Versions: 2.1.0 >Reporter: Alex Dunayevsky >Priority: Minor > > //TODO: > New feature: Provide ConsumerGroupCommand with ability to query/manage > multiple consumer groups using a single regex pattern. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7817) Multiple Consumer Group Management with Regex
Alex Dunayevsky created KAFKA-7817: -- Summary: Multiple Consumer Group Management with Regex Key: KAFKA-7817 URL: https://issues.apache.org/jira/browse/KAFKA-7817 Project: Kafka Issue Type: New Feature Components: tools Affects Versions: 2.1.0 Reporter: Alex Dunayevsky New feature: Multiple Consumer Group Management with regular expressions (kafka-consumer-groups.sh). //TODO: Provide ConsumerGroupCommand with ability to query/manage multiple consumer groups using a single regex pattern. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-6724) ConsumerPerformance resets offsets on every startup
[ https://issues.apache.org/jira/browse/KAFKA-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Dunayevsky reassigned KAFKA-6724: -- Assignee: Alex Dunayevsky > ConsumerPerformance resets offsets on every startup > --- > > Key: KAFKA-6724 > URL: https://issues.apache.org/jira/browse/KAFKA-6724 > Project: Kafka > Issue Type: Bug > Components: core, tools >Affects Versions: 0.11.0.1 >Reporter: Alex Dunayevsky >Assignee: Alex Dunayevsky >Priority: Minor > Fix For: 2.0.0 > > > ConsumerPerformance used in kafka-consumer-perf-test.sh resets offsets for > it's group on every startup. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7471) Multiple Consumer Group Management (Describe, Reset, Delete)
[ https://issues.apache.org/jira/browse/KAFKA-7471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635805#comment-16635805 ] Alex Dunayevsky commented on KAFKA-7471: Pull request: https://github.com/apache/kafka/pull/5726 > Multiple Consumer Group Management (Describe, Reset, Delete) > > > Key: KAFKA-7471 > URL: https://issues.apache.org/jira/browse/KAFKA-7471 > Project: Kafka > Issue Type: New Feature > Components: tools >Affects Versions: 1.0.0, 2.0.0 >Reporter: Alex Dunayevsky >Assignee: Alex Dunayevsky >Priority: Major > Fix For: 2.0.1 > > > Functionality needed: > * Describe/Delete/Reset offsets on multiple consumer groups at a time > (including each group by repeating `--group` parameter) > * Describe/Delete/Reset offsets on ALL consumer groups at a time (add new > --groups-all option similar to --topics-all) > * Generate CSV for multiple consumer groups > What are the benifits? > * No need to start a new JVM to perform each query on every single consumer > group > * Abiltity to query groups by their status (for instance, `-v grepping` by > `Stable` to spot problematic/dead/empty groups) > * Ability to export offsets to reset for multiple consumer groups to a CSV > file (needs CSV generation export/import format rework) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7471) Multiple Consumer Group Management (Describe, Reset, Delete)
[ https://issues.apache.org/jira/browse/KAFKA-7471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Dunayevsky updated KAFKA-7471: --- Description: Functionality needed: * Describe/Delete/Reset offsets on multiple consumer groups at a time (including each group by repeating `--group` parameter) * Describe/Delete/Reset offsets on ALL consumer groups at a time (add new --groups-all option similar to --topics-all) * Generate CSV for multiple consumer groups What are the benifits? * No need to start a new JVM to perform each query on every single consumer group * Abiltity to query groups by their status (for instance, `-v grepping` by `Stable` to spot problematic/dead/empty groups) * Ability to export offsets to reset for multiple consumer groups to a CSV file (needs CSV generation export/import format rework) was: Functionality needed: * Describe/Delete/Reset offsets on multiple consumer groups at a time (including each group by repeating `--group` parameter) * Describe/Delete/Reset offsets on ALL consumer groups at a time (add new --groups-all option, similar to --topics-all) * Generate CSV for multiple consumer groups What are the benifits? * No need to start a new JVM to perform each query on every single consumer group * Abiltity to query groups by their status (for instance, `-v grepping` by `Stable` to spot problematic/dead/empty groups) * Ability to export offsets to reset for multiple consumer groups to a CSV file (needs CSV generation export/import format rework) > Multiple Consumer Group Management (Describe, Reset, Delete) > > > Key: KAFKA-7471 > URL: https://issues.apache.org/jira/browse/KAFKA-7471 > Project: Kafka > Issue Type: New Feature > Components: tools >Affects Versions: 1.0.0, 2.0.0 >Reporter: Alex Dunayevsky >Assignee: Alex Dunayevsky >Priority: Major > Fix For: 2.0.1 > > > Functionality needed: > * Describe/Delete/Reset offsets on multiple consumer groups at a time > (including each group by repeating `--group` parameter) > * Describe/Delete/Reset offsets on ALL consumer groups at a time (add new > --groups-all option similar to --topics-all) > * Generate CSV for multiple consumer groups > What are the benifits? > * No need to start a new JVM to perform each query on every single consumer > group > * Abiltity to query groups by their status (for instance, `-v grepping` by > `Stable` to spot problematic/dead/empty groups) > * Ability to export offsets to reset for multiple consumer groups to a CSV > file (needs CSV generation export/import format rework) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7471) Multiple Consumer Group Management (Describe, Reset, Delete)
[ https://issues.apache.org/jira/browse/KAFKA-7471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Dunayevsky updated KAFKA-7471: --- Description: Functionality needed: * Describe/Delete/Reset offsets on multiple consumer groups at a time (including each group by repeating `--group` parameter) * Describe/Delete/Reset offsets on ALL consumer groups at a time (add `-groups-all` option, similar to `-topics-all`) * Generate CSV for multiple consumer groups What are the benifits? * No need to start a new JVM to perform each query on every single consumer group * Abiltity to query groups by their status (for instance, `-v grepping` by `Stable` to spot problematic/dead/empty groups) * Ability to export offsets to reset for multiple consumer groups to a CSV file (needs CSV generation export/import format rework) was: Functionality needed: * Describe/Delete/Reset offsets on multiple consumer groups at a time (including each group by repeating `--group` parameter) * Describe/Delete/Reset offsets on ALL consumer groups at a time (add key `-groups-all`, similar to `-topics-all`) * Generate CSV for multiple consumer groups What are the benifits? * No need to start a new JVM to perform each query on every single consumer group * Abiltity to query groups by their status (for instance, `-v grepping` by `Stable` to spot problematic/dead/empty groups) * Ability to export offsets to reset for multiple consumer groups to a CSV file (needs CSV generation export/import format rework) > Multiple Consumer Group Management (Describe, Reset, Delete) > > > Key: KAFKA-7471 > URL: https://issues.apache.org/jira/browse/KAFKA-7471 > Project: Kafka > Issue Type: New Feature > Components: tools >Affects Versions: 1.0.0, 2.0.0 >Reporter: Alex Dunayevsky >Assignee: Alex Dunayevsky >Priority: Major > Fix For: 2.0.1 > > > Functionality needed: > * Describe/Delete/Reset offsets on multiple consumer groups at a time > (including each group by repeating `--group` parameter) > * Describe/Delete/Reset offsets on ALL consumer groups at a time (add > `-groups-all` option, similar to `-topics-all`) > * Generate CSV for multiple consumer groups > What are the benifits? > * No need to start a new JVM to perform each query on every single consumer > group > * Abiltity to query groups by their status (for instance, `-v grepping` by > `Stable` to spot problematic/dead/empty groups) > * Ability to export offsets to reset for multiple consumer groups to a CSV > file (needs CSV generation export/import format rework) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7471) Multiple Consumer Group Management (Describe, Reset, Delete)
[ https://issues.apache.org/jira/browse/KAFKA-7471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Dunayevsky updated KAFKA-7471: --- Description: Functionality needed: * Describe/Delete/Reset offsets on multiple consumer groups at a time (including each group by repeating `--group` parameter) * Describe/Delete/Reset offsets on ALL consumer groups at a time (add new --groups-all option, similar to --topics-all) * Generate CSV for multiple consumer groups What are the benifits? * No need to start a new JVM to perform each query on every single consumer group * Abiltity to query groups by their status (for instance, `-v grepping` by `Stable` to spot problematic/dead/empty groups) * Ability to export offsets to reset for multiple consumer groups to a CSV file (needs CSV generation export/import format rework) was: Functionality needed: * Describe/Delete/Reset offsets on multiple consumer groups at a time (including each group by repeating `--group` parameter) * Describe/Delete/Reset offsets on ALL consumer groups at a time (add `-groups-all` option, similar to `-topics-all`) * Generate CSV for multiple consumer groups What are the benifits? * No need to start a new JVM to perform each query on every single consumer group * Abiltity to query groups by their status (for instance, `-v grepping` by `Stable` to spot problematic/dead/empty groups) * Ability to export offsets to reset for multiple consumer groups to a CSV file (needs CSV generation export/import format rework) > Multiple Consumer Group Management (Describe, Reset, Delete) > > > Key: KAFKA-7471 > URL: https://issues.apache.org/jira/browse/KAFKA-7471 > Project: Kafka > Issue Type: New Feature > Components: tools >Affects Versions: 1.0.0, 2.0.0 >Reporter: Alex Dunayevsky >Assignee: Alex Dunayevsky >Priority: Major > Fix For: 2.0.1 > > > Functionality needed: > * Describe/Delete/Reset offsets on multiple consumer groups at a time > (including each group by repeating `--group` parameter) > * Describe/Delete/Reset offsets on ALL consumer groups at a time (add new > --groups-all option, similar to --topics-all) > * Generate CSV for multiple consumer groups > What are the benifits? > * No need to start a new JVM to perform each query on every single consumer > group > * Abiltity to query groups by their status (for instance, `-v grepping` by > `Stable` to spot problematic/dead/empty groups) > * Ability to export offsets to reset for multiple consumer groups to a CSV > file (needs CSV generation export/import format rework) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7471) Multiple Consumer Group Management (Describe, Reset, Delete)
[ https://issues.apache.org/jira/browse/KAFKA-7471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Dunayevsky updated KAFKA-7471: --- Description: Functionality needed: * Describe/Delete/Reset offsets on multiple consumer groups at a time (including each group by repeating `--group` parameter) * Describe/Delete/Reset offsets on ALL consumer groups at a time (add key `-groups-all`, similar to `-topics-all`) * Generate CSV for multiple consumer groups What are the benifits? * No need to start a new JVM to perform each query on every single consumer group * Abiltity to query groups by their status (for instance, `-v grepping` by `Stable` to spot problematic/dead/empty groups) * Ability to export offsets to reset for multiple consumer groups to a CSV file (needs CSV generation export/import format rework) was: Functionality needed: * Describe/Delete/Reset offsets on multiple consumer groups at a time (including each group by repeating `--group` parameter) * Describe/Delete/Reset offsets on ALL consumer groups at a time (add key ```--groups-all```, similar to ```-topics-all```) * Generate CSV for multiple consumer groups What are the benifits? * No need to start a new JVM to perform each query on every single consumer group * Abiltity to query groups by their status (for instance, `-v grepping` by `Stable` to spot problematic/dead/empty groups) * Ability to export offsets to reset for multiple consumer groups to a CSV file (needs CSV generation export/import format rework) > Multiple Consumer Group Management (Describe, Reset, Delete) > > > Key: KAFKA-7471 > URL: https://issues.apache.org/jira/browse/KAFKA-7471 > Project: Kafka > Issue Type: New Feature > Components: tools >Affects Versions: 1.0.0, 2.0.0 >Reporter: Alex Dunayevsky >Assignee: Alex Dunayevsky >Priority: Major > Fix For: 2.0.1 > > > Functionality needed: > * Describe/Delete/Reset offsets on multiple consumer groups at a time > (including each group by repeating `--group` parameter) > * Describe/Delete/Reset offsets on ALL consumer groups at a time (add key > `-groups-all`, similar to `-topics-all`) > * Generate CSV for multiple consumer groups > What are the benifits? > * No need to start a new JVM to perform each query on every single consumer > group > * Abiltity to query groups by their status (for instance, `-v grepping` by > `Stable` to spot problematic/dead/empty groups) > * Ability to export offsets to reset for multiple consumer groups to a CSV > file (needs CSV generation export/import format rework) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7471) Multiple Consumer Group Management (Describe, Reset, Delete)
[ https://issues.apache.org/jira/browse/KAFKA-7471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Dunayevsky updated KAFKA-7471: --- Description: Functionality needed: * Describe/Delete/Reset offsets on multiple consumer groups at a time (including each group by repeating `--group` parameter) * Describe/Delete/Reset offsets on ALL consumer groups at a time (add key ```--groups-all```, similar to ```-topics-all```) * Generate CSV for multiple consumer groups What are the benifits? * No need to start a new JVM to perform each query on every single consumer group * Abiltity to query groups by their status (for instance, `-v grepping` by `Stable` to spot problematic/dead/empty groups) * Ability to export offsets to reset for multiple consumer groups to a CSV file (needs CSV generation export/import format rework) was: Functionality needed: * Describe/Delete/Reset offsets on multiple consumer groups at a time (including each group by repeating `--group` parameter) * Describe/Delete/Reset offsets on ALL consumer groups at a time (add key `--groups-all`, similar to `--topics-all`) * Generate CSV for multiple consumer groups What are the benifits? * No need to start a new JVM to perform each query on every single consumer group * Abiltity to query groups by their status (for instance, `-v grepping` by `Stable` to spot problematic/dead/empty groups) * Ability to export offsets to reset for multiple consumer groups to a CSV file (needs CSV generation export/import format rework) > Multiple Consumer Group Management (Describe, Reset, Delete) > > > Key: KAFKA-7471 > URL: https://issues.apache.org/jira/browse/KAFKA-7471 > Project: Kafka > Issue Type: New Feature > Components: tools >Affects Versions: 1.0.0, 2.0.0 >Reporter: Alex Dunayevsky >Assignee: Alex Dunayevsky >Priority: Major > Fix For: 2.0.1 > > > Functionality needed: > * Describe/Delete/Reset offsets on multiple consumer groups at a time > (including each group by repeating `--group` parameter) > * Describe/Delete/Reset offsets on ALL consumer groups at a time (add key > ```--groups-all```, similar to ```-topics-all```) > * Generate CSV for multiple consumer groups > What are the benifits? > * No need to start a new JVM to perform each query on every single consumer > group > * Abiltity to query groups by their status (for instance, `-v grepping` by > `Stable` to spot problematic/dead/empty groups) > * Ability to export offsets to reset for multiple consumer groups to a CSV > file (needs CSV generation export/import format rework) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7471) Multiple Consumer Group Management (Describe, Reset, Delete)
Alex Dunayevsky created KAFKA-7471: -- Summary: Multiple Consumer Group Management (Describe, Reset, Delete) Key: KAFKA-7471 URL: https://issues.apache.org/jira/browse/KAFKA-7471 Project: Kafka Issue Type: New Feature Components: tools Affects Versions: 2.0.0, 1.0.0 Reporter: Alex Dunayevsky Assignee: Alex Dunayevsky Fix For: 2.0.1 Functionality needed: * Describe/Delete/Reset offsets on multiple consumer groups at a time (including each group by repeating `--group` parameter) * Describe/Delete/Reset offsets on ALL consumer groups at a time (add key `--groups-all`, similar to `--topics-all`) * Generate CSV for multiple consumer groups What are the benifits? * No need to start a new JVM to perform each query on every single consumer group * Abiltity to query groups by their status (for instance, `-v grepping` by `Stable` to spot problematic/dead/empty groups) * Ability to export offsets to reset for multiple consumer groups to a CSV file (needs CSV generation export/import format rework) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6743) ConsumerPerformance fails to consume all messages on topics with large number of partitions
[ https://issues.apache.org/jira/browse/KAFKA-6743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424145#comment-16424145 ] Alex Dunayevsky commented on KAFKA-6743: https://github.com/apache/kafka/pull/4818 > ConsumerPerformance fails to consume all messages on topics with large number > of partitions > --- > > Key: KAFKA-6743 > URL: https://issues.apache.org/jira/browse/KAFKA-6743 > Project: Kafka > Issue Type: Bug > Components: core, tools >Affects Versions: 0.11.0.2 >Reporter: Alex Dunayevsky >Priority: Minor > > ConsumerPerformance fails to consume all messages on topics with large number > of partitions due to a relatively short default polling loop timeout (1000 > ms) that is not reachable and modifiable by the end user. > Demo: Create a topic of 10 000 partitions, send a 50 000 000 of 100 byte > records using kafka-producer-perf-test and consume them using > kafka-consumer-perf-test (ConsumerPerformance). You will likely notice that > the number of records returned by the kafka-consumer-perf-test is many times > less than expected 50 000 000. This happens due to specific > ConsumerPerformance implementation. As the result, in some rough cases it may > take a long enough time to process/iterate through the records polled in > batches, thus, the time may exceed the default hardcoded polling loop timeout > and this is probably not what we want from this utility. > We have two options: > 1) Increasing polling loop timeout in ConsumerPerformance implementation. It > defaults to 1000 ms and is hardcoded, thus cannot be changed but we could > export it as an OPTIONAL kafka-consumer-perf-test parameter to enable it on a > script level configuration and available to the end user. > 2) Decreasing max.poll.records on a Consumer config level. This is not a fine > option though since we do not want to touch the default settings. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6743) ConsumerPerformance fails to consume all messages on topics with large number of partitions
Alex Dunayevsky created KAFKA-6743: -- Summary: ConsumerPerformance fails to consume all messages on topics with large number of partitions Key: KAFKA-6743 URL: https://issues.apache.org/jira/browse/KAFKA-6743 Project: Kafka Issue Type: Bug Components: core, tools Affects Versions: 0.11.0.2 Reporter: Alex Dunayevsky ConsumerPerformance fails to consume all messages on topics with large number of partitions due to a relatively short default polling loop timeout (1000 ms) that is not reachable and modifiable by the end user. Demo: Create a topic of 10 000 partitions, send a 50 000 000 of 100 byte records using kafka-producer-perf-test and consume them using kafka-consumer-perf-test (ConsumerPerformance). You will likely notice that the number of records returned by the kafka-consumer-perf-test is many times less than expected 50 000 000. This happens due to specific ConsumerPerformance implementation. As the result, in some rough cases it may take a long enough time to process/iterate through the records polled in batches, thus, the time may exceed the default hardcoded polling loop timeout and this is probably not what we want from this utility. We have two options: 1) Increasing polling loop timeout in ConsumerPerformance implementation. It defaults to 1000 ms and is hardcoded, thus cannot be changed but we could export it as an OPTIONAL kafka-consumer-perf-test parameter to enable it on a script level configuration and available to the end user. 2) Decreasing max.poll.records on a Consumer config level. This is not a fine option though since we do not want to touch the default settings. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6724) ConsumerPerformance resets offsets on every startup
[ https://issues.apache.org/jira/browse/KAFKA-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Dunayevsky updated KAFKA-6724: --- Reviewer: Alex Dunayevsky > ConsumerPerformance resets offsets on every startup > --- > > Key: KAFKA-6724 > URL: https://issues.apache.org/jira/browse/KAFKA-6724 > Project: Kafka > Issue Type: Bug > Components: core, tools >Affects Versions: 0.11.0.1 >Reporter: Alex Dunayevsky >Priority: Minor > > ConsumerPerformance used in kafka-consumer-perf-test.sh resets offsets for > it's group on every startup. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6724) ConsumerPerformance resets offsets on every startup
[ https://issues.apache.org/jira/browse/KAFKA-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Dunayevsky updated KAFKA-6724: --- Reviewer: (was: Alex Dunayevsky) > ConsumerPerformance resets offsets on every startup > --- > > Key: KAFKA-6724 > URL: https://issues.apache.org/jira/browse/KAFKA-6724 > Project: Kafka > Issue Type: Bug > Components: core, tools >Affects Versions: 0.11.0.1 >Reporter: Alex Dunayevsky >Priority: Minor > > ConsumerPerformance used in kafka-consumer-perf-test.sh resets offsets for > it's group on every startup. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6724) ConsumerPerformance resets offsets on every startup
Alex Dunayevsky created KAFKA-6724: -- Summary: ConsumerPerformance resets offsets on every startup Key: KAFKA-6724 URL: https://issues.apache.org/jira/browse/KAFKA-6724 Project: Kafka Issue Type: Bug Components: core, tools Affects Versions: 0.11.0.1 Reporter: Alex Dunayevsky ConsumerPerformance used in kafka-consumer-perf-test.sh resets offsets for it's group on every startup. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6597) Issues with Zookeeper and Kafka startup in Windows environment
[ https://issues.apache.org/jira/browse/KAFKA-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Dunayevsky updated KAFKA-6597: --- Environment: MS Windows 7 Corporate Edition > Issues with Zookeeper and Kafka startup in Windows environment > -- > > Key: KAFKA-6597 > URL: https://issues.apache.org/jira/browse/KAFKA-6597 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.9.0.1, 0.10.0.1, 0.11.0.1 > Environment: MS Windows 7 Corporate Edition >Reporter: Alex Dunayevsky >Priority: Trivial > > Inability to start Zookeeper and Kafka services using standard Kafka .bat > utilities for Windows environment > *Problem 1:* CLASSPATH string not being formed correctly in > bin\windows\kafka-run-class.bat. > |bin\windows\zookeeper-server-start.bat config\zookeeper.properties > ... class not found ...| > *Possible working solution*: > Assign CLASSPATH correctly in *bin\windows\kafka-run-class.bat:* > |set CLASSPATH=%~dp0..\..\libs*| > > *Problem 2:* *call :concat* may crash *bin\windows\kafka-run-class.bat* : > |rem Classpath addition for release > call :concat %BASE_DIR%\libs*| > *Possible working solution:* > Comment or delete those lines of code. > |rem Classpath addition for release > rem call :concat %BASE_DIR%\libs*| > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6597) Issues with Zookeeper and Kafka startup in Windows environment
[ https://issues.apache.org/jira/browse/KAFKA-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Dunayevsky updated KAFKA-6597: --- Description: Inability to start Zookeeper and Kafka services using standard Kafka .bat utilities for Windows environment *Problem 1:* CLASSPATH string not being formed correctly in bin\windows\kafka-run-class.bat. |bin\windows\zookeeper-server-start.bat config\zookeeper.properties ... class not found ...| *Possible working solution*: Assign CLASSPATH correctly in *bin\windows\kafka-run-class.bat:* |set CLASSPATH=%~dp0..\..\libs*| *Problem 2:* *call :concat* may crash *bin\windows\kafka-run-class.bat* : |rem Classpath addition for release call :concat %BASE_DIR%\libs*| *Possible working solution:* Comment or delete those lines of code. |rem Classpath addition for release rem call :concat %BASE_DIR%\libs*| was: Inability to start Zookeeper and Kafka services using standard Kafka .bat utilities for Windows environment *Problem 1:* CLASSPATH string not being formed correctly in bin\windows\kafka-run-class.bat. |bin\windows\zookeeper-server-start.bat config\zookeeper.properties ... class not found ...| *Possible working solution*: Assign CLASSPATH correctly in *bin\windows\kafka-run-class.bat:* |set CLASSPATH=%~dp0..\..\libs*| *Problem 2:* In Kafka distro the *call :concat* may crash *bin\windows\kafka-run-class.bat* : |rem Classpath addition for release call :concat %BASE_DIR%\libs*| *Possible working solution:* Comment or delete those lines of code. |rem Classpath addition for release rem call :concat %BASE_DIR%\libs*| > Issues with Zookeeper and Kafka startup in Windows environment > -- > > Key: KAFKA-6597 > URL: https://issues.apache.org/jira/browse/KAFKA-6597 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.9.0.1, 0.10.0.1, 0.11.0.1 >Reporter: Alex Dunayevsky >Priority: Trivial > > Inability to start Zookeeper and Kafka services using standard Kafka .bat > utilities for Windows environment > *Problem 1:* CLASSPATH string not being formed correctly in > bin\windows\kafka-run-class.bat. > |bin\windows\zookeeper-server-start.bat config\zookeeper.properties > ... class not found ...| > *Possible working solution*: > Assign CLASSPATH correctly in *bin\windows\kafka-run-class.bat:* > |set CLASSPATH=%~dp0..\..\libs*| > > *Problem 2:* *call :concat* may crash *bin\windows\kafka-run-class.bat* : > |rem Classpath addition for release > call :concat %BASE_DIR%\libs*| > *Possible working solution:* > Comment or delete those lines of code. > |rem Classpath addition for release > rem call :concat %BASE_DIR%\libs*| > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6597) Issues with Zookeeper and Kafka startup in Windows environment
[ https://issues.apache.org/jira/browse/KAFKA-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Dunayevsky updated KAFKA-6597: --- Description: Inability to start Zookeeper and Kafka services using standard Kafka .bat utilities for Windows environment *Problem 1:* CLASSPATH string not being formed correctly in bin\windows\kafka-run-class.bat. |bin\windows\zookeeper-server-start.bat config\zookeeper.properties ... class not found ...| *Possible working solution*: Assign CLASSPATH correctly in *bin\windows\kafka-run-class.bat:* |set CLASSPATH=%~dp0..\..\libs*| *Problem 2:* In Kafka distro the *call :concat* may crash *bin\windows\kafka-run-class.bat* : |rem Classpath addition for release call :concat %BASE_DIR%\libs*| *Possible working solution:* Comment or delete those lines of code. |rem Classpath addition for release rem call :concat %BASE_DIR%\libs*| was: Inability to start Zookeeper and Kafka services using standard Kafka .bat utilities for Windows environment *Problem 1:* CLASSPATH string not being formed correctly in bin\windows\kafka-run-class.bat. |bin\windows\zookeeper-server-start.bat config\zookeeper.properties *** ... class not found ...| *Possible working solution*: Assign CLASSPATH correctly in *bin\windows\kafka-run-class.bat:* |set CLASSPATH=%~dp0..\..\libs\*| *Problem 2:* In Kafka distro the *call :concat* may crash *bin\windows\kafka-run-class.bat* : |rem Classpath addition for release call :concat %BASE_DIR%\libs\*| *Possible working solution:* Comment or delete those lines of code. |rem Classpath addition for release rem call :concat %BASE_DIR%\libs\*| > Issues with Zookeeper and Kafka startup in Windows environment > -- > > Key: KAFKA-6597 > URL: https://issues.apache.org/jira/browse/KAFKA-6597 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.9.0.1, 0.10.0.1, 0.11.0.1 >Reporter: Alex Dunayevsky >Priority: Trivial > > Inability to start Zookeeper and Kafka services using standard Kafka .bat > utilities for Windows environment > *Problem 1:* CLASSPATH string not being formed correctly in > bin\windows\kafka-run-class.bat. > |bin\windows\zookeeper-server-start.bat config\zookeeper.properties > ... class not found ...| > *Possible working solution*: > Assign CLASSPATH correctly in *bin\windows\kafka-run-class.bat:* > |set CLASSPATH=%~dp0..\..\libs*| > > *Problem 2:* In Kafka distro the *call :concat* may crash > *bin\windows\kafka-run-class.bat* : > |rem Classpath addition for release > call :concat %BASE_DIR%\libs*| > *Possible working solution:* > Comment or delete those lines of code. > |rem Classpath addition for release > rem call :concat %BASE_DIR%\libs*| > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6597) Issues with Zookeeper and Kafka startup in Windows environment
Alex Dunayevsky created KAFKA-6597: -- Summary: Issues with Zookeeper and Kafka startup in Windows environment Key: KAFKA-6597 URL: https://issues.apache.org/jira/browse/KAFKA-6597 Project: Kafka Issue Type: Bug Affects Versions: 0.11.0.1, 0.10.0.1, 0.9.0.1 Reporter: Alex Dunayevsky Inability to start Zookeeper and Kafka services using standard Kafka .bat utilities for Windows environment *Problem 1:* CLASSPATH string not being formed correctly in bin\windows\kafka-run-class.bat. |bin\windows\zookeeper-server-start.bat config\zookeeper.properties *** ... class not found ...| *Possible working solution*: Assign CLASSPATH correctly in *bin\windows\kafka-run-class.bat:* |set CLASSPATH=%~dp0..\..\libs\*| *Problem 2:* In Kafka distro the *call :concat* may crash *bin\windows\kafka-run-class.bat* : |rem Classpath addition for release call :concat %BASE_DIR%\libs\*| *Possible working solution:* Comment or delete those lines of code. |rem Classpath addition for release rem call :concat %BASE_DIR%\libs\*| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6343:
---
Affects Version/s: 0.10.2.0
0.10.2.1
0.11.0.1
1.0.0
0.11.0.2
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1, 0.10.2.0, 0.10.2.1, 0.11.0.1, 1.0.0, 0.11.0.2
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G, 512G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
> ... 28 more
> {code}
> Broker restart results the same OOM issues. All brokers will not be able to
> start again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16287435#comment-16287435
]
Alex Dunayevsky edited comment on KAFKA-6343 at 12/12/17 10:55 AM:
---
Ismael, I mean are there any initial steps or rules to follow before submitting
a pull request?
As it says in /contributing section: "If you are interested in becoming a
committer, let one of the existing committers know and they can help guide you
through the process".
was (Author: alex.dunayevsky):
Ismael, I mean are there any initial steps or rules to follow before submitting
a pull request?
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G, 512G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
> ... 28 more
> {code}
> Broker restart results the same OOM issues. All brokers will not be able to
> start again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16287435#comment-16287435
]
Alex Dunayevsky commented on KAFKA-6343:
Ismael, I mean are there any initial steps or rules to follow before submitting
a pull request?
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G, 512G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
> ... 28 more
> {code}
> Broker restart results the same OOM issues. All brokers will not be able to
> start again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16287362#comment-16287362
]
Alex Dunayevsky commented on KAFKA-6343:
Ismael Juma, good idea, what should I do?
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G, 512G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
> ... 28 more
> {code}
> Broker restart results the same OOM issues. All brokers will not be able to
> start again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6343:
---
Description:
*Reproducing*: Create 5k topics *from the code* quickly, without any delays.
Wait until brokers will finish loading them. This will actually never happen,
since all brokers will go down one by one after approx 10-15 minutes or more,
depending on the hardware.
*Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G, 512G
*Topology*: 3 brokers, 3 zk.
*Code for 5k topic creation:*
{code:java}
package kafka
import kafka.admin.AdminUtils
import kafka.utils.{Logging, ZkUtils}
object TestCreateTopics extends App with Logging {
val zkConnect = "grid978:2185"
var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
for (topic <- 1 to 5000) {
AdminUtils.createTopic(
topic = s"${topic.toString}",
partitions= 10,
replicationFactor = 2,
zkUtils = zkUtils
)
logger.info(s"Created topic ${topic.toString}")
}
}
{code}
*Cause of death:*
{code:java}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
at kafka.cluster.Partition.makeLeader(Partition.scala:168)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
at
kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
... 28 more
{code}
Broker restart results the same OOM issues. All brokers will not be able to
start again.
was:
*Reproducing*: Create 5k topics *from the code* quickly, without any delays.
Wait until brokers will finish loading them. This will actually never happen,
since all brokers will go down one by one after approx 10-15 minutes or more,
depending on the hardware.
*Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
*Topology*: 3 brokers, 3 zk.
*Code for 5k topic creation:*
{code:java}
package kafka
import kafka.admin.AdminUtils
import kafka.utils.{Logging, ZkUtils}
object TestCreateTopics extends App with Logging {
val zkConnect = "grid978:2185"
var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
for (topic <- 1 to 5000) {
AdminUtils.createTopic(
topic = s"${topic.toString}",
partitions= 10,
replicationFactor = 2,
zkUtils = zkUtils
)
logger.info(s"Created topic ${topic.toString}")
}
}
{code}
*Cause of death:*
{code:java}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at
[jira] [Comment Edited] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285968#comment-16285968
]
Alex Dunayevsky edited comment on KAFKA-6343 at 12/11/17 4:16 PM:
--
Ismael Juma, once again, thank you! This time it looks like the core problem.
Reproducing:
{code:java}
// Max number of memory map operations is:
$ /sbin/sysctl vm.max_map_count
vm.max_map_count = 65530
// Tracking vm map size:
$ cat /proc//maps | wc -l
898 <--- grows from this value
...
65532 <--- up to this value (it's even a bit larger than
m.max_map_count=65530). This is the point where broker fails... So you are
right!
// Then all we have to do is to increase vm.max_map size to a larger value
(ex., by 65536 * 4):
$ /sbin/sysctl -w vm.max_map_count=262144
{code}
Ismael, awesome job!
was (Author: alex.dunayevsky):
Ismael Juma, once again, thank you! This time it looks like the core problem.
Reproducing:
{code:java}
// Max number memory map operations is:
$ /sbin/sysctl vm.max_map_count
vm.max_map_count = 65530
// Tracking vm map size:
$ cat /proc//maps | wc -l
898 <--- grows from this value
...
65532 <--- up to this value (it's even a bit larger than
m.max_map_count=65530). This is the point where broker fails... So you are
right!
// Then all we have to do is to increase vm.max_map size to a larger value
(ex., by 65536 * 4):
$ /sbin/sysctl -w vm.max_map_count=262144
{code}
Ismael, awesome job!
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
[jira] [Comment Edited] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285968#comment-16285968
]
Alex Dunayevsky edited comment on KAFKA-6343 at 12/11/17 2:31 PM:
--
Ismael Juma, once again, thank you! This time it looks like the core problem.
Reproducing:
{code:java}
// Max number memory map operations is:
$ /sbin/sysctl vm.max_map_count
vm.max_map_count = 65530
// Tracking vm map size:
$ cat /proc//maps | wc -l
898 <--- grows from this value
...
65532 <--- up to this value (it's even a bit larger than
m.max_map_count=65530). This is the point where broker fails... So you are
right!
// Then all we have to do is to increase vm.max_map size to a larger value
(ex., by 65536 * 4):
$ /sbin/sysctl -w vm.max_map_count=262144
{code}
Ismael, awesome job!
was (Author: alex.dunayevsky):
Ismael Juma, once again, thank you! This time it looks like the core problem.
Reproducing:
{code:java}
// Max number memory map operations is:
$ /sbin/sysctl vm.max_map_count
vm.max_map_count = 65530
// Tracking vm map size:
$ cat /proc//maps | wc -l
898 <--- grows from this value
...
65532 <--- up to this value (it's even a bit larger than m.max_map=65530).
This is the point where broker fails... So you are right!
// Then all we have to do is to increase vm.max_map size to a larger value
(ex., by 65536 * 4):
$ /sbin/sysctl -w vm.max_map_count=262144
{code}
Ismael, awesome job!
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
>
[jira] [Comment Edited] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285968#comment-16285968
]
Alex Dunayevsky edited comment on KAFKA-6343 at 12/11/17 2:28 PM:
--
Ismael Juma, once again, thank you! This time it looks like the core problem.
Reproducing:
{code:java}
// Max number memory map operations is:
$ /sbin/sysctl vm.max_map_count
vm.max_map_count = 65530
// Tracking vm map size:
$ cat /proc//maps | wc -l
898 <--- grows from this value
...
65532 <--- up to this value (it's even a bit larger than m.max_map=65530).
This is the point where broker fails... So you are right!
// Then all we have to do is to increase vm.max_map size to a larger value
(ex., by 65536 * 4):
$ /sbin/sysctl -w vm.max_map_count=262144
{code}
Ismael, awesome job!
was (Author: alex.dunayevsky):
Ismael Juma, once again, thank you! This time it looks like the core problem.
Reproducing:
{code:java}
// Max number memory map operations is:
$ /sbin/sysctl vm.max_map_count
vm.max_map_count = 65530
// Tracking vm map size:
$ cat /proc//maps | wc -l
898 <--- grows from this value
...
65532 <--- up to this value (it's even a bit larger than m.max_map=65530).
This is the point where broker fails... So you are right!
// Then all we have to do is to increase vm.max_map size to a larger value
(ex., by 65536 * 4):
$ /sbin/sysctl -w vm.max_map=262144
{code}
Ismael, awesome job!
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at
[jira] [Comment Edited] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285968#comment-16285968
]
Alex Dunayevsky edited comment on KAFKA-6343 at 12/11/17 2:23 PM:
--
Ismael Juma, once again, thank you! This time it looks like the core problem.
Reproducing:
{code:java}
// Max number memory map operations is:
$ /sbin/sysctl vm.max_map_count
vm.max_map_count = 65530
// Tracking vm map size:
$ cat /proc//maps | wc -l
898 <--- grows from this value
...
65532 <--- up to this value (it's even a bit larger than m.max_map=65530).
This is the point where broker fails... So you are right!
// Then all we have to do is to increase vm.max_map size to a larger value
(ex., by 65536 * 4):
$ /sbin/sysctl -w vm.max_map=262144
{code}
Ismael, awesome job!
was (Author: alex.dunayevsky):
Ismael Juma, once again, thank you! This time it looks like the core problem.
Reproducing:
{code:java}
// Max number memory map operations is:
$ /sbin/sysctl vm.max_map_count
vm.max_map_count = 65530
// Tracking vm map size:
$ cat /proc//maps | wc -l
898 <--- grows from this value
...
65532 <--- up to this value (it's even a bit larger than m.max_map=65530). This
is the point where broker fails... So you are right!
// Then all we have to do is to increase vm.max_map size to a larger value
(ex., by 65536 * 4):
$ /sbin/sysctl -w vm.max_map=262144
{code}
Ismael, awesome job!
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at
[jira] [Comment Edited] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285968#comment-16285968
]
Alex Dunayevsky edited comment on KAFKA-6343 at 12/11/17 2:23 PM:
--
Ismael Juma, once again, thank you! This time it looks like the core problem.
Reproducing:
{code:java}
// Max number memory map operations is:
$ /sbin/sysctl vm.max_map_count
vm.max_map_count = 65530
// Tracking vm map size:
$ cat /proc//maps | wc -l
898 <--- grows from this value
...
65532 <--- up to this value (it's even a bit larger than m.max_map=65530). This
is the point where broker fails... So you are right!
// Then all we have to do is to increase vm.max_map size to a larger value
(ex., by 65536 * 4):
$ /sbin/sysctl -w vm.max_map=262144
{code}
Ismael, awesome job!
was (Author: alex.dunayevsky):
Ismael Juma, once again, thank you! This time it looks like the core problem.
*Reproducing: *
{code:java}
// vm.max_map_count is:
$ /sbin/sysctl vm.max_map_count
vm.max_map_count = 65530
// Tracking vm map size:
$ cat /proc//maps | wc -l
898 <--- grows from this value
...
65532 <--- up to this value (it's even a bit larger than m.max_map=65530). This
is the point where broker fails... So you are right!
Then all we have to do is to increase vm.max_map size to a larger value (ex.,
by 65536 * 4):
$ /sbin/sysctl -w vm.max_map=262144
{code}
Ismael, awesome job!
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at
[jira] [Comment Edited] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285968#comment-16285968
]
Alex Dunayevsky edited comment on KAFKA-6343 at 12/11/17 2:23 PM:
--
Ismael Juma, once again, thank you! This time it looks like the core problem.
Reproducing:
{code:java}
// Max number memory map operations is:
$ /sbin/sysctl vm.max_map_count
vm.max_map_count = 65530
// Tracking vm map size:
$ cat /proc//maps | wc -l
898 <--- grows from this value
...
65532 <--- up to this value (it's even a bit larger than m.max_map=65530). This
is the point where broker fails... So you are right!
// Then all we have to do is to increase vm.max_map size to a larger value
(ex., by 65536 * 4):
$ /sbin/sysctl -w vm.max_map=262144
{code}
Ismael, awesome job!
was (Author: alex.dunayevsky):
Ismael Juma, once again, thank you! This time it looks like the core problem.
Reproducing:
{code:java}
// Max number memory map operations is:
$ /sbin/sysctl vm.max_map_count
vm.max_map_count = 65530
// Tracking vm map size:
$ cat /proc//maps | wc -l
898 <--- grows from this value
...
65532 <--- up to this value (it's even a bit larger than m.max_map=65530). This
is the point where broker fails... So you are right!
// Then all we have to do is to increase vm.max_map size to a larger value
(ex., by 65536 * 4):
$ /sbin/sysctl -w vm.max_map=262144
{code}
Ismael, awesome job!
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at
[jira] [Comment Edited] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285968#comment-16285968
]
Alex Dunayevsky edited comment on KAFKA-6343 at 12/11/17 2:21 PM:
--
Ismael Juma, once again, thank you! This time it looks like the core problem.
*Reproducing: *
{code:java}
// vm.max_map_count is:
$ /sbin/sysctl vm.max_map_count
vm.max_map_count = 65530
// Tracking vm map size:
$ cat /proc//maps | wc -l
898 <--- grows from this value
...
65532 <--- up to this value (it's even a bit larger than m.max_map=65530). This
is the point where broker fails... So you are right!
Then all we have to do is to increase vm.max_map size to a larger value (ex.,
by 65536 * 4):
$ /sbin/sysctl -w vm.max_map=262144
{code}
Ismael, awesome job!
was (Author: alex.dunayevsky):
Ismael Juma, once again, thank you! This time it looks like the core problem.
*Reproducing: *
{code:java}
$ /sbin/sysctl vm.max_map_count
vm.max_map_count = 65530
$ cat /proc//maps | wc -l
898 <--- grows from this value
...
65532 <--- up to this value (it's even a bit larger than m.max_map=65530). This
is the point where broker fails... So you are right!
Then all we have to do is to increase vm.max_map size to a larger value (ex.,
by 65536 * 4):
$ /sbin/sysctl -w vm.max_map=262144
{code}
Ismael, awesome job!
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at
[jira] [Commented] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285968#comment-16285968
]
Alex Dunayevsky commented on KAFKA-6343:
Ismael Juma, once again, thank you! This time it looks like the core problem.
*Reproducing: *
{code:java}
$ /sbin/sysctl vm.max_map_count
vm.max_map_count = 65530
$ cat /proc//maps | wc -l
898 <--- grows from this value
...
65532 <--- up to this value (it's even a bit larger than m.max_map=65530). This
is the point where broker fails... So you are right!
Then all we have to do is to increase vm.max_map size to a larger value (ex.,
by 65536 * 4):
$ /sbin/sysctl -w vm.max_map=262144
{code}
Ismael, awesome job!
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
> ... 28 more
> {code}
> Broker restart results the same OOM issues. All brokers will not be able to
> start again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285862#comment-16285862
]
Alex Dunayevsky edited comment on KAFKA-6343 at 12/11/17 1:02 PM:
--
Manikumar, sure
{code:java}
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 3093521
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 256000
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 256000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
$ ps -ef | grep Kafka
pprbusr 30147 1 99 15:44 pts/400:02:31 java -Xmx512G -Xms512G -server
-XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35
-XX:+DisableExplicitGC -Djava.awt.headless=true -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dkafka.logs.dir=/home/pprbusr/kafka/bin/../logs
-Dlog4j.configuration=file:/home/pprbusr/kafka/config/log4j.properties
-Dcom.sun.management.config.file=/home/pprbusr/kafka/config/jmx/kafka_jmx.properties
kafka.Kafka config/server.properties
{code}
was (Author: alex.dunayevsky):
Manikumar, sure
{code:bash}
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 3093521
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 256000
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 256000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
$ ps -ef | grep Kafka
pprbusr 30147 1 99 15:44 pts/400:02:31 java -Xmx512G -Xms512G -server
-XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35
-XX:+DisableExplicitGC -Djava.awt.headless=true -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dkafka.logs.dir=/home/pprbusr/kafka/bin/../logs
-Dlog4j.configuration=file:/home/pprbusr/kafka/config/log4j.properties
-Dcom.sun.management.config.file=/home/pprbusr/kafka/config/jmx/kafka_jmx.properties
kafka.Kafka config/server.properties
{code}
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
>
[jira] [Commented] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285862#comment-16285862
]
Alex Dunayevsky commented on KAFKA-6343:
Manikumar, sure
{code:bash}
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 3093521
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 256000
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 256000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
$ ps -ef | grep Kafka
pprbusr 30147 1 99 15:44 pts/400:02:31 java -Xmx512G -Xms512G -server
-XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35
-XX:+DisableExplicitGC -Djava.awt.headless=true -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dkafka.logs.dir=/home/pprbusr/kafka/bin/../logs
-Dlog4j.configuration=file:/home/pprbusr/kafka/config/log4j.properties
-Dcom.sun.management.config.file=/home/pprbusr/kafka/config/jmx/kafka_jmx.properties
kafka.Kafka config/server.properties
{code}
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at
[jira] [Updated] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6343:
---
Description:
*Reproducing*: Create 5k topics *from the code* quickly, without any delays.
Wait until brokers will finish loading them. This will actually never happen,
since all brokers will go down one by one after approx 10-15 minutes or more,
depending on the hardware.
*Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
*Topology*: 3 brokers, 3 zk.
*Code for 5k topic creation:*
{code:java}
package kafka
import kafka.admin.AdminUtils
import kafka.utils.{Logging, ZkUtils}
object TestCreateTopics extends App with Logging {
val zkConnect = "grid978:2185"
var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
for (topic <- 1 to 5000) {
AdminUtils.createTopic(
topic = s"${topic.toString}",
partitions= 10,
replicationFactor = 2,
zkUtils = zkUtils
)
logger.info(s"Created topic ${topic.toString}")
}
}
{code}
*Cause of death:*
{code:java}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
at kafka.cluster.Partition.makeLeader(Partition.scala:168)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
at
kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
... 28 more
{code}
Broker restart results the same OOM issues. All brokers will not be able to
start again.
was:
*Reproducing*: Create 5k topics *from the code* quickly, without any delays.
Wait until brokers will finish loading them. This will actually never happen,
since all brokers will go down one by one after approx 10-15 minutes or more,
depending on the hardware.
*Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
*Topology*: 3 brokers, 3 zk.
*Code for 5k topic creation:*
{code:java}
package kafka
import kafka.admin.AdminUtils
import kafka.utils.{Logging, ZkUtils}
object TestCreateTopics extends App with Logging {
val zkConnect = "grid978:2185"
var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
for (topic <- 1 to 5000) {
AdminUtils.createTopic(
topic = s"${topic.toString}",
partitions= 10,
replicationFactor = 2,
zkUtils = zkUtils
)
logger.info(s"Created topic ${topic.toString}")
}
}
{code}
*Cause of death:*
{code:java}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at
[jira] [Updated] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6343:
---
Description:
*Reproducing*: Create 5k topics *from the code* quickly, without any delays.
Wait until brokers will finish loading them. This will actually never happen,
since all brokers will go down one by one after approx 10-15 minutes or more,
depending on the hardware.
*Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
*Topology*: 3 brokers, 3 zk.
*Code for 5k topic creation:*
{code:java}
package kafka
import kafka.admin.AdminUtils
import kafka.utils.{Logging, ZkUtils}
object TestCreateTopics extends App with Logging {
val zkConnect = "grid978:2185"
var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
for (topic <- 1 to 5000) {
AdminUtils.createTopic(
topic = s"${topic.toString}",
partitions= 10,
replicationFactor = 2,
zkUtils = zkUtils
)
logger.info(s"Created topic ${topic.toString}")
}
}
{code}
*Cause of death:*
{code:java}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
at kafka.cluster.Partition.makeLeader(Partition.scala:168)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
at
kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
... 28 more
{code}
Broker restart results the same OOM issues.
was:
*Reproducing*: Create 5k topics *from the code* quickly, without any delays.
Wait until brokers will finish loading them. This will actually never happen,
since all brokers will go down one by one after approx 10-15 minutes or more,
depending on the hardware.
*Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
*Topology*: 3 brokers, 3 zk.
*Code for 5k topic creation:*
{code:java}
package kafka
import kafka.admin.AdminUtils
import kafka.utils.{Logging, ZkUtils}
object TestCreateTopics extends App with Logging {
val zkConnect = "grid978:2185"
var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
for (topic <- 1 to 5000) {
AdminUtils.createTopic(
topic = s"${topic.toString}",
partitions= 10,
replicationFactor = 2,
zkUtils = zkUtils
)
logger.info(s"Created topic ${topic.toString}")
}
}
{code}
*Cause of death:*
{code:java}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
at
[jira] [Comment Edited] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285834#comment-16285834
]
Alex Dunayevsky edited comment on KAFKA-6343 at 12/11/17 12:25 PM:
---
Ismael Juma, we have just reproduced the issue once again while keeping track
of open file handles. Here are the results:
{code:java}
$ while true; do cat /proc/sys/fs/file-nr; sleep 1; done
3024 0 30 <--- starting topic creation
...
66192 0 30 <--- all 5k topics created
... <--- broker continues topic loading
98560 0 30 <--- breaks here, this is where broker dies
1568 0 30 <--- after broker death
Where: the first column stands for "open file handles" and the last column
(30) stands for total file handles available in the system.
{code}
was (Author: alex.dunayevsky):
Ismael Juma, we have just reproduced the issue once again while keeping track
of open file handles. Here are the results:
{code:java}
$ while true; do cat /proc/sys/fs/file-nr; sleep 1; done
3024 0 30 <--- starting topic creation
...
66192 0 30 <--- all 5k topics created
... <--- broker continues topic loading
98560 0 30 <--- breaks here, this is where broker dies
1568 0 30 <--- after broker death
{code}
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at
[jira] [Comment Edited] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285834#comment-16285834
]
Alex Dunayevsky edited comment on KAFKA-6343 at 12/11/17 12:22 PM:
---
Ismael Juma, we have just reproduced the issue once again while keeping track
of open file handles. Here are the results:
{code:java}
$ while true; do cat /proc/sys/fs/file-nr; sleep 1; done
3024 0 30 <--- starting topic creation
...
66192 0 30 <--- all 5k topics created
... <--- broker continues topic loading
98560 0 30 <--- breaks here, this is where broker dies
1568 0 30 <--- after broker death
{code}
was (Author: alex.dunayevsky):
Ismael Juma, we have just reproduced the issue once again while keeping track
of open file handles. Here are the results:
{code:java}
$ while true; do cat /proc/sys/fs/file-nr; sleep 1; done
3024 0 30 <--- starting topic creation
...
66192 0 30 <--- all 5k topics created
... <--- broker continues topic loading
98560 0 30 <--- breaks here, this is where broker dies
1568 0 30 <--- after broker death
{code}
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at
[jira] [Commented] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285834#comment-16285834
]
Alex Dunayevsky commented on KAFKA-6343:
Ismael Juma, we have just reproduced the issue once again while keeping track
of open file handles. Here are the results:
{code:java}
$ while true; do cat /proc/sys/fs/file-nr; sleep 1; done
30240 30 <--- starting topic creation
...
66192 0 30 <--- all 5k topics created
... <--- broker continues topic loading
98560 0 30 <--- breaks here, this is where broker dies
15680 30 <--- after broker death
{code}
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
> ... 28 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285834#comment-16285834
]
Alex Dunayevsky edited comment on KAFKA-6343 at 12/11/17 12:22 PM:
---
Ismael Juma, we have just reproduced the issue once again while keeping track
of open file handles. Here are the results:
{code:java}
$ while true; do cat /proc/sys/fs/file-nr; sleep 1; done
3024 0 30 <--- starting topic creation
...
66192 0 30 <--- all 5k topics created
... <--- broker continues topic loading
98560 0 30 <--- breaks here, this is where broker dies
1568 0 30 <--- after broker death
{code}
was (Author: alex.dunayevsky):
Ismael Juma, we have just reproduced the issue once again while keeping track
of open file handles. Here are the results:
{code:java}
$ while true; do cat /proc/sys/fs/file-nr; sleep 1; done
30240 30 <--- starting topic creation
...
66192 0 30 <--- all 5k topics created
... <--- broker continues topic loading
98560 0 30 <--- breaks here, this is where broker dies
15680 30 <--- after broker death
{code}
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Map failed
>
[jira] [Commented] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285817#comment-16285817
]
Alex Dunayevsky commented on KAFKA-6343:
Ismael Juma, thank you for the answer. What I currently see is 30 file
handles according to /proc/sys/fs/file-max. Isn't this enough? If not then,
perhaps, is there a way to calculate this somehow?
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
> ... 28 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6343:
---
Environment: RHEL 7, RAM 755GB per host (was: RHEL 7, RAM 755GB)
> OOM as the result of creation of 5k topics
> --
>
> Key: KAFKA-6343
> URL: https://issues.apache.org/jira/browse/KAFKA-6343
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1
> Environment: RHEL 7, RAM 755GB per host
>Reporter: Alex Dunayevsky
>
> *Reproducing*: Create 5k topics *from the code* quickly, without any delays.
> Wait until brokers will finish loading them. This will actually never happen,
> since all brokers will go down one by one after approx 10-15 minutes or more,
> depending on the hardware.
> *Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
>
> *Topology*: 3 brokers, 3 zk.
> *Code for 5k topic creation:*
> {code:java}
> package kafka
> import kafka.admin.AdminUtils
> import kafka.utils.{Logging, ZkUtils}
> object TestCreateTopics extends App with Logging {
> val zkConnect = "grid978:2185"
> var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
> for (topic <- 1 to 5000) {
> AdminUtils.createTopic(
> topic = s"${topic.toString}",
> partitions= 10,
> replicationFactor = 2,
> zkUtils = zkUtils
> )
> logger.info(s"Created topic ${topic.toString}")
> }
> }
> {code}
> *Cause of death:*
> {code:java}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
> at
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
> at
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
> at
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
> ... 28 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6343:
---
Description:
*Reproducing*: Create 5k topics *from the code* quickly, without any delays.
Wait until brokers will finish loading them. This will actually never happen,
since all brokers will go down one by one after approx 10-15 minutes or more,
depending on the hardware.
*Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
*Topology*: 3 brokers, 3 zk.
*Code for 5k topic creation:*
{code:java}
package kafka
import kafka.admin.AdminUtils
import kafka.utils.{Logging, ZkUtils}
object TestCreateTopics extends App with Logging {
val zkConnect = "grid978:2185"
var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
for (topic <- 1 to 5000) {
AdminUtils.createTopic(
topic = s"${topic.toString}",
partitions= 10,
replicationFactor = 2,
zkUtils = zkUtils
)
logger.info(s"Created topic ${topic.toString}")
}
}
{code}
*OOM:*
{code:java}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
at kafka.cluster.Partition.makeLeader(Partition.scala:168)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
at
kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
... 28 more
{code}
was:
*Reproducing*: Create 5k topics *from the code* quickly, without any delays.
Wait until brokers will finish loading them. This will actually never happen,
since all brokers will go down one by one after approx 10-15 minutes or more,
depending on the hardware.
*Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G...
*Topology*: 3 brokers, 3 zk.
*Code for 5k topic creation:*
{code:java}
package kafka
import kafka.admin.AdminUtils
import kafka.utils.{Logging, ZkUtils}
object TestCreateTopics extends App with Logging {
val zkConnect = "grid978:2185"
var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
for (topic <- 1 to 5000) {
AdminUtils.createTopic(
topic = s"${topic.toString}",
partitions= 10,
replicationFactor = 2,
zkUtils = zkUtils
)
logger.info(s"Created topic ${topic.toString}")
}
}
{code}
*OOM:*
{code:java}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at
[jira] [Updated] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6343:
---
Description:
*Reproducing*: Create 5k topics *from the code* quickly, without any delays.
Wait until brokers will finish loading them. This will actually never happen,
since all brokers will go down one by one after approx 10-15 minutes or more,
depending on the hardware.
*Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
*Topology*: 3 brokers, 3 zk.
*Code for 5k topic creation:*
{code:java}
package kafka
import kafka.admin.AdminUtils
import kafka.utils.{Logging, ZkUtils}
object TestCreateTopics extends App with Logging {
val zkConnect = "grid978:2185"
var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
for (topic <- 1 to 5000) {
AdminUtils.createTopic(
topic = s"${topic.toString}",
partitions= 10,
replicationFactor = 2,
zkUtils = zkUtils
)
logger.info(s"Created topic ${topic.toString}")
}
}
{code}
*Cause of death:*
{code:java}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
at kafka.cluster.Partition.makeLeader(Partition.scala:168)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
at
kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
... 28 more
{code}
was:
*Reproducing*: Create 5k topics *from the code* quickly, without any delays.
Wait until brokers will finish loading them. This will actually never happen,
since all brokers will go down one by one after approx 10-15 minutes or more,
depending on the hardware.
*Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G
*Topology*: 3 brokers, 3 zk.
*Code for 5k topic creation:*
{code:java}
package kafka
import kafka.admin.AdminUtils
import kafka.utils.{Logging, ZkUtils}
object TestCreateTopics extends App with Logging {
val zkConnect = "grid978:2185"
var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
for (topic <- 1 to 5000) {
AdminUtils.createTopic(
topic = s"${topic.toString}",
partitions= 10,
replicationFactor = 2,
zkUtils = zkUtils
)
logger.info(s"Created topic ${topic.toString}")
}
}
{code}
*OOM:*
{code:java}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
[jira] [Updated] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6343:
---
Description:
Create 5k topics *from the code* quickly, without any delays. Wait until
brokers will finish loading them. This will actually never happen, since all
brokers will go down after approx 10-15 minutes or more, depending on the
hardware.
*Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G...
*Topology*: 3 brokers, 3 zk.
*Code for 5k topic creation:*
{code:java}
package kafka
import kafka.admin.AdminUtils
import kafka.utils.{Logging, ZkUtils}
object TestCreateTopics extends App with Logging {
val zkConnect = "grid978:2185"
var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
for (topic <- 1 to 5000) {
AdminUtils.createTopic(
topic = s"${topic.toString}",
partitions= 10,
replicationFactor = 2,
zkUtils = zkUtils
)
logger.info(s"Created topic ${topic.toString}")
}
}
{code}
*OOM:*
{code:java}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
at kafka.cluster.Partition.makeLeader(Partition.scala:168)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
at
kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
... 28 more
{code}
was:
Create 5k topics *from the code* - quickly, without any delays. Wait until
brokers will finish loading them. This will actually never happen, since all
brokers will go down after approx 10-15 minutes or more, depending on the
hardware.
*Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G...
*Topology*: 3 brokers, 3 zk.
*Code for 5k topic creation:*
{code:java}
package kafka
import kafka.admin.AdminUtils
import kafka.utils.{Logging, ZkUtils}
object TestCreateTopics extends App with Logging {
val zkConnect = "grid978:2185"
var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
for (topic <- 1 to 5000) {
AdminUtils.createTopic(
topic = s"${topic.toString}",
partitions= 10,
replicationFactor = 2,
zkUtils = zkUtils
)
logger.info(s"Created topic ${topic.toString}")
}
}
{code}
*OOM:*
{code:java}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at
[jira] [Updated] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6343:
---
Description:
*Reproducing*: Create 5k topics *from the code* quickly, without any delays.
Wait until brokers will finish loading them. This will actually never happen,
since all brokers will go down after approx 10-15 minutes or more, depending on
the hardware.
*Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G...
*Topology*: 3 brokers, 3 zk.
*Code for 5k topic creation:*
{code:java}
package kafka
import kafka.admin.AdminUtils
import kafka.utils.{Logging, ZkUtils}
object TestCreateTopics extends App with Logging {
val zkConnect = "grid978:2185"
var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
for (topic <- 1 to 5000) {
AdminUtils.createTopic(
topic = s"${topic.toString}",
partitions= 10,
replicationFactor = 2,
zkUtils = zkUtils
)
logger.info(s"Created topic ${topic.toString}")
}
}
{code}
*OOM:*
{code:java}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
at kafka.cluster.Partition.makeLeader(Partition.scala:168)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
at
kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
... 28 more
{code}
was:
Create 5k topics *from the code* quickly, without any delays. Wait until
brokers will finish loading them. This will actually never happen, since all
brokers will go down after approx 10-15 minutes or more, depending on the
hardware.
*Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G...
*Topology*: 3 brokers, 3 zk.
*Code for 5k topic creation:*
{code:java}
package kafka
import kafka.admin.AdminUtils
import kafka.utils.{Logging, ZkUtils}
object TestCreateTopics extends App with Logging {
val zkConnect = "grid978:2185"
var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
for (topic <- 1 to 5000) {
AdminUtils.createTopic(
topic = s"${topic.toString}",
partitions= 10,
replicationFactor = 2,
zkUtils = zkUtils
)
logger.info(s"Created topic ${topic.toString}")
}
}
{code}
*OOM:*
{code:java}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at
[jira] [Updated] (KAFKA-6343) OOM as the result of creation of 5k topics
[
https://issues.apache.org/jira/browse/KAFKA-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6343:
---
Description:
*Reproducing*: Create 5k topics *from the code* quickly, without any delays.
Wait until brokers will finish loading them. This will actually never happen,
since all brokers will go down one by one after approx 10-15 minutes or more,
depending on the hardware.
*Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G...
*Topology*: 3 brokers, 3 zk.
*Code for 5k topic creation:*
{code:java}
package kafka
import kafka.admin.AdminUtils
import kafka.utils.{Logging, ZkUtils}
object TestCreateTopics extends App with Logging {
val zkConnect = "grid978:2185"
var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
for (topic <- 1 to 5000) {
AdminUtils.createTopic(
topic = s"${topic.toString}",
partitions= 10,
replicationFactor = 2,
zkUtils = zkUtils
)
logger.info(s"Created topic ${topic.toString}")
}
}
{code}
*OOM:*
{code:java}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
at kafka.cluster.Partition.makeLeader(Partition.scala:168)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
at
kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
... 28 more
{code}
was:
*Reproducing*: Create 5k topics *from the code* quickly, without any delays.
Wait until brokers will finish loading them. This will actually never happen,
since all brokers will go down after approx 10-15 minutes or more, depending on
the hardware.
*Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G...
*Topology*: 3 brokers, 3 zk.
*Code for 5k topic creation:*
{code:java}
package kafka
import kafka.admin.AdminUtils
import kafka.utils.{Logging, ZkUtils}
object TestCreateTopics extends App with Logging {
val zkConnect = "grid978:2185"
var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
for (topic <- 1 to 5000) {
AdminUtils.createTopic(
topic = s"${topic.toString}",
partitions= 10,
replicationFactor = 2,
zkUtils = zkUtils
)
logger.info(s"Created topic ${topic.toString}")
}
}
{code}
*OOM:*
{code:java}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at
[jira] [Created] (KAFKA-6343) OOM as the result of creation of 5k topics
Alex Dunayevsky created KAFKA-6343:
--
Summary: OOM as the result of creation of 5k topics
Key: KAFKA-6343
URL: https://issues.apache.org/jira/browse/KAFKA-6343
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.10.1.1
Environment: RHEL 7, RAM 755GB
Reporter: Alex Dunayevsky
Create 5k topics *from the code* - quickly, without any delays. Wait until
brokers will finish loading them. This will actually never happen, since all
brokers will go down after approx 10-15 minutes or more, depending on the
hardware.
*Heap*: -Xmx/Xms: 5G, 10G, 50G, 256G...
*Topology*: 3 brokers, 3 zk.
*Code for 5k topic creation:*
{code:java}
package kafka
import kafka.admin.AdminUtils
import kafka.utils.{Logging, ZkUtils}
object TestCreateTopics extends App with Logging {
val zkConnect = "grid978:2185"
var zkUtils = ZkUtils(zkConnect, 6000, 6000, isZkSecurityEnabled = false)
for (topic <- 1 to 5000) {
AdminUtils.createTopic(
topic = s"${topic.toString}",
partitions= 10,
replicationFactor = 2,
zkUtils = zkUtils
)
logger.info(s"Created topic ${topic.toString}")
}
}
{code}
*OOM:*
{code:java}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:920)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
at kafka.cluster.Partition.makeLeader(Partition.scala:168)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:758)
at
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:757)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:757)
at
kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:703)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:917)
... 28 more
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-6221) ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic creation
[
https://issues.apache.org/jira/browse/KAFKA-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6221:
---
Priority: Minor (was: Major)
> ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic
> creation
> ---
>
> Key: KAFKA-6221
> URL: https://issues.apache.org/jira/browse/KAFKA-6221
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1, 0.10.2.0, 0.10.2.1, 0.11.0.1, 1.0.0
> Environment: RHEL 7
>Reporter: Alex Dunayevsky
>Priority: Minor
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> This issue appeared to happen frequently on 0.10.2.0.
> On 0.10.2.1 and 1.0.0 it's a way harder to reproduce.
> We'll focus on reproducing it on 0.10.2.1 and 1.0.0.
> *TOPOLOGY:*
> 3 brokers, 1 zk.
> *REPRODUCING STRATEGY:*
> Create a few dozens topics (say, 40) one by one, each with replication factor
> 2. Number of partitions, generally, does not matter but, for easier
> reproduction, should not be too small (around 30 or so).
> *CREATE 40 TOPICS:*
> {code:java} for i in {1..40}; do bin/kafka-topics.sh --create --topic
> "topic${i}_p28_r2" --partitions 28 --replication-factor 2 --zookeeper :2165;
> done {code}
> *ERRORS*
> {code:java}
> *BROKER 1*
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> *BROKER 2*
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,0] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition.
[jira] [Updated] (KAFKA-6221) ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic creation
[
https://issues.apache.org/jira/browse/KAFKA-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6221:
---
Priority: Major (was: Minor)
> ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic
> creation
> ---
>
> Key: KAFKA-6221
> URL: https://issues.apache.org/jira/browse/KAFKA-6221
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1, 0.10.2.0, 0.10.2.1, 0.11.0.1, 1.0.0
> Environment: RHEL 7
>Reporter: Alex Dunayevsky
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> This issue appeared to happen frequently on 0.10.2.0.
> On 0.10.2.1 and 1.0.0 it's a way harder to reproduce.
> We'll focus on reproducing it on 0.10.2.1 and 1.0.0.
> *TOPOLOGY:*
> 3 brokers, 1 zk.
> *REPRODUCING STRATEGY:*
> Create a few dozens topics (say, 40) one by one, each with replication factor
> 2. Number of partitions, generally, does not matter but, for easier
> reproduction, should not be too small (around 30 or so).
> *CREATE 40 TOPICS:*
> {code:java} for i in {1..40}; do bin/kafka-topics.sh --create --topic
> "topic${i}_p28_r2" --partitions 28 --replication-factor 2 --zookeeper :2165;
> done {code}
> *ERRORS*
> {code:java}
> *BROKER 1*
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> *BROKER 2*
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,0] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
>
[jira] [Updated] (KAFKA-6221) ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic creation
[
https://issues.apache.org/jira/browse/KAFKA-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6221:
---
Affects Version/s: 0.10.1.1
> ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic
> creation
> ---
>
> Key: KAFKA-6221
> URL: https://issues.apache.org/jira/browse/KAFKA-6221
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.1.1, 0.10.2.0, 0.10.2.1, 0.11.0.1, 1.0.0
> Environment: RHEL 7
>Reporter: Alex Dunayevsky
>Priority: Minor
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> This issue appeared to happen frequently on 0.10.2.0.
> On 0.10.2.1 and 1.0.0 it's a way harder to reproduce.
> We'll focus on reproducing it on 0.10.2.1 and 1.0.0.
> *TOPOLOGY:*
> 3 brokers, 1 zk.
> *REPRODUCING STRATEGY:*
> Create a few dozens topics (say, 40) one by one, each with replication factor
> 2. Number of partitions, generally, does not matter but, for easier
> reproduction, should not be too small (around 30 or so).
> *CREATE 40 TOPICS:*
> {code:java} for i in {1..40}; do bin/kafka-topics.sh --create --topic
> "topic${i}_p28_r2" --partitions 28 --replication-factor 2 --zookeeper :2165;
> done {code}
> *ERRORS*
> {code:java}
> *BROKER 1*
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> *BROKER 2*
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,0] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition.
[jira] [Updated] (KAFKA-6221) ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic creation
[
https://issues.apache.org/jira/browse/KAFKA-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6221:
---
Affects Version/s: 0.11.0.1
> ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic
> creation
> ---
>
> Key: KAFKA-6221
> URL: https://issues.apache.org/jira/browse/KAFKA-6221
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.2.0, 0.10.2.1, 0.11.0.1, 1.0.0
> Environment: RHEL 7
>Reporter: Alex Dunayevsky
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> This issue appeared to happen frequently on 0.10.2.0.
> On 0.10.2.1 and 1.0.0 it's a way harder to reproduce.
> We'll focus on reproducing it on 0.10.2.1 and 1.0.0.
> *TOPOLOGY:*
> 3 brokers, 1 zk.
> *REPRODUCING STRATEGY:*
> Create a few dozens topics (say, 40) one by one, each with replication factor
> 2. Number of partitions, generally, does not matter but, for easier
> reproduction, should not be too small (around 30 or so).
> *CREATE 40 TOPICS:*
> {code:java} for i in {1..40}; do bin/kafka-topics.sh --create --topic
> "topic${i}_p28_r2" --partitions 28 --replication-factor 2 --zookeeper :2165;
> done {code}
> *ERRORS*
> {code:java}
> *BROKER 1*
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> *BROKER 2*
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,0] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15
[jira] [Updated] (KAFKA-6221) ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic creation
[
https://issues.apache.org/jira/browse/KAFKA-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6221:
---
Description:
This issue appeared to happen frequently on 0.10.2.0.
On 0.10.2.1 and 1.0.0 it's a way harder to reproduce.
We'll focus on reproducing it on 0.10.2.1 and 1.0.0.
*TOPOLOGY:*
3 brokers, 1 zk.
*REPRODUCING STRATEGY:*
Create a few dozens topics (say, 40) one by one, each with replication factor
2. Number of partitions, generally, does not matter but, for easier
reproduction, should not be too small (around 30 or so).
*CREATE 40 TOPICS:*
{code:java} for i in {1..40}; do bin/kafka-topics.sh --create --topic
"topic${i}_p28_r2" --partitions 28 --replication-factor 2 --zookeeper :2165;
done {code}
*ERRORS*
{code:java}
*BROKER 1*
[2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,27] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,27] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,9] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,9] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,3] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,3] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,15] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,15] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,21] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,21] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
*BROKER 2*
[2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,12] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,12] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,0] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,0] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,6] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,6] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
[jira] [Updated] (KAFKA-6221) ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic creation
[
https://issues.apache.org/jira/browse/KAFKA-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6221:
---
Affects Version/s: 1.0.0
> ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic
> creation
> ---
>
> Key: KAFKA-6221
> URL: https://issues.apache.org/jira/browse/KAFKA-6221
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.2.0, 0.10.2.1, 1.0.0
> Environment: RHEL 7
>Reporter: Alex Dunayevsky
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> This issue appeared to happen frequently on 0.10.2.0.
> On 0.10.2.1 it's a way harder to reproduce.
> We'll focus on reproducing it on 0.10.2.1.
> *TOPOLOGY:*
> 3 brokers, 1 zk.
> *REPRODUCING STRATEGY:*
> Create a few dozens topics (say, 40) one by one, each with replication factor
> 2. Number of partitions, generally, does not matter but, for easier
> reproduction, should not be too small (around 30 or so).
> *CREATE 40 TOPICS:*
> {code:java} for i in {1..40}; do bin/kafka-topics.sh --create --topic
> "topic${i}_p28_r2" --partitions 28 --replication-factor 2 --zookeeper :2165;
> done {code}
> *ERRORS*
> {code:java}
> *BROKER 1*
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> *BROKER 2*
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,0] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,410] ERROR
[jira] [Comment Edited] (KAFKA-6221) ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic creation
[
https://issues.apache.org/jira/browse/KAFKA-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256703#comment-16256703
]
Alex Dunayevsky edited comment on KAFKA-6221 at 11/17/17 9:20 AM:
--
*huxihx*, thank you for the explanation! Nope, no exceptions later, everything
works fine, but it's quite confusing to observe this when deploying Kafka in
production... I believe this should *not* be considered as a normal Kafka
behavior and should be fixed. What do you think?
was (Author: alex.dunayevsky):
*huxihx*, thank you for the explanation! Nope, no exceptions later, everything
works fine, but it's quite confusing to observe this when deploying Kafka in
production... I believe this should *not * be considered as a normal Kafka
behavior and should be fixed. What do you think?
> ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic
> creation
> ---
>
> Key: KAFKA-6221
> URL: https://issues.apache.org/jira/browse/KAFKA-6221
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.2.0, 0.10.2.1
> Environment: RHEL 7
>Reporter: Alex Dunayevsky
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> This issue appeared to happen frequently on 0.10.2.0.
> On 0.10.2.1 it's a way harder to reproduce.
> We'll focus on reproducing it on 0.10.2.1.
> *TOPOLOGY:*
> 3 brokers, 1 zk.
> *REPRODUCING STRATEGY:*
> Create a few dozens topics (say, 40) one by one, each with replication factor
> 2. Number of partitions, generally, does not matter but, for easier
> reproduction, should not be too small (around 30 or so).
> *CREATE 40 TOPICS:*
> {code:java} for i in {1..40}; do bin/kafka-topics.sh --create --topic
> "topic${i}_p28_r2" --partitions 28 --replication-factor 2 --zookeeper :2165;
> done {code}
> *ERRORS*
> {code:java}
> *BROKER 1*
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> *BROKER 2*
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not
[jira] [Comment Edited] (KAFKA-6221) ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic creation
[
https://issues.apache.org/jira/browse/KAFKA-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256703#comment-16256703
]
Alex Dunayevsky edited comment on KAFKA-6221 at 11/17/17 9:20 AM:
--
*huxihx*, thank you for the explanation! Nope, no exceptions later, everything
works fine, but it's quite confusing to observe this when deploying Kafka in
production... I believe this should *not * be considered as a normal Kafka
behavior and should be fixed. What do you think?
was (Author: alex.dunayevsky):
*huxihx*, thank you for the explanation! Nope, no exceptions later, everything
works fine, but it's quite confusing to observe this when deploying Kafka in
production... I believe this should not be considered as a normal Kafka
behavior and should be fixed. What do you think?
> ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic
> creation
> ---
>
> Key: KAFKA-6221
> URL: https://issues.apache.org/jira/browse/KAFKA-6221
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.2.0, 0.10.2.1
> Environment: RHEL 7
>Reporter: Alex Dunayevsky
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> This issue appeared to happen frequently on 0.10.2.0.
> On 0.10.2.1 it's a way harder to reproduce.
> We'll focus on reproducing it on 0.10.2.1.
> *TOPOLOGY:*
> 3 brokers, 1 zk.
> *REPRODUCING STRATEGY:*
> Create a few dozens topics (say, 40) one by one, each with replication factor
> 2. Number of partitions, generally, does not matter but, for easier
> reproduction, should not be too small (around 30 or so).
> *CREATE 40 TOPICS:*
> {code:java} for i in {1..40}; do bin/kafka-topics.sh --create --topic
> "topic${i}_p28_r2" --partitions 28 --replication-factor 2 --zookeeper :2165;
> done {code}
> *ERRORS*
> {code:java}
> *BROKER 1*
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> *BROKER 2*
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not
[jira] [Commented] (KAFKA-6221) ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic creation
[
https://issues.apache.org/jira/browse/KAFKA-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256703#comment-16256703
]
Alex Dunayevsky commented on KAFKA-6221:
*huxihx*, thank you for the explanation! Nope, no exceptions later, everything
works fine, but it's quite confusing to observe this when deploying Kafka in
production... I believe this should not be considered as a normal Kafka
behavior and should be fixed. What do you think?
> ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic
> creation
> ---
>
> Key: KAFKA-6221
> URL: https://issues.apache.org/jira/browse/KAFKA-6221
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.2.0, 0.10.2.1
> Environment: RHEL 7
>Reporter: Alex Dunayevsky
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> This issue appeared to happen frequently on 0.10.2.0.
> On 0.10.2.1 it's a way harder to reproduce.
> We'll focus on reproducing it on 0.10.2.1.
> *TOPOLOGY:*
> 3 brokers, 1 zk.
> *REPRODUCING STRATEGY:*
> Create a few dozens topics (say, 40) one by one, each with replication factor
> 2. Number of partitions, generally, does not matter but, for easier
> reproduction, should not be too small (around 30 or so).
> *CREATE 40 TOPICS:*
> {code:java} for i in {1..40}; do bin/kafka-topics.sh --create --topic
> "topic${i}_p28_r2" --partitions 28 --replication-factor 2 --zookeeper :2165;
> done {code}
> *ERRORS*
> {code:java}
> *BROKER 1*
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> *BROKER 2*
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,410] ERROR
[jira] [Updated] (KAFKA-6221) ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic creation
[
https://issues.apache.org/jira/browse/KAFKA-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6221:
---
Summary: ReplicaFetcherThread throws UnknownTopicOrPartitionException on
topic creation (was: ReplicaFetcherThread throws
UnknownTopicOrPartitionExeption on topic creation )
> ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic
> creation
> ---
>
> Key: KAFKA-6221
> URL: https://issues.apache.org/jira/browse/KAFKA-6221
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.2.0, 0.10.2.1
> Environment: RHEL 7
>Reporter: Alex Dunayevsky
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> This issue appeared to happen frequently on 0.10.2.0.
> On 0.10.2.1 it's a way harder to reproduce.
> We'll focus on reproducing it on 0.10.2.1.
> *TOPOLOGY:*
> 3 brokers, 1 zk.
> *REPRODUCING STRATEGY:*
> Create a few dozens topics (say, 40) one by one, each with replication factor
> 2. Number of partitions, generally, does not matter but, for easier
> reproduction, should not be too small (around 30 or so).
> *CREATE 40 TOPICS:*
> {code:java} for i in {1..40}; do bin/kafka-topics.sh --create --topic
> "topic${i}_p28_r2" --partitions 28 --replication-factor 2 --zookeeper :2165;
> done {code}
> *ERRORS*
> {code:java}
> *BROKER 1*
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> *BROKER 2*
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,0] to broker
>
[jira] [Updated] (KAFKA-6221) ReplicaFetcherThread throws UnknownTopicOrPartitionExeption on topic creation
[
https://issues.apache.org/jira/browse/KAFKA-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6221:
---
Description:
This issue appeared to happen frequently on 0.10.2.0.
On 0.10.2.1 it's a way harder to reproduce.
We'll focus on reproducing it on 0.10.2.1.
*TOPOLOGY:*
3 brokers, 1 zk.
*REPRODUCING STRATEGY:*
Create a few dozens topics (say, 40) one by one, each with replication factor
2. Number of partitions, generally, does not matter but, for easier
reproduction, should not be too small (around 30 or so).
*CREATE 40 TOPICS:*
{code:java} for i in {1..40}; do bin/kafka-topics.sh --create --topic
"topic${i}_p28_r2" --partitions 28 --replication-factor 2 --zookeeper :2165;
done {code}
*ERRORS*
{code:java}
*BROKER 1*
[2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,27] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,27] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,9] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,9] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,3] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,3] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,15] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,15] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,21] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,21] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
*BROKER 2*
[2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,12] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,12] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,0] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,0] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,6] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,6] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not
[jira] [Closed] (KAFKA-6220) ReplicaFetcherThread throws UnknownTopicOrPartitionExeption on topic creation
[
https://issues.apache.org/jira/browse/KAFKA-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky closed KAFKA-6220.
--
DUP: https://issues.apache.org/jira/browse/KAFKA-6221
> ReplicaFetcherThread throws UnknownTopicOrPartitionExeption on topic creation
> --
>
> Key: KAFKA-6220
> URL: https://issues.apache.org/jira/browse/KAFKA-6220
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.2.0, 0.10.2.1
> Environment: RHEL 7
>Reporter: Alex Dunayevsky
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> This issue appeared to happen frequently on 0.10.2.0. On 0.10.2.1 it's a way
> harder to reproduce. We'll focus on reproducing it on 0.10.2.1.
> *Topology:* 3 brokers, 1 zk.
> *Reproducing strategy:* create a few dozens topics (say, 40) one by one, each
> with replication factor 2. Number of partitions, generally, does not matter
> but, for easier reproduction, should not be very small (around 30 or so).
> *CREATE 40 TOPICS:*
> for i in {1..40}; do bin/kafka-topics.sh --create --topic "topic${i}_p28_r2"
> --partitions 28 --replication-factor 2 --zookeeper :2165; done
> {code:java}
> *BROKER 1*
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,27] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,9] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,3] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,15] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for
> partition [topic1_p28_r2,21] to broker
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> *BROKER 2*
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,12] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for
> partition [topic20_p28_r2,0] to broker
> 3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> [2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for
>
[jira] [Updated] (KAFKA-6221) ReplicaFetcherThread throws UnknownTopicOrPartitionExeption on topic creation
[
https://issues.apache.org/jira/browse/KAFKA-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6221:
---
Description:
This issue appeared to happen frequently on 0.10.2.0.
On 0.10.2.1 it's a way harder to reproduce.
We'll focus on reproducing it on 0.10.2.1.
*TOPOLOGY:*
3 brokers, 1 zk.
*REPRODUCING STRATEGY:*
Create a few dozens topics (say, 40) one by one, each with replication factor
2. Number of partitions, generally, does not matter but, for easier
reproduction, should not be too small (around 30 or so).
*CREATE 40 TOPICS (PARTITIONS: 28, REPLICATION FACTOR: 2):*
{code:java} for i in {1..40}; do bin/kafka-topics.sh --create --topic
"topic${i}_p28_r2" --partitions 28 --replication-factor 2 --zookeeper :2165;
done {code}
*ERRORS*
{code:java}
*BROKER 1*
[2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,27] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,27] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,9] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,9] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,3] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,3] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,15] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,15] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,21] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,21] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
*BROKER 2*
[2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,12] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,12] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,0] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,0] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,6] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,6] to broker
[jira] [Updated] (KAFKA-6221) ReplicaFetcherThread throws UnknownTopicOrPartitionExeption on topic creation
[
https://issues.apache.org/jira/browse/KAFKA-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6221:
---
Description:
This issue appeared to happen frequently on 0.10.2.0.
On 0.10.2.1 it's a way harder to reproduce.
We'll focus on reproducing it on 0.10.2.1.
*TOPOLOGY:*
3 brokers, 1 zk.
*REPRODUCING STRATEGY:*
Create a few dozens topics (say, 40) one by one, each with replication factor
2. Number of partitions, generally, does not matter but, for easier
reproduction, should not be too small (around 30 or so).
*CREATE 40 TOPICS:*
{code:java} for i in {1..40}; do bin/kafka-topics.sh --create --topic
"topic${i}_p28_r2" --partitions 28 --replication-factor 2 --zookeeper :2165;
done {code}
*ERRORS*
{code:java}
*BROKER 1*
[2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,27] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,27] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,9] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,9] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,3] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,3] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,15] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,15] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,21] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,21] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
*BROKER 2*
[2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,12] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,12] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,0] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,0] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,6] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,6] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not
[jira] [Updated] (KAFKA-6221) ReplicaFetcherThread throws UnknownTopicOrPartitionExeption on topic creation
[
https://issues.apache.org/jira/browse/KAFKA-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Dunayevsky updated KAFKA-6221:
---
Description:
This issue appeared to happen frequently on 0.10.2.0.
On 0.10.2.1 it's a way harder to reproduce.
We'll focus on reproducing it on 0.10.2.1.
*TOPOLOGY:*
3 brokers, 1 zk.
*REPRODUCING STRATEGY:*
Create a few dozens topics (say, 40) one by one, each with replication factor
2. Number of partitions, generally, does not matter but, for easier
reproduction, should not be too small (around 30 or so).
*CREATE 40 TOPICS:*
{code:java} for i in {1..40}; do bin/kafka-topics.sh --create --topic
"topic${i}_p28_r2" --partitions 28 --replication-factor 2 --zookeeper :2165;
done {code}
*ERRORS*
{code:java}
*BROKER 1*
[2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,27] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,853] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,27] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,9] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,9] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,3] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,3] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,15] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,15] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,21] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:00,854] ERROR [ReplicaFetcherThread-0-2], Error for partition
[topic1_p28_r2,21] to broker
2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
*BROKER 2*
[2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,12] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,408] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,12] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,0] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,0] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,6] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-11-15 16:46:36,410] ERROR [ReplicaFetcherThread-0-3], Error for partition
[topic20_p28_r2,6] to broker
3:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not
