[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-11-02 Thread Vinoth Chandar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985707#comment-14985707 ] Vinoth Chandar commented on KAFKA-2580: --- [~jkreps] ah ok. point taken :) > Kafka Broker keeps file

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-11-02 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985696#comment-14985696 ] Jay Kreps commented on KAFKA-2580: -- [~vinothchandar] All I'm saying is that you have to kind of do some

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-11-02 Thread Grant Henke (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985875#comment-14985875 ] Grant Henke commented on KAFKA-2580: Marking as Unassigned as I need to shift my focus to some other

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-10-31 Thread Vinoth Chandar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984243#comment-14984243 ] Vinoth Chandar commented on KAFKA-2580: --- Based on this, looks like we can close this? >> So a lot

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-10-19 Thread Grant Henke (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963467#comment-14963467 ] Grant Henke commented on KAFKA-2580: I have some time to work on this, but would like to see if we can

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-10-19 Thread Todd Palino (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964003#comment-14964003 ] Todd Palino commented on KAFKA-2580: It's about as graceful as an OOM, which is to say "not very".

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-10-19 Thread Todd Palino (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963986#comment-14963986 ] Todd Palino commented on KAFKA-2580: I agree with [~jkreps] here, that having a high FD limit is not a

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-10-19 Thread Grant Henke (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963998#comment-14963998 ] Grant Henke commented on KAFKA-2580: If we decide not to implement this and recommend setting a high

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-10-19 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964382#comment-14964382 ] Jay Kreps commented on KAFKA-2580: -- Yeah as [~toddpalino] says it is totally not graceful--it's a hard

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-10-19 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963632#comment-14963632 ] Jay Kreps commented on KAFKA-2580: -- 10TB of space with 1GB segment files means about 10k FDs (though

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-10-19 Thread Vinoth Chandar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963574#comment-14963574 ] Vinoth Chandar commented on KAFKA-2580: --- [~jjkoshy] Good point. if I understand correctly, even if

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-10-05 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943698#comment-14943698 ] Joel Koshy commented on KAFKA-2580: --- I was wondering if we can do with something much simpler -

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-10-02 Thread Vinoth Chandar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941910#comment-14941910 ] Vinoth Chandar commented on KAFKA-2580: --- A LRU file handle is something very commonly employed in

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-09-30 Thread Grant Henke (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937017#comment-14937017 ] Grant Henke commented on KAFKA-2580: A few notes/questions from my initial look at the LogManager: -

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-09-28 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14933781#comment-14933781 ] Joel Koshy commented on KAFKA-2580: --- [~guozhang] actually it is now at 400k. [~vinothchandar] yes that

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-09-25 Thread Vinoth Chandar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908298#comment-14908298 ] Vinoth Chandar commented on KAFKA-2580: --- More context on how we determined this {code}

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-09-25 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908905#comment-14908905 ] Guozhang Wang commented on KAFKA-2580: -- Hey [~vinothchandar], could you let me know the log file

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

2015-09-25 Thread Vinoth Chandar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908910#comment-14908910 ] Vinoth Chandar commented on KAFKA-2580: --- Thanks for jumping in [~guozhang] .we have 256MB segment