Thanks Ismael!
I agree clear failures or no failures is optimal. I did some hacky analysis
of the open files by running the tests and utilizing the lsof command.
In one run of the core tests I found the following:
- 4584 regular files (REG)
- 376 .jar files
- Not much one can/should do here. Many are from gradle itself.
- 2392 kafka .log files
- why are these being leaked?
- after a single test no file handles should remain
- 1162 kafka .log.deleted files
- why are these being leaked?
- 469 kafka .index files
- This is due to Java's handling of MappedByteBuffer
- A mapped byte buffer and the file mapping that it represents
remain valid until the buffer itself is garbage-collected.
- http://bugs.java.com/view_bug.do?bug_id=4724038
-
http://stackoverflow.com/questions/2972986/how-to-unmap-a-file-from-memory-mapped-using-filechannel-in-java
- Perhaps setting mmap to null when kafka.log.OffsetIndex.close
is called would help ensure this gets GC'd asap.
- 943 of types PIPE & KQUEUE
- 629 PIPE
- 314 KQUEUE
- should do some analysis sometime
- 47 of other types (TCP, unix, IPv6, ...)
On Sun, Jul 19, 2015 at 3:16 PM, Ismael Juma <ism...@juma.me.uk> wrote:
> Hello Grant,
>
> Thanks for figuring this out. I have also run into this issue when running
> the tests on OS X Yosemite.
>
> Ideally the tests would fail in a way that would make it clear what the
> issue is. That may be complicated, so we should at least document it as you
> suggest.
>
> I'll let you know if the issues goes away for me too with this change.
>
> Best,
> Ismael
>
> On Sun, Jul 19, 2015 at 4:24 PM, Grant Henke <ghe...@cloudera.com> wrote:
>
> > When running all Kafka tests I had been getting failures most every time.
> > Usually in the SocketServerTest class. However, when I would run
> individual
> > tests, there were no failures. After a bit of digging I found this is due
> > to the small default open files limit in Mac Yosemite. I am positing how
> to
> > increase the limit here in case anyone else has been running into the
> > issue. Let me know if this helped you too. If it is fairly common we can
> > put something on the wiki.
> >
> > *Adjusting Open File Limits in Yosemite:*
> > Note: You can choose your own limits as appropriate
> >
> > 1. Write the following xml to
> /Library/LaunchDaemons/limit.maxfiles.plist:
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> >
> > <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "
> > http://www.apple.com/DTDs/PropertyList-1.0.dtd";>
> >
> > <plist version="1.0">
> >
> > <dict>
> >
> > <key>Label</key>
> >
> > <string>limit.maxfiles</string>
> >
> > <key>ProgramArguments</key>
> >
> > <array>
> >
> > <string>launchctl</string>
> >
> > <string>limit</string>
> >
> > <string>maxfiles</string>
> >
> > <string>65536</string>
> >
> > <string>65536</string>
> >
> > </array>
> >
> > <key>RunAtLoad</key>
> >
> > <true/>
> >
> > <key>ServiceIPC</key>
> >
> > <false/>
> >
> > </dict>
> >
> > </plist>
> >
> >
> > 2. Then write the following to
> /Library/LaunchDaemons/limit.maxproc.plist:
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> > <!DOCTYPE plist PUBLIC "-//Apple/DTD PLIST 1.0//EN" "
> > http://www.apple.com/DTDs/PropertyList-1.0.dtd";>
> > <plist version="1.0">
> > <dict>
> > <key>Label</key>
> > <string>limit.maxproc</string>
> > <key>ProgramArguments</key>
> > <array>
> > <string>launchctl</string>
> > <string>limit</string>
> > <string>maxproc</string>
> > <string>2048</string>
> > <string>2048</string>
> > </array>
> > <key>RunAtLoad</key>
> > <true />
> > <key>ServiceIPC</key>
> > <false />
> > </dict>
> > </plist>
> >
> >
> > 3. Add the following to your bashrc or bashprofile:
> >
> > ulimit -n 65536
> >
> > ulimit -u 2048
> >
> >
> > 4. Restart your computer. After restart validate settings by executing:
> >
> > launchctl limit
> >
> >
> >
> > *Adjusting Open File Limits in Older Versions of OS X:*
> > Note: You can choose your own limits as appropriate
> >
> > 1. Add the following command to /etc/launchd.conf:
> >
> > limit maxfiles 32768 65536
> >
> >
> > 2. Restart your computer. After restart validate settings by executing:
> >
> > launchctl limit
> >
>
--
Grant Henke
Solutions Consultant | Cloudera
ghe...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
