Re: High commit size
It is memory-mapped I/O. I wouldn't worry about it. BTW, Windows might not be the best choice to run Cassandra on. My experience running Cassandra on Windows has not been positive one. We no longer support Windows as our production platform. Regards, Oleg On 2012-09-10 09:00:02 +, Rene Kochen said: Hi all, On my test cluster I have three Windows Server 2008 R2 machines running Cassandra 1.0.11 If i use memory mapped IO (the default), then the nodes freeze after a while. Paging is disabled. The private bytes are OK (8GB). That is the amount I use in the -Xms and -Xmx arguments. The virtual size is big as expected because of the memory mapped IO. However, the working set size (size in RAM) is 24 GB (my total RAM usage). If I look with Process Explorer to the physical memory section I see a very high value in the WS Sharable section. Anyone has a clue what is going om here? Many thanks! Rene image -- Regards, Oleg Dulin NYC Java Big Data Engineer http://www.olegdulin.com/
Re: High commit size
The problem is that the system just freezes and nodes are dying. The system becomes very unresponsive and it always happens when the shareable amount of RAM reaches the total number of bytes in the system. Is there something in Windows that I can tune in order to avoid this behavior? I cannot easily migrate to Linux right now. Thanks, Rene 2012/9/10 Oleg Dulin oleg.du...@gmail.com It is memory-mapped I/O. I wouldn't worry about it. BTW, Windows might not be the best choice to run Cassandra on. My experience running Cassandra on Windows has not been positive one. We no longer support Windows as our production platform. Regards, Oleg On 2012-09-10 09:00:02 +, Rene Kochen said: Hi all, On my test cluster I have three Windows Server 2008 R2 machines running Cassandra 1.0.11 If i use memory mapped IO (the default), then the nodes freeze after a while. Paging is disabled. The private bytes are OK (8GB). That is the amount I use in the -Xms and -Xmx arguments. The virtual size is big as expected because of the memory mapped IO. However, the working set size (size in RAM) is 24 GB (my total RAM usage). If I look with Process Explorer to the physical memory section I see a very high value in the WS Sharable section. Anyone has a clue what is going om here? Many thanks! Rene image -- Regards, Oleg Dulin NYC Java Big Data Engineer http://www.olegdulin.com/
RE: High commit size
Do not use mmap/auto on Windows, standard access mode only. In cassandra.yaml: disk_access_mode: standard Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.commailto:viktor.jevdoki...@adform.com Phone: +370 5 212 3063, Fax +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Follow us on Twitter: @adforminsiderhttp://twitter.com/#!/adforminsider What is Adform: watch this short videohttp://vimeo.com/adform/display [Adform News] http://www.adform.com Visit us at Dmexco: Hall: 7, Aisle: A, Stand: 030 September 12-13 Cologne, Germany [dmexco] http://www.dmexco.de/ Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. From: Rene Kochen [mailto:rene.koc...@emea.schange.com] Sent: Monday, September 10, 2012 14:47 To: user@cassandra.apache.org Subject: Re: High commit size The problem is that the system just freezes and nodes are dying. The system becomes very unresponsive and it always happens when the shareable amount of RAM reaches the total number of bytes in the system. Is there something in Windows that I can tune in order to avoid this behavior? I cannot easily migrate to Linux right now. Thanks, Rene 2012/9/10 Oleg Dulin oleg.du...@gmail.commailto:oleg.du...@gmail.com It is memory-mapped I/O. I wouldn't worry about it. BTW, Windows might not be the best choice to run Cassandra on. My experience running Cassandra on Windows has not been positive one. We no longer support Windows as our production platform. Regards, Oleg On 2012-09-10 09:00:02 +, Rene Kochen said: Hi all, On my test cluster I have three Windows Server 2008 R2 machines running Cassandra 1.0.11 If i use memory mapped IO (the default), then the nodes freeze after a while. Paging is disabled. The private bytes are OK (8GB). That is the amount I use in the -Xms and -Xmx arguments. The virtual size is big as expected because of the memory mapped IO. However, the working set size (size in RAM) is 24 GB (my total RAM usage). If I look with Process Explorer to the physical memory section I see a very high value in the WS Sharable section. Anyone has a clue what is going om here? Many thanks! Rene image -- Regards, Oleg Dulin NYC Java Big Data Engineer http://www.olegdulin.com/ inline: signature-logoc64.pnginline: dmexco-logofa6.png
Re: High commit size
For performance reasons I switched to memory mapped IO. Is there a way to make memory-mapped IO work in Windows? Thanks! 2012/9/10 Viktor Jevdokimov viktor.jevdoki...@adform.com Do not use mmap/auto on Windows, standard access mode only. In cassandra.yaml: disk_access_mode: standard ** ** ** ** Best regards / Pagarbiai *Viktor Jevdokimov* Senior Developer Email: viktor.jevdoki...@adform.com Phone: +370 5 212 3063, Fax +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Follow us on Twitter: @adforminsider http://twitter.com/#!/adforminsider What is Adform: watch this short video http://vimeo.com/adform/display [image: Adform News] http://www.adform.com * * *Visit us at Dmexco:* Hall: 7, Aisle: A, Stand: 030 September 12-13 Cologne, Germany [image: dmexco] http://www.dmexco.de/ Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. *From:* Rene Kochen [mailto:rene.koc...@emea.schange.com] *Sent:* Monday, September 10, 2012 14:47 *To:* user@cassandra.apache.org *Subject:* Re: High commit size ** ** The problem is that the system just freezes and nodes are dying. The system becomes very unresponsive and it always happens when the shareable amount of RAM reaches the total number of bytes in the system. Is there something in Windows that I can tune in order to avoid this behavior? I cannot easily migrate to Linux right now. Thanks, Rene 2012/9/10 Oleg Dulin oleg.du...@gmail.com It is memory-mapped I/O. I wouldn't worry about it. BTW, Windows might not be the best choice to run Cassandra on. My experience running Cassandra on Windows has not been positive one. We no longer support Windows as our production platform. Regards, Oleg On 2012-09-10 09:00:02 +, Rene Kochen said: Hi all, On my test cluster I have three Windows Server 2008 R2 machines running Cassandra 1.0.11 If i use memory mapped IO (the default), then the nodes freeze after a while. Paging is disabled. The private bytes are OK (8GB). That is the amount I use in the -Xms and -Xmx arguments. The virtual size is big as expected because of the memory mapped IO. However, the working set size (size in RAM) is 24 GB (my total RAM usage). If I look with Process Explorer to the physical memory section I see a very high value in the WS Sharable section. Anyone has a clue what is going om here? Many thanks! Rene image -- Regards, Oleg Dulin NYC Java Big Data Engineer http://www.olegdulin.com/ ** ** signature-logoc64.pngdmexco-logofa6.png
RE: High commit size
We used Cassandra on Windows for more than a year in production for RTB and other staff, that requires lowest possible latency. We used mmap before issues like yours, switched to mmap index only and finally to standard. No big difference in performance, standard was most stable. The huge difference is to run C* on Linux instead of Windows. Migration was pretty easy. Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.commailto:viktor.jevdoki...@adform.com Phone: +370 5 212 3063, Fax +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Follow us on Twitter: @adforminsiderhttp://twitter.com/#!/adforminsider What is Adform: watch this short videohttp://vimeo.com/adform/display [Adform News] http://www.adform.com Visit us at Dmexco: Hall: 7, Aisle: A, Stand: 030 September 12-13 Cologne, Germany [dmexco] http://www.dmexco.de/ Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. From: Rene Kochen [mailto:rene.koc...@emea.schange.com] Sent: Monday, September 10, 2012 17:03 To: user@cassandra.apache.org Subject: Re: High commit size For performance reasons I switched to memory mapped IO. Is there a way to make memory-mapped IO work in Windows? Thanks! 2012/9/10 Viktor Jevdokimov viktor.jevdoki...@adform.commailto:viktor.jevdoki...@adform.com Do not use mmap/auto on Windows, standard access mode only. In cassandra.yaml: disk_access_mode: standard Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.commailto:viktor.jevdoki...@adform.com Phone: +370 5 212 3063tel:%2B370%205%20212%203063, Fax +370 5 261 0453tel:%2B370%205%20261%200453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Follow us on Twitter: @adforminsiderhttp://twitter.com/#!/adforminsider What is Adform: watch this short videohttp://vimeo.com/adform/display [Adform News]http://www.adform.com Visit us at Dmexco: Hall: 7, Aisle: A, Stand: 030 September 12-13 Cologne, Germany [dmexco]http://www.dmexco.de/ Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. From: Rene Kochen [mailto:rene.koc...@emea.schange.commailto:rene.koc...@emea.schange.com] Sent: Monday, September 10, 2012 14:47 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: High commit size The problem is that the system just freezes and nodes are dying. The system becomes very unresponsive and it always happens when the shareable amount of RAM reaches the total number of bytes in the system. Is there something in Windows that I can tune in order to avoid this behavior? I cannot easily migrate to Linux right now. Thanks, Rene 2012/9/10 Oleg Dulin oleg.du...@gmail.commailto:oleg.du...@gmail.com It is memory-mapped I/O. I wouldn't worry about it. BTW, Windows might not be the best choice to run Cassandra on. My experience running Cassandra on Windows has not been positive one. We no longer support Windows as our production platform. Regards, Oleg On 2012-09-10 09:00:02 +, Rene Kochen said: Hi all, On my test cluster I have three Windows Server 2008 R2 machines running Cassandra 1.0.11 If i use memory mapped IO (the default), then the nodes freeze after a while. Paging is disabled. The private bytes are OK (8GB). That is the amount I use in the -Xms and -Xmx arguments. The virtual size is big as expected because of the memory mapped IO. However, the working set size (size in RAM) is 24 GB (my total RAM usage). If I look with Process Explorer to the physical memory section I see a very high value in the WS Sharable section. Anyone has a clue what is going om here? Many thanks! Rene image -- Regards, Oleg Dulin NYC Java Big Data Engineer http://www.olegdulin.com/ inline: image001.pnginline: image002.pnginline: signature-logo5395.pnginline: dmexco-logo634e.png
Re: High commit size
When we ran Cassandra on windows, we got better performance without memory mapped IO. We had the same problems your are describing, what happens is that Windows is rather aggressive about swapping out memory when all the memory is used, and it starts swapping out unused parts of the heap, which causes problems later when it has to swap it in again. We tried a bunch of solutions to limit the mmap memory, to reduce swappiness, etc, but nothing worked. Switching to standard disk access mode worked like a charm though, and performance improved. You still have the disk cache instead of the mmap, so you still use all the RAM of the machine, but you won't run into swapping hell. So, disable mmap, benchmark it, and see if it outperforms running with mmap. I'm pretty sure you will find that it does. /Henrik On Mon, Sep 10, 2012 at 4:02 PM, Rene Kochen rene.koc...@emea.schange.comwrote: For performance reasons I switched to memory mapped IO. Is there a way to make memory-mapped IO work in Windows? Thanks! 2012/9/10 Viktor Jevdokimov viktor.jevdoki...@adform.com Do not use mmap/auto on Windows, standard access mode only. In cassandra.yaml: disk_access_mode: standard ** ** ** ** Best regards / Pagarbiai *Viktor Jevdokimov* Senior Developer Email: viktor.jevdoki...@adform.com Phone: +370 5 212 3063, Fax +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Follow us on Twitter: @adforminsiderhttp://twitter.com/#!/adforminsider What is Adform: watch this short video http://vimeo.com/adform/display [image: Adform News] http://www.adform.com * * *Visit us at Dmexco:* Hall: 7, Aisle: A, Stand: 030 September 12-13 Cologne, Germany [image: dmexco] http://www.dmexco.de/ Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. *From:* Rene Kochen [mailto:rene.koc...@emea.schange.com] *Sent:* Monday, September 10, 2012 14:47 *To:* user@cassandra.apache.org *Subject:* Re: High commit size ** ** The problem is that the system just freezes and nodes are dying. The system becomes very unresponsive and it always happens when the shareable amount of RAM reaches the total number of bytes in the system. Is there something in Windows that I can tune in order to avoid this behavior? I cannot easily migrate to Linux right now. Thanks, Rene 2012/9/10 Oleg Dulin oleg.du...@gmail.com It is memory-mapped I/O. I wouldn't worry about it. BTW, Windows might not be the best choice to run Cassandra on. My experience running Cassandra on Windows has not been positive one. We no longer support Windows as our production platform. Regards, Oleg On 2012-09-10 09:00:02 +, Rene Kochen said: Hi all, On my test cluster I have three Windows Server 2008 R2 machines running Cassandra 1.0.11 If i use memory mapped IO (the default), then the nodes freeze after a while. Paging is disabled. The private bytes are OK (8GB). That is the amount I use in the -Xms and -Xmx arguments. The virtual size is big as expected because of the memory mapped IO. However, the working set size (size in RAM) is 24 GB (my total RAM usage). If I look with Process Explorer to the physical memory section I see a very high value in the WS Sharable section. Anyone has a clue what is going om here? Many thanks! Rene image -- Regards, Oleg Dulin NYC Java Big Data Engineer http://www.olegdulin.com/ ** ** signature-logoc64.pngdmexco-logofa6.png
