Re: PHP - Cassandra integration
I'm using /McFrazier/PhpBinaryCql/ On Mon, Nov 10, 2014 at 1:48 AM, Akshay Ballarpure akshay.ballarp...@tcs.com wrote: Hello, I am working on PHP cassandra integration, please let me know which library is good from scalability and performance perspective ? Best Regards Akshay Ballarpure Tata Consultancy Services Cell:- 9985084075 Mailto: akshay.ballarp...@tcs.com Website: http://www.tcs.com Experience certainty.IT Services Business Solutions Consulting =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you
What actually causing java.lang.OutOfMemoryError: unable to create new native thread
Hello people, below is an extraction from cassandra system log.
ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[Thread-273,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
at
org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)
Investigated into the call until the java native call,
http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l2698
if (native_thread-osthread() == NULL) {
// No one should hold a reference to the 'native_thread'.
delete native_thread;
if (JvmtiExport::should_post_resource_exhausted()) {
JvmtiExport::post_resource_exhausted(
JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR |
JVMTI_RESOURCE_EXHAUSTED_THREADS,
unable to create new native thread);
}
THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
unable to create new native thread);
}
Question. Is that out of memory error due to native os memory or java heap?
Stacked size to the jvm is -Xss128k. Operating system file descriptor max
user processes 26. open files capped at 65536
Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR
and JVMTI_RESOURCE_EXHAUSTED_THREADS means too?
Thank you.
Jason
Increase in dropped mutations after major upgrade from 1.2.18 to 2.0.10
Hey, We've seen a considerable increase in the number of dropped mutations after a major upgrade from 1.2.18 to 2.0.10. I initially thought it was due to the extra load incurred by upgradesstables, but the dropped mutations continue even after all sstables are upgraded. Additional info: Overall (read, write and range) latency improved with the upgrade, which is great, but I don't understand why dropped mutations has increased. I/O and CPU load is pretty much the same, number of completed tasks is the only metric that increased together with dropped mutations. I also noticed that the number of all time blocked FlushWriter operations is about 5% of completed operations, don't know if this is related, but in case it helps out... Anyone has a clue on what could that be? Or what should we monitor to find out? Any help or JIRA pointers would be kindly appreciated. Cheers, -- *Paulo Motta* Chaordic | *Platform* *www.chaordic.com.br http://www.chaordic.com.br/* +55 48 3232.3200
Re: Increase in dropped mutations after major upgrade from 1.2.18 to 2.0.10
Hi Paulo, On 10/11/14 15:18, Paulo Ricardo Motta Gomes wrote: Hey, We've seen a considerable increase in the number of dropped mutations after a major upgrade from 1.2.18 to 2.0.10. I initially thought it was due to the extra load incurred by upgradesstables, but the dropped mutations continue even after all sstables are upgraded. are the clocks on all your nodes synchronized with each other? Ciao, Duncan. Additional info: Overall (read, write and range) latency improved with the upgrade, which is great, but I don't understand why dropped mutations has increased. I/O and CPU load is pretty much the same, number of completed tasks is the only metric that increased together with dropped mutations. I also noticed that the number of all time blocked FlushWriter operations is about 5% of completed operations, don't know if this is related, but in case it helps out... Anyone has a clue on what could that be? Or what should we monitor to find out? Any help or JIRA pointers would be kindly appreciated. Cheers, -- *Paulo Motta* Chaordic | /Platform/ _www.chaordic.com.br http://www.chaordic.com.br/_ +55 48 3232.3200
Re: Increase in dropped mutations after major upgrade from 1.2.18 to 2.0.10
On Mon, Nov 10, 2014 at 12:46 PM, Duncan Sands duncan.sa...@gmail.com wrote: Hi Paulo, On 10/11/14 15:18, Paulo Ricardo Motta Gomes wrote: Hey, We've seen a considerable increase in the number of dropped mutations after a major upgrade from 1.2.18 to 2.0.10. I initially thought it was due to the extra load incurred by upgradesstables, but the dropped mutations continue even after all sstables are upgraded. are the clocks on all your nodes synchronized with each other? Ciao, Duncan. Yes, the servers are synchronized via NTP. Cheers! Additional info: Overall (read, write and range) latency improved with the upgrade, which is great, but I don't understand why dropped mutations has increased. I/O and CPU load is pretty much the same, number of completed tasks is the only metric that increased together with dropped mutations. I also noticed that the number of all time blocked FlushWriter operations is about 5% of completed operations, don't know if this is related, but in case it helps out... Anyone has a clue on what could that be? Or what should we monitor to find out? Any help or JIRA pointers would be kindly appreciated. Cheers, -- *Paulo Motta* Chaordic | /Platform/ _www.chaordic.com.br http://www.chaordic.com.br/_ +55 48 3232.3200 -- *Paulo Motta* Chaordic | *Platform* *www.chaordic.com.br http://www.chaordic.com.br/* +55 48 3232.3200
Re: query tracing
Be cautious enabling query tracing. Great tool for dev/testing/diagnosing etc.. - but it does persist data to the system_traces keyspace with a TTL of 24 hours and will, as a consequence, consume resources. http://www.datastax.com/dev/blog/advanced-request-tracing-in-cassandra-1-2 http://www.datastax.com/dev/blog/advanced-request-tracing-in-cassandra-1-2 On 7 Nov 2014, at 20:20, Jonathan Haddad j...@jonhaddad.com wrote: Personally I've found that using query timing + log aggregation on the client side is more effective than trying to mess with tracing probability in order to find a single query which has recently become a problem. I recommend wrapping your session with something that can automatically log the statement on a slow query, then use tracing to identify exactly what happened. This way finding your problem is not a matter of chance. On Fri Nov 07 2014 at 9:41:38 AM Chris Lohfink clohfin...@gmail.com mailto:clohfin...@gmail.com wrote: It saves a lot of information for each request thats traced so there is significant overhead. If you start at a low probability and move it up based on the load impact it will provide a lot of insight and you can control the cost. --- Chris Lohfink On Fri, Nov 7, 2014 at 11:35 AM, Jimmy Lin y2klyf+w...@gmail.com mailto:y2klyf+w...@gmail.com wrote: is there any significant performance penalty if one turn on Cassandra query tracing, through DataStax java driver (say, per every query request of some trouble query)? More sampling seems better but then doing so may also slow down the system in some other ways? thanks
Re: query tracing
As Jonathan said, it's better to activate query tracing client side. It'll give you better flexibility of when to turn on off tracing and on which table. Server-side tracing is global (all tables) and probabilistic, thus may not give satisfactory level of debugging. Programmatically it's pretty simple to achieve and coupled with a good logging framework (LogBack for Java), you'll even have dynamic logging on production without having to redeploy client code. I have implemented it in Achilles very easily by wrapping over the Regular/Bound/Simple statements of Java driver and display the bound values at runtime : https://github.com/doanduyhai/Achilles/wiki/Statements-Logging-and-Tracing#dynamic-statements-logging On Mon, Nov 10, 2014 at 3:52 PM, Johnny Miller johnny.p.mil...@gmail.com wrote: Be cautious enabling query tracing. Great tool for dev/testing/diagnosing etc.. - but it does persist data to the system_traces keyspace with a TTL of 24 hours and will, as a consequence, consume resources. http://www.datastax.com/dev/blog/advanced-request-tracing-in-cassandra-1-2 On 7 Nov 2014, at 20:20, Jonathan Haddad j...@jonhaddad.com wrote: Personally I've found that using query timing + log aggregation on the client side is more effective than trying to mess with tracing probability in order to find a single query which has recently become a problem. I recommend wrapping your session with something that can automatically log the statement on a slow query, then use tracing to identify exactly what happened. This way finding your problem is not a matter of chance. On Fri Nov 07 2014 at 9:41:38 AM Chris Lohfink clohfin...@gmail.com wrote: It saves a lot of information for each request thats traced so there is significant overhead. If you start at a low probability and move it up based on the load impact it will provide a lot of insight and you can control the cost. --- Chris Lohfink On Fri, Nov 7, 2014 at 11:35 AM, Jimmy Lin y2klyf+w...@gmail.com wrote: is there any significant performance penalty if one turn on Cassandra query tracing, through DataStax java driver (say, per every query request of some trouble query)? More sampling seems better but then doing so may also slow down the system in some other ways? thanks
Re: PHP - Cassandra integration
Hi Akshay, Please let me know if you have any issues using PhpBinaryCql, and submit pull requests if you fix any bugs. Thanks, Robert McFrazier On Mon, Nov 10, 2014 at 6:04 AM, Spencer Brown lilspe...@gmail.com wrote: I'm using /McFrazier/PhpBinaryCql/ On Mon, Nov 10, 2014 at 1:48 AM, Akshay Ballarpure akshay.ballarp...@tcs.com wrote: Hello, I am working on PHP cassandra integration, please let me know which library is good from scalability and performance perspective ? Best Regards Akshay Ballarpure Tata Consultancy Services Cell:- 9985084075 Mailto: akshay.ballarp...@tcs.com Website: http://www.tcs.com Experience certainty.IT Services Business Solutions Consulting =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -- Thanks, Robert McFrazier
[RELEASE] Apache Cassandra 2.1.2 released
The Cassandra team is pleased to announce the release of Apache Cassandra version 2.1.2. Apache Cassandra is a fully distributed database. It is the right choice when you need scalability and high availability without compromising performance. http://cassandra.apache.org/ Downloads of source and binary distributions are listed in our download section: http://cassandra.apache.org/download/ This version is a bug fix release[1] on the 2.1 series. As always, please pay attention to the release notes[2] and Let us know[3] if you were to encounter any problems. Enjoy! [1]: http://goo.gl/pi45XF (CHANGES.txt) [2]: http://goo.gl/vtSXzZ (NEWS.txt) [3]: https://issues.apache.org/jira/browse/CASSANDRA
Re: What actually causing java.lang.OutOfMemoryError: unable to create new native thread
First question are you running 32bit or 64bit… on 32bit you can easily run out
of virtual address space for thread stacks.
On Nov 10, 2014, at 8:25 AM, Jason Wee peich...@gmail.com wrote:
Hello people, below is an extraction from cassandra system log.
ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java (line
139) Fatal exception in thread Thread[Thread-273,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
at
org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)
Investigated into the call until the java native call,
http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l2698
http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l2698
if (native_thread-osthread() == NULL) {
// No one should hold a reference to the 'native_thread'.
delete native_thread;
if (JvmtiExport::should_post_resource_exhausted()) {
JvmtiExport::post_resource_exhausted(
JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR | JVMTI_RESOURCE_EXHAUSTED_THREADS,
unable to create new native thread);
}
THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
unable to create new native thread);
}
Question. Is that out of memory error due to native os memory or java heap?
Stacked size to the jvm is -Xss128k. Operating system file descriptor max
user processes 26. open files capped at 65536
Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR and
JVMTI_RESOURCE_EXHAUSTED_THREADS means too?
Thank you.
Jason
smime.p7s
Description: S/MIME cryptographic signature
Re: What actually causing java.lang.OutOfMemoryError: unable to create new native thread
if your using 64 bit, check output of:
cat /proc/{cassandra pid}/limits
some older linux kernels wont work with above so if it doesnt exist check
the ulimit -a output for the cassandra user. max processes per user may be
your issue as well.
---
Chris Lohfink
On Mon, Nov 10, 2014 at 11:21 AM, graham sanderson gra...@vast.com wrote:
First question are you running 32bit or 64bit… on 32bit you can easily run
out of virtual address space for thread stacks.
On Nov 10, 2014, at 8:25 AM, Jason Wee peich...@gmail.com wrote:
Hello people, below is an extraction from cassandra system log.
ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[Thread-273,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
at
org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)
Investigated into the call until the java native call,
http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l2698
if (native_thread-osthread() == NULL) {
// No one should hold a reference to the 'native_thread'.
delete native_thread;
if (JvmtiExport::should_post_resource_exhausted()) {
JvmtiExport::post_resource_exhausted(
JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR |
JVMTI_RESOURCE_EXHAUSTED_THREADS,
unable to create new native thread);
}
THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
unable to create new native thread);
}
Question. Is that out of memory error due to native os memory or java
heap? Stacked size to the jvm is -Xss128k. Operating system file descriptor
max user processes 26. open files capped at 65536
Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR
and JVMTI_RESOURCE_EXHAUSTED_THREADS means too?
Thank you.
Jason
Re: What actually causing java.lang.OutOfMemoryError: unable to create new native thread
Hi, thank you for response.
using 64bit and kernel 2.6.32-358.18.1.el6.x86_64.
# cat /proc/13405/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimitedunlimitedseconds
Max file size unlimitedunlimitedbytes
Max data size unlimitedunlimitedbytes
Max stack size10485760 unlimitedbytes
Max core file size0unlimitedbytes
Max resident set unlimitedunlimitedbytes
Max processes 26 26
processes
Max open files6553665536files
Max locked memory 6553665536bytes
Max address space unlimitedunlimitedbytes
Max file locksunlimitedunlimitedlocks
Max pending signals 255762 255762 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 00
Max realtime priority 00
Max realtime timeout unlimitedunlimitedus
Is the stack size and / or Max open files is sufficient? The rest fd pretty
much infinite.
Jason
On Tue, Nov 11, 2014 at 4:09 AM, Chris Lohfink clohfin...@gmail.com wrote:
if your using 64 bit, check output of:
cat /proc/{cassandra pid}/limits
some older linux kernels wont work with above so if it doesnt exist check
the ulimit -a output for the cassandra user. max processes per user may
be your issue as well.
---
Chris Lohfink
On Mon, Nov 10, 2014 at 11:21 AM, graham sanderson gra...@vast.com
wrote:
First question are you running 32bit or 64bit… on 32bit you can easily
run out of virtual address space for thread stacks.
On Nov 10, 2014, at 8:25 AM, Jason Wee peich...@gmail.com wrote:
Hello people, below is an extraction from cassandra system log.
ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[Thread-273,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
at
org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)
Investigated into the call until the java native call,
http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l2698
if (native_thread-osthread() == NULL) {
// No one should hold a reference to the 'native_thread'.
delete native_thread;
if (JvmtiExport::should_post_resource_exhausted()) {
JvmtiExport::post_resource_exhausted(
JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR |
JVMTI_RESOURCE_EXHAUSTED_THREADS,
unable to create new native thread);
}
THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
unable to create new native thread);
}
Question. Is that out of memory error due to native os memory or java
heap? Stacked size to the jvm is -Xss128k. Operating system file descriptor
max user processes 26. open files capped at 65536
Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR
and JVMTI_RESOURCE_EXHAUSTED_THREADS means too?
Thank you.
Jason
