[jira] [Commented] (CASSANDRA-8518) Cassandra Query Request Size Estimator
[ https://issues.apache.org/jira/browse/CASSANDRA-8518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277953#comment-14277953 ] Benedict commented on CASSANDRA-8518: - This is one of the two methods I proposed, and I'm comfortable aiming for the global threshold. Per-request thresholds are also a possibility, and seem reasonable also. Whether or not we _throttle_ or simply discard some in-flight queries on exceeding our limit is another matter though. I would prefer to go the route of discarding some random in-flight queries, as this brings the system back to full health immediately, instead of letting it crawl along until the blockage clears. > Cassandra Query Request Size Estimator > -- > > Key: CASSANDRA-8518 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8518 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Cheng Ren > > We have been suffering from cassandra node crash due to out of memory for a > long time. The heap dump from the recent crash shows there are 22 native > transport request threads each of which consumes 3.3% of heap size, taking > more than 70% in total. > Heap dump: > !https://dl-web.dropbox.com/get/attach1.png?_subject_uid=303980955&w=AAAVOoncBoZ5aOPbDg2TpRkUss7B-2wlrnhUAv19b27OUA|height=400,width=600! > Expanded view of one thread: > !https://dl-web.dropbox.com/get/Screen%20Shot%202014-12-18%20at%204.06.29%20PM.png?_subject_uid=303980955&w=AACUO4wrbxheRUxv8fwQ9P52T6gBOm5_g9zeIe8odu3V3w|height=400,width=600! > The cassandra we are using now (2.0.4) utilized MemoryAwareThreadPoolExecutor > as the request executor and provided a default request size estimator which > constantly returns 1, meaning it limits only the number of requests being > pushed to the pool. To have more fine-grained control on handling requests > and better protect our node from OOM issue, we propose implementing a more > precise estimator. > Here is our two cents: > For update/delete/insert request: Size could be estimated by adding size of > all class members together. > For scan query, the major part of the request is response, which can be > estimated from the history data. For example if we receive a scan query on a > column family for a certain token range, we keep track of its response size > used as the estimated response size for later scan query on the same cf. > For future requests on the same cf, response size could be calculated by > token range*recorded size/ recorded token range. The request size should be > estimated as (query size + estimated response size). > We believe what we're proposing here can be useful for other people in the > Cassandra community as well. Would you mind providing us feedbacks? Please > let us know if you have any concerns or suggestions regarding this proposal. > Thanks, > Cheng -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8518) Cassandra Query Request Size Estimator
[ https://issues.apache.org/jira/browse/CASSANDRA-8518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277942#comment-14277942 ] Cheng Ren commented on CASSANDRA-8518: -- Thanks so much for your helpful reply, we believe your approach which coordinates all queries and kill queries if we exceed an limit would be the potential solution for us. In our understanding, we will have a global size counter keeping track of the total size of the current queries being processed. Query serving is a multiple-stage process (read->response->repair). Each stage will add to the total size. Once total size exceeds the limit we need to throttle future queries until it goes below the threshold. We could have this threshold set from cassandra yaml file and reset via nodetool in the runtime so that users could customize based on their available memory resource and SLA(if the threshold is over small, the query latency will increase) So is our understanding correct? We would like to hear your feedback. Thanks > Cassandra Query Request Size Estimator > -- > > Key: CASSANDRA-8518 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8518 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Cheng Ren > > We have been suffering from cassandra node crash due to out of memory for a > long time. The heap dump from the recent crash shows there are 22 native > transport request threads each of which consumes 3.3% of heap size, taking > more than 70% in total. > Heap dump: > !https://dl-web.dropbox.com/get/attach1.png?_subject_uid=303980955&w=AAAVOoncBoZ5aOPbDg2TpRkUss7B-2wlrnhUAv19b27OUA|height=400,width=600! > Expanded view of one thread: > !https://dl-web.dropbox.com/get/Screen%20Shot%202014-12-18%20at%204.06.29%20PM.png?_subject_uid=303980955&w=AACUO4wrbxheRUxv8fwQ9P52T6gBOm5_g9zeIe8odu3V3w|height=400,width=600! > The cassandra we are using now (2.0.4) utilized MemoryAwareThreadPoolExecutor > as the request executor and provided a default request size estimator which > constantly returns 1, meaning it limits only the number of requests being > pushed to the pool. To have more fine-grained control on handling requests > and better protect our node from OOM issue, we propose implementing a more > precise estimator. > Here is our two cents: > For update/delete/insert request: Size could be estimated by adding size of > all class members together. > For scan query, the major part of the request is response, which can be > estimated from the history data. For example if we receive a scan query on a > column family for a certain token range, we keep track of its response size > used as the estimated response size for later scan query on the same cf. > For future requests on the same cf, response size could be calculated by > token range*recorded size/ recorded token range. The request size should be > estimated as (query size + estimated response size). > We believe what we're proposing here can be useful for other people in the > Cassandra community as well. Would you mind providing us feedbacks? Please > let us know if you have any concerns or suggestions regarding this proposal. > Thanks, > Cheng -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8518) Cassandra Query Request Size Estimator
[ https://issues.apache.org/jira/browse/CASSANDRA-8518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14268054#comment-14268054 ] Benedict commented on CASSANDRA-8518: - I agree it would be great to explicitly constrain the amount of in-flight data. In many ways this is a duplicate of CASSANDRA-5039, although it is subtly differently framed. The problem with the extra goal here, though, is predicting how much heap will be consumed by a request is very difficult. Ultimately any prediction scheme is unlikely to be substantially better than simply tuning the number of in-flight requests on the system. We could use Count-Min sketches on partitions, say, but this is getting into complex, ugly territory for something that hopefully has a short shelf life, and won't likely work well at all for non-local reads. In my opinion the "right" solution is to implement streaming reads, so that we can explicitly bound the amount of in-flight data per request. Short of that, we could in the mean time impose an explicit _bound_ on memory used per-query, and terminate a query we detect to have gone above this bound, so that if you tune the number in flight you can guarantee the heap won't absolutely explode. We could even have all queries coordinate lightly and if we exceed an "in-flight" limit, kill one.. This could also be enforced by MessagingService as we read data off the wire, so that it never pollutes the system (other schemes would be hard to include MS in, I think). I would rather a "best effort" than exact bound, so that we could implement it easily, and I would probably stick to the simple per-request limit, as a stop-gap until we get streaming reads. > Cassandra Query Request Size Estimator > -- > > Key: CASSANDRA-8518 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8518 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Cheng Ren > > We have been suffering from cassandra node crash due to out of memory for a > long time. The heap dump from the recent crash shows there are 22 native > transport request threads each of which consumes 3.3% of heap size, taking > more than 70% in total. > Heap dump: > !https://dl-web.dropbox.com/get/attach1.png?_subject_uid=303980955&w=AAAVOoncBoZ5aOPbDg2TpRkUss7B-2wlrnhUAv19b27OUA|height=400,width=600! > Expanded view of one thread: > !https://dl-web.dropbox.com/get/Screen%20Shot%202014-12-18%20at%204.06.29%20PM.png?_subject_uid=303980955&w=AACUO4wrbxheRUxv8fwQ9P52T6gBOm5_g9zeIe8odu3V3w|height=400,width=600! > The cassandra we are using now (2.0.4) utilized MemoryAwareThreadPoolExecutor > as the request executor and provided a default request size estimator which > constantly returns 1, meaning it limits only the number of requests being > pushed to the pool. To have more fine-grained control on handling requests > and better protect our node from OOM issue, we propose implementing a more > precise estimator. > Here is our two cents: > For update/delete/insert request: Size could be estimated by adding size of > all class members together. > For scan query, the major part of the request is response, which can be > estimated from the history data. For example if we receive a scan query on a > column family for a certain token range, we keep track of its response size > used as the estimated response size for later scan query on the same cf. > For future requests on the same cf, response size could be calculated by > token range*recorded size/ recorded token range. The request size should be > estimated as (query size + estimated response size). > We believe what we're proposing here can be useful for other people in the > Cassandra community as well. Would you mind providing us feedbacks? Please > let us know if you have any concerns or suggestions regarding this proposal. > Thanks, > Cheng -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8518) Cassandra Query Request Size Estimator
[ https://issues.apache.org/jira/browse/CASSANDRA-8518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266926#comment-14266926 ] Jonathan Ellis commented on CASSANDRA-8518: --- WDYT [~benedict]? > Cassandra Query Request Size Estimator > -- > > Key: CASSANDRA-8518 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8518 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Cheng Ren > > We have been suffering from cassandra node crash due to out of memory for a > long time. The heap dump from the recent crash shows there are 22 native > transport request threads each of which consumes 3.3% of heap size, taking > more than 70% in total. > Heap dump: > !https://dl-web.dropbox.com/get/attach1.png?_subject_uid=303980955&w=AAAVOoncBoZ5aOPbDg2TpRkUss7B-2wlrnhUAv19b27OUA|height=400,width=600! > Expanded view of one thread: > !https://dl-web.dropbox.com/get/Screen%20Shot%202014-12-18%20at%204.06.29%20PM.png?_subject_uid=303980955&w=AACUO4wrbxheRUxv8fwQ9P52T6gBOm5_g9zeIe8odu3V3w|height=400,width=600! > The cassandra we are using now (2.0.4) utilized MemoryAwareThreadPoolExecutor > as the request executor and provided a default request size estimator which > constantly returns 1, meaning it limits only the number of requests being > pushed to the pool. To have more fine-grained control on handling requests > and better protect our node from OOM issue, we propose implementing a more > precise estimator. > Here is our two cents: > For update/delete/insert request: Size could be estimated by adding size of > all class members together. > For scan query, the major part of the request is response, which can be > estimated from the history data. For example if we receive a scan query on a > column family for a certain token range, we keep track of its response size > used as the estimated response size for later scan query on the same cf. > For future requests on the same cf, response size could be calculated by > token range*recorded size/ recorded token range. The request size should be > estimated as (query size + estimated response size). > We believe what we're proposing here can be useful for other people in the > Cassandra community as well. Would you mind providing us feedbacks? Please > let us know if you have any concerns or suggestions regarding this proposal. > Thanks, > Cheng -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8518) Cassandra Query Request Size Estimator
[ https://issues.apache.org/jira/browse/CASSANDRA-8518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14257606#comment-14257606 ] Cheng Ren commented on CASSANDRA-8518: -- Thanks for your reply. The feature you suggested is very useful, and we will definitely consider it in future upgrade. But as you mentioned in CASSANDRA-7402, it's not going to solve the underlying OOM issue caused by large response data of requests. The feature we are eager to have should help to prevent OOM in system itself. What we are proposing here is to take advantage of existing code and build a better memory estimator so that system can throttle requests. Please let us know If there is any other related issue. We also would like to hear your feedback on our proposal. Thanks > Cassandra Query Request Size Estimator > -- > > Key: CASSANDRA-8518 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8518 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Cheng Ren > > We have been suffering from cassandra node crash due to out of memory for a > long time. The heap dump from the recent crash shows there are 22 native > transport request threads each of which consumes 3.3% of heap size, taking > more than 70% in total. > Heap dump: > !https://dl-web.dropbox.com/get/attach1.png?_subject_uid=303980955&w=AAAVOoncBoZ5aOPbDg2TpRkUss7B-2wlrnhUAv19b27OUA|height=400,width=600! > Expanded view of one thread: > !https://dl-web.dropbox.com/get/Screen%20Shot%202014-12-18%20at%204.06.29%20PM.png?_subject_uid=303980955&w=AACUO4wrbxheRUxv8fwQ9P52T6gBOm5_g9zeIe8odu3V3w|height=400,width=600! > The cassandra we are using now (2.0.4) utilized MemoryAwareThreadPoolExecutor > as the request executor and provided a default request size estimator which > constantly returns 1, meaning it limits only the number of requests being > pushed to the pool. To have more fine-grained control on handling requests > and better protect our node from OOM issue, we propose implementing a more > precise estimator. > Here is our two cents: > For update/delete/insert request: Size could be estimated by adding size of > all class members together. > For scan query, the major part of the request is response, which can be > estimated from the history data. For example if we receive a scan query on a > column family for a certain token range, we keep track of its response size > used as the estimated response size for later scan query on the same cf. > For future requests on the same cf, response size could be calculated by > token range*recorded size/ recorded token range. The request size should be > estimated as (query size + estimated response size). > We believe what we're proposing here can be useful for other people in the > Cassandra community as well. Would you mind providing us feedbacks? Please > let us know if you have any concerns or suggestions regarding this proposal. > Thanks, > Cheng -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8518) Cassandra Query Request Size Estimator
[ https://issues.apache.org/jira/browse/CASSANDRA-8518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14257123#comment-14257123 ] T Jake Luciani commented on CASSANDRA-8518: --- Is this a duplicate of CASSANDRA-7402 ? > Cassandra Query Request Size Estimator > -- > > Key: CASSANDRA-8518 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8518 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Cheng Ren > > We have been suffering from cassandra node crash due to out of memory for a > long time. The heap dump from the recent crash shows there are 22 native > transport request threads each of which consumes 3.3% of heap size, taking > more than 70% in total. > Heap dump: > !https://dl-web.dropbox.com/get/attach1.png?_subject_uid=303980955&w=AAAVOoncBoZ5aOPbDg2TpRkUss7B-2wlrnhUAv19b27OUA|height=400,width=600! > Expanded view of one thread: > !https://dl-web.dropbox.com/get/Screen%20Shot%202014-12-18%20at%204.06.29%20PM.png?_subject_uid=303980955&w=AACUO4wrbxheRUxv8fwQ9P52T6gBOm5_g9zeIe8odu3V3w|height=400,width=600! > The cassandra we are using now (2.0.4) utilized MemoryAwareThreadPoolExecutor > as the request executor and provided a default request size estimator which > constantly returns 1, meaning it limits only the number of requests being > pushed to the pool. To have more fine-grained control on handling requests > and better protect our node from OOM issue, we propose implementing a more > precise estimator. > Here is our two cents: > For update/delete/insert request: Size could be estimated by adding size of > all class members together. > For scan query, the major part of the request is response, which can be > estimated from the history data. For example if we receive a scan query on a > column family for a certain token range, we keep track of its response size > used as the estimated response size for later scan query on the same cf. > For future requests on the same cf, response size could be calculated by > token range*recorded size/ recorded token range. The request size should be > estimated as (query size + estimated response size). > We believe what we're proposing here can be useful for other people in the > Cassandra community as well. Would you mind providing us feedbacks? Please > let us know if you have any concerns or suggestions regarding this proposal. > Thanks, > Cheng -- This message was sent by Atlassian JIRA (v6.3.4#6332)
