[jira] [Commented] (OAK-9112) High CPU IOWAIT on Mongo due to Oak Version Garbage Collection

2020-06-30 Thread David Snowsill (Jira) 


[ 
https://issues.apache.org/jira/browse/OAK-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148379#comment-17148379
 ] 

David Snowsill commented on OAK-9112:
-

Here is a screenshot one of the developers took for me while debugging the issue

 !screenshot-1.png! 

> High CPU IOWAIT on Mongo due to Oak Version Garbage Collection
> --
>
> Key: OAK-9112
> URL: https://issues.apache.org/jira/browse/OAK-9112
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: mongomk
>Affects Versions: 1.8.22
> Environment: Mongo: 3.4.9
> Hosteding: Scalegrid
> Instance Size: Small
>Reporter: David Snowsill
>Priority: Major
> Attachments: image-2020-06-15-19-41-04-852.png, 
> image-2020-06-15-19-41-40-385.png, screenshot-1.png
>
>
> Over the previous weekend, our existing application went from having 100,000 
> documents in the "nodes" collection to 600,000 documents in the "nodes" 
> collection. Since that time there appears to be an Oak background task that 
> commences running near 100% CPU IOWAIT and heavy disk activity (read only). 
> Please refer to charts below
> !image-2020-06-15-19-41-04-852.png!
> !image-2020-06-15-19-41-40-385.png!
> According to currentOp Mongo command, the queries being running look as 
> follows:
> {code}
> { desc: "conn4944", threadId: "140061124310784", connectionId: 4944, client: 
> "18.207.114.70:47008", clientMetadata: \{ driver: { name: 
> "mongo-java-driver", version: "unknown" }, os: \{ type: "Linux", name: 
> "Linux", architecture: "amd64", version: "4.4.115-k8s" }, platform: 
> "Java/IcedTea/1.8.0_212-b04" }, active: true, opid: 2215767, secs_running: 2, 
> microsecs_running: \{ "$numberLong": "2280333" }, op: "query", ns: 
> "prod-content-service.nodes", query: \{ find: "nodes", filter: { _id: 
> "2:/prod/18d6b4d0-9ee8-4a06-804e-b3669f67dda8" }, limit: 1, singleBatch: true 
> }, planSummary: "IDHACK", numYields: 0, locks: \{ Global: "r", Database: "r", 
> Collection: "r" }, waitingForLock: false, lockStats: \{ Global: { 
> acquireCount: { r: { "$numberLong": "2" } } }, Database: \{ acquireCount: { 
> r: { "$numberLong": "1" } } }, Collection: \{ acquireCount: { r: { 
> "$numberLong": "1" } } } } }
> {code}
> and appears to be iterating over the entire collection.
> Can anyone provide any guidance on what is happening?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (OAK-9112) High CPU IOWAIT on Mongo due to Oak Version Garbage Collection

2020-06-30 Thread Marcel Reutegger (Jira) 


[ 
https://issues.apache.org/jira/browse/OAK-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148359#comment-17148359
 ] 

Marcel Reutegger commented on OAK-9112:
---

Thanks for the feedback [~djsnowsill]. I'd still be interested in a thread dump 
if you can share it. The node type should be irrelevant to the revision GC.

> High CPU IOWAIT on Mongo due to Oak Version Garbage Collection
> --
>
> Key: OAK-9112
> URL: https://issues.apache.org/jira/browse/OAK-9112
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: mongomk
>Affects Versions: 1.8.22
> Environment: Mongo: 3.4.9
> Hosteding: Scalegrid
> Instance Size: Small
>Reporter: David Snowsill
>Priority: Major
> Attachments: image-2020-06-15-19-41-04-852.png, 
> image-2020-06-15-19-41-40-385.png
>
>
> Over the previous weekend, our existing application went from having 100,000 
> documents in the "nodes" collection to 600,000 documents in the "nodes" 
> collection. Since that time there appears to be an Oak background task that 
> commences running near 100% CPU IOWAIT and heavy disk activity (read only). 
> Please refer to charts below
> !image-2020-06-15-19-41-04-852.png!
> !image-2020-06-15-19-41-40-385.png!
> According to currentOp Mongo command, the queries being running look as 
> follows:
> {code}
> { desc: "conn4944", threadId: "140061124310784", connectionId: 4944, client: 
> "18.207.114.70:47008", clientMetadata: \{ driver: { name: 
> "mongo-java-driver", version: "unknown" }, os: \{ type: "Linux", name: 
> "Linux", architecture: "amd64", version: "4.4.115-k8s" }, platform: 
> "Java/IcedTea/1.8.0_212-b04" }, active: true, opid: 2215767, secs_running: 2, 
> microsecs_running: \{ "$numberLong": "2280333" }, op: "query", ns: 
> "prod-content-service.nodes", query: \{ find: "nodes", filter: { _id: 
> "2:/prod/18d6b4d0-9ee8-4a06-804e-b3669f67dda8" }, limit: 1, singleBatch: true 
> }, planSummary: "IDHACK", numYields: 0, locks: \{ Global: "r", Database: "r", 
> Collection: "r" }, waitingForLock: false, lockStats: \{ Global: { 
> acquireCount: { r: { "$numberLong": "2" } } }, Database: \{ acquireCount: { 
> r: { "$numberLong": "1" } } }, Collection: \{ acquireCount: { r: { 
> "$numberLong": "1" } } } } }
> {code}
> and appears to be iterating over the entire collection.
> Can anyone provide any guidance on what is happening?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)