[
https://issues.apache.org/jira/browse/OAK-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136099#comment-17136099
]
David Snowsill edited comment on OAK-9112 at 6/15/20, 6:46 PM:
---------------------------------------------------------------
Hi Marcel, thanks for your help.
I can confirm that over the period the background process was running it was
iterating through the nodes collection performing queries such as:
{code}
{ desc: "conn1069", threadId: "140061220099840", connectionId: 1069, client:
"18.207.114.70:53082", clientMetadata: { driver: { name: "mongo-java-driver",
version: "unknown" }, os: { type: "Linux", name: "Linux", architecture:
"amd64", version: "4.4.115-k8s" }, platform: "Java/IcedTea/1.8.0_212-b04" },
active: true, opid: 390894, secs_running: 0, microsecs_running: {
"$numberLong": "187909" }, op: "query", ns: "prod-content-service.nodes",
query: { find: "nodes", filter: { _id:
"2:/prod/c79fc74d-99b8-45b2-a83e-961158cb64d9" }, limit: 1, singleBatch: true
}, planSummary: "IDHACK", numYields: 0, locks: { Global: "r", Database: "r",
Collection: "r" }, waitingForLock: false, lockStats: { Global: { acquireCount:
{ r: { "$numberLong": "2" } } }, Database: { acquireCount: { r: {
"$numberLong": "1" } } }, Collection: { acquireCount: { r: { "$numberLong": "1"
} } } } }
{code}
with the only parameter changing in the query being the "_id"
The next time it runs we will try to profile it and provide a stack trace
Regards,
David
was (Author: djsnowsill):
Hi Marcel, thanks for your help.
I can confirm that over the period the background process was running it was
iterating through the nodes collection performing queries such as:
{code:java}
{ desc: "conn1069", threadId: "140061220099840", connectionId: 1069, client:
"18.207.114.70:53082", clientMetadata: { driver: { name: "mongo-java-driver",
version: "unknown" }, os: { type: "Linux", name: "Linux", architecture:
"amd64", version: "4.4.115-k8s" }, platform: "Java/IcedTea/1.8.0_212-b04" },
active: true, opid: 390894, secs_running: 0, microsecs_running: {
"$numberLong": "187909" }, op: "query", ns: "prod-content-service.nodes",
query: { find: "nodes", filter: { _id:
"2:/prod/c79fc74d-99b8-45b2-a83e-961158cb64d9" }, limit: 1, singleBatch: true
}, planSummary: "IDHACK", numYields: 0, locks: { Global: "r", Database: "r",
Collection: "r" }, waitingForLock: false, lockStats: { Global: { acquireCount:
{ r: { "$numberLong": "2" } } }, Database: { acquireCount: { r: {
"$numberLong": "1" } } }, Collection: { acquireCount: { r: { "$numberLong": "1"
} } } } }
{code:java}
with the only parameter changing in the query being the "_id"
The next time it runs we will try to profile it and provide a stack trace
Regards,
David
> High CPU IOWAIT on Mongo due to Oak background task
> ---------------------------------------------------
>
> Key: OAK-9112
> URL: https://issues.apache.org/jira/browse/OAK-9112
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: mongomk
> Affects Versions: 1.8.22
> Environment: Mongo: 3.4.9
> Hosteding: Scalegrid
> Instance Size: Small
> Reporter: David Snowsill
> Priority: Major
> Fix For: 1.8.22
>
> Attachments: image-2020-06-15-19-41-04-852.png,
> image-2020-06-15-19-41-40-385.png
>
>
> Over the previous weekend, our existing application went from having 100,000
> documents in the "nodes" collection to 600,000 documents in the "nodes"
> collection. Since that time there appears to be an Oak background task that
> commences running near 100% CPU IOWAIT and heavy disk activity (read only).
> Please refer to charts below
> !image-2020-06-15-19-41-04-852.png!
> !image-2020-06-15-19-41-40-385.png!
> According to currentOp Mongo command, the queries being running look as
> follows:
> { desc: "conn4944", threadId: "140061124310784", connectionId: 4944, client:
> "18.207.114.70:47008", clientMetadata: \{ driver: { name:
> "mongo-java-driver", version: "unknown" }, os: \{ type: "Linux", name:
> "Linux", architecture: "amd64", version: "4.4.115-k8s" }, platform:
> "Java/IcedTea/1.8.0_212-b04" }, active: true, opid: 2215767, secs_running: 2,
> microsecs_running: \{ "$numberLong": "2280333" }, op: "query", ns:
> "prod-content-service.nodes", query: \{ find: "nodes", filter: { _id:
> "2:/prod/18d6b4d0-9ee8-4a06-804e-b3669f67dda8" }, limit: 1, singleBatch: true
> }, planSummary: "IDHACK", numYields: 0, locks: \{ Global: "r", Database: "r",
> Collection: "r" }, waitingForLock: false, lockStats: \{ Global: {
> acquireCount: { r: { "$numberLong": "2" } } }, Database: \{ acquireCount: {
> r: { "$numberLong": "1" } } }, Collection: \{ acquireCount: { r: {
> "$numberLong": "1" } } } } }
> and appears to be iterating over the entire collection.
> Can anyone provide any guidance on what is happening?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
