[
https://issues.apache.org/jira/browse/CONNECTORS-281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133951#comment-13133951
]
Karl Wright commented on CONNECTORS-281:
----------------------------------------
A thread dump shows all worker threads waiting on database functionality, but
this is interesting. A full 11/30 threads are waiting to RETURN connections to
the pool:
"Worker thread '5'" daemon prio=6 tid=0x055b5c00 nid=0x1c14 waiting for monitor
entry [0x05def000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
com.bitmechanic.sql.ConnectionPool.returnConnection(ConnectionPool.java:474)
- waiting to lock <0x292ad3a0> (a com.bitmechanic.sql.ConnectionPool)
at com.bitmechanic.sql.PooledConnection.close(PooledConnection.java:202)
at
org.apache.manifoldcf.core.database.ConnectionFactory.releaseConnection(ConnectionFactory.java:113)
at
org.apache.manifoldcf.core.database.Database.endTransaction(Database.java:330)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.endTransaction(DBInterfacePostgreSQL.java:1112)
at
org.apache.manifoldcf.core.database.BaseTable.endTransaction(BaseTable.java:274)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.noteDocumentIngest(IncrementalIngester.java:1373)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.performIngestion(IncrementalIngester.java:503)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentRecord(IncrementalIngester.java:325)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.recordDocument(WorkerThread.java:1556)
at
org.apache.manifoldcf.crawler.connectors.rss.RSSConnector.processDocuments(RSSConnector.java:1281)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:561)
It's not clear at all why this should be. The only possible hint is that
there's one thread waiting on GETTING a connection from the pool:
"Worker thread '19'" daemon prio=6 tid=0x045e1400 nid=0x1b94 waiting for
monitor entry [0x0624f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
com.bitmechanic.sql.ConnectionPool.getConnection(ConnectionPool.java:375)
- waiting to lock <0x292ad3a0> (a com.bitmechanic.sql.ConnectionPool)
at
com.bitmechanic.sql.ConnectionPoolManager.connect(ConnectionPoolManager.java:442)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:207)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:144)
at
org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:90)
at
org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:502)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1152)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:860)
at
org.apache.manifoldcf.crawler.jobs.Carrydown.getDataValuesAsFiles(Carrydown.java:603)
at
org.apache.manifoldcf.crawler.jobs.JobManager.retrieveParentDataAsFiles(JobManager.java:4263)
at
org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.retrieveParentDataAsFiles(WorkerThread.java:1211)
at
org.apache.manifoldcf.crawler.connectors.rss.RSSConnector.getDocumentVersions(RSSConnector.java:818)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:318)
This has me wondering if we're seeing a bug in the pool driver. I'll try to
confirm with further stack traces, since this does not explain the high CPU
usage.
> RSS connector takes nearly a second to fetch a document even with no
> throttling
> -------------------------------------------------------------------------------
>
> Key: CONNECTORS-281
> URL: https://issues.apache.org/jira/browse/CONNECTORS-281
> Project: ManifoldCF
> Issue Type: Bug
> Components: RSS connector
> Affects Versions: ManifoldCF 0.4
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.4
>
>
> The RSS connector load test shows that the RSS connector is overthrottling,
> for some reason.
> 10-24-2011 05:30:50.423 fetch
> http://localhost:8189/rss/gen.php?doc=4&feed=782&type=doc
> 200 46 843
> ... Where 843 ms is taken to fetch a document of size 46 bytes. This is with
> connection parameters as follows:
> Parameters: Robots usage=none
> Max fetches per minute=1000000
> Email [email protected]
> KB per second=1000000
> Max server connections=100
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
