Hello, we recently updated our OC installation from 5.0.14 to 5.0.17. Since then we noticed a lot of new failures in our owncloud.log. Those errors are also visible in / propagated to the users sync client. So the user gets a sync problem notification and scaring error messages in the activity log. I did not see those errors for browser based access.
There are two categories of failures, Deadlocks and "Duplicate entries for
key“. I’d already logged a bit more verbose … sorry but this will take some
lines. User- and filenames are replaced.
Case 1: Duplicate Entries in oc_permissions
####################################
owncloud-06 - [Aug 4 13:48:28] {webdav} no permissions found, fileid:
113065191 user:userA
owncloud-06 - [Aug 4 13:48:28] {webdav} no permissions found, fileid:
113065191 user:userA
owncloud-06 - [Aug 4 13:48:28] {webdav} no permissions found, fileid:
113065191 user:userA
owncloud-06 - [Aug 4 13:48:28] {webdav} no permissions found, fileid:
113065191 user:userA
=> those lines come from lib/file/cache/permissions.php
public function get($fileId, $user)
There I added an errorlog in case no permissions where found. This results in
an Insert statement rather than an Update statement on oc_permissions.
owncloud-06 - [Aug 4 13:48:28] {webdav} PDOException: SQLSTATE[23000]:
Integrity constraint violation: 1062 Duplicate entry '113065191-userA' for key
'PRIMARY' [23000]
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #1
/WEBROOT/lib/files/cache/permissions.php(61):
PDOStatementWrapper->execute(Array)
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #2
/WEBROOT/lib/files/view.php(811): OC\Files\Cache\Permissions->set(113065191,
‚user…‘A, 31)
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #3
/WEBROOT/lib/files/filesystem.php(640): OC\Files\View->getFileInfo('/file1...',
true)
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #4
/WEBROOT/lib/connector/sabre/directory.php(145):
OC\Files\Filesystem::getFileInfo(‚file1…‘)
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #5
/WEBROOT/3rdparty/Sabre/DAV/ObjectTree.php(64):
OC_Connector_Sabre_Directory->getChild('file1...')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #6
/WEBROOT/3rdparty/Sabre/DAV/ObjectTree.php(67):
Sabre_DAV_ObjectTree->getNodeForPath('/file1...')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #7
/WEBROOT/3rdparty/Sabre/DAV/ObjectTree.php(67):
Sabre_DAV_ObjectTree->getNodeForPath('/file1...')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #8
/WEBROOT/3rdparty/Sabre/DAV/ObjectTree.php(96):
Sabre_DAV_ObjectTree->getNodeForPath('/file1...')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #9
/WEBROOT/3rdparty/Sabre/DAV/Server.php(874):
Sabre_DAV_ObjectTree->nodeExists('/file1...')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #10 [internal function]:
Sabre_DAV_Server->httpPut('/file1...')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #11
/WEBROOT/3rdparty/Sabre/DAV/Server.php(473): call_user_func(Array, 'file1...')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #12
/WEBROOT/3rdparty/Sabre/DAV/Server.php(213):
Sabre_DAV_Server->invokeMethod('PUT', 'file1...')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #13
/WEBROOT/apps/files/appinfo/remote.php(56): Sabre_DAV_Server->exec()
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #14
/WEBROOT/remote.php(40): require_once(‚/WEBROOT/...')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #15 {main}
=> Although there was no entry found in the previous check, the database moans
about an duplicate entry for PRIMARY key in the oc_permissions table. I had a
look in the table a minute after the logentry was created, and there actually
was no entry for this key. Some more minutes later, the entry exists.
Case 2: Deadlock in oc_filecache
###########################
The same user is getting a Deadlock on oc_filecache in the very same second.
This time it seems to be from a background tasks.
owncloud-06 - [Aug 4 13:48:28] {webdav} no permissions found, fileid:
113067062 user:userA
owncloud-06 - [Aug 4 13:48:28] {webdav} no permissions found, fileid:
113067062 user:userA
owncloud-06 - [Aug 4 13:48:28] {webdav} no permissions found, fileid:
113067063 user:userA
owncloud-06 - [Aug 4 13:48:28] {webdav} no permissions found, fileid:
113067063 user:userA
owncloud-06 - [Aug 4 13:48:28] {webdav} PDOException: SQLSTATE[40001]:
Serialization failure: 1213 Deadlock found when trying to get lock; try
restarting transaction [40001]
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #1
/WEBROOT/lib/db.php(467): PDOStatementWrapper->execute(Array)
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #2
/WEBROOT/lib/files/cache/cache.php(175): OC_DB::executeAudited('SELECT
`fileid`...', Array)
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #3
/WEBROOT/lib/files/cache/scanner.php(107): OC\Files\Cache\Cache->get('file1')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #4
/WEBROOT/lib/files/cache/scanner.php(203):
OC\Files\Cache\Scanner->scanFile(‚file1', 3, true)
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #5
/WEBROOT/lib/files/cache/scanner.php(170):
OC\Files\Cache\Scanner->scanChildren('files', false, 3)
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #6
/WEBROOT/lib/files/cache/watcher.php(48): OC\Files\Cache\Scanner->scan('files',
false)
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #7
/WEBROOT/lib/files/view.php(789): OC\Files\Cache\Watcher->checkUpdate('files')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #8
/WEBROOT/lib/fileproxy/quota.php(82): OC\Files\View->getFileInfo('/', false)
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #9
/WEBROOT/lib/fileproxy/quota.php(91):
OC_FileProxy_Quota->getFreeSpace('/userA-...')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #10
/WEBROOT/lib/fileproxy.php(107):
OC_FileProxy_Quota->postFree_space('/userA-...', 6.5323421663232E+14)
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #11
/WEBROOT/lib/files/view.php(674): OC_FileProxy::runPostProxies('free_space',
'/userA-...', 6.5323421663232E+14)
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #12
/WEBROOT/lib/files/view.php(639): OC\Files\View->basicOperation('free_space',
‚/file1...')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #13
/WEBROOT/lib/connector/sabre/quotaplugin.php(94):
OC\Files\View->free_space('/file1...')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #14
/WEBROOT/lib/connector/sabre/quotaplugin.php(59):
OC_Connector_Sabre_QuotaPlugin->getFreeSpace('/file1...')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #15 [internal function]:
OC_Connector_Sabre_QuotaPlugin->checkQuota('file1...', Resource id #125,
Object(OC_Connector_Sabre_Directory))
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #16
/WEBROOT/3rdparty/Sabre/DAV/Server.php(432): call_user_func_array(Array, Array)
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #17
/WEBROOT/3rdparty/Sabre/DAV/Server.php(1635):
Sabre_DAV_Server->broadcastEvent('beforeCreateFil...', Array)
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #18
/WEBROOT/3rdparty/Sabre/DAV/Server.php(897):
Sabre_DAV_Server->createFile('file1...', Resource id #125, NULL)
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #19 [internal function]:
Sabre_DAV_Server->httpPut('file1...')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #20
/WEBROOT/3rdparty/Sabre/DAV/Server.php(473): call_user_func(Array, 'file1...')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #21
/WEBROOT/3rdparty/Sabre/DAV/Server.php(213):
Sabre_DAV_Server->invokeMethod('PUT', 'file1...')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #22
/WEBROOT/apps/files/appinfo/remote.php(56): Sabre_DAV_Server->exec()
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #23
/WEBROOT/remote.php(40): require_once('/WEBROOT/...')
owncloud-06 - [Aug 4 13:48:28] {webdav} Exception: #24 {main}
owncloud-06 - [Aug 4 13:48:28] {webdav} no permissions found, fileid: 5030749
user:userA
owncloud-06 - [Aug 4 13:48:28] {webdav} no permissions found, fileid: 5030749
user:userA
Case 3: Duplicate entry in oc_filecache
################################
owncloud-06 - [Aug 4 13:58:24] {webdav} PDOException: SQLSTATE[23000]:
Integrity constraint violation: 1062 Duplicate entry '967-userB' for key
'fs_storage_path_hash' [23000]
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #1
/WEBROOT/lib/db.php(467): PDOStatementWrapper->execute(Array)
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #2
/WEBROOT/lib/files/cache/cache.php(276): OC_DB::executeAudited('?????INSERT
INT...', Array)
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #3
/WEBROOT/lib/files/cache/scanner.php(145):
OC\Files\Cache\Cache->put('file2...', Array)
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #4
/WEBROOT/lib/files/cache/scanner.php(203):
OC\Files\Cache\Scanner->scanFile('file2...', 3, true)
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #5
/WEBROOT/lib/files/cache/scanner.php(170):
OC\Files\Cache\Scanner->scanChildren('file2...', false, 3)
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #6
/WEBROOT/lib/files/cache/watcher.php(48):
OC\Files\Cache\Scanner->scan('file2...', false)
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #7
/WEBROOT/lib/files/view.php(789):
OC\Files\Cache\Watcher->checkUpdate('file2...')
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #8
/WEBROOT/lib/files/filesystem.php(640): OC\Files\View->getFileInfo('/file2...',
true)
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #9
/WEBROOT/lib/connector/sabre/directory.php(145):
OC\Files\Filesystem::getFileInfo('/file2...')
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #10
/WEBROOT/3rdparty/Sabre/DAV/ObjectTree.php(72):
OC_Connector_Sabre_Directory->getChild('xxx')
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #11
/WEBROOT/3rdparty/Sabre/DAV/ObjectTree.php(96):
Sabre_DAV_ObjectTree->getNodeForPath('file2 ...')
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #12
/WEBROOT/3rdparty/Sabre/DAV/Server.php(874):
Sabre_DAV_ObjectTree->nodeExists('file2 ...')
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #13 [internal function]:
Sabre_DAV_Server->httpPut('file2 ...')
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #14
/WEBROOT/3rdparty/Sabre/DAV/Server.php(473): call_user_func(Array, 'file2 ...')
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #15
/WEBROOT/3rdparty/Sabre/DAV/Server.php(213):
Sabre_DAV_Server->invokeMethod('PUT', 'file2 ...')
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #16
/WEBROOT/apps/files/appinfo/remote.php(56): Sabre_DAV_Server->exec()
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #17
/WEBROOT/remote.php(40): require_once(‚/WEBROOT/...')
owncloud-06 - [Aug 4 13:58:24] {webdav} Exception: #18 {main}
The errors occurs on all webservers and for different users. It seems that
there are some users (~100) that are affected more often, where the majority
(~6500) is not affected. Affected users seem to be quite active having more
than several thousand files / directories and some of them even more than
several 100.000 files / directories. Affected users seem to have entries in
oc_filecache with a size of „-1“, which also leads to a wrong quota being
displayed, and in general less entries than files in filesystem. After using
"./occ files:scan“ for the affected users the size and number of entries has
been corrected and the error messages disappeared for some time, but came back
after a while. There seem to be certain conditions triggering those errors but
I could not reproduce or identify them yet. Maybe there are some long running
background tasks (file scans?) interfering with user actions? The "./occ
files:scan“ command took up to 30 - 60 minutes for some users.
I’d like to ask around before opening an issue on github. Maybe someone has an
idea how to solve that?
Some additional infos about our setup:
OC-Version: 5.0.17 (upgraded from 5.0.14)
Database: 3-node MySQL Galera Cluster (Percona XtraDB Cluster Ver 5.5.37), OS:
RHEL 6
Webserver: Apache 2.0 with mod-php 5.3.27, OS: RHEL 6
Apps: Deleted files, Image Viewer, LDAP user and group backend, Media, ODF
Viewer, PDF Viewer, Pictures, Share Files, Text Editor, Updatere, Versions,
Video Viewer, ownCloud Instance Migration, ownCloud dependencies info
Sync clients: At least two of the affected users are using the windows client
v1.6.1
Thanks in advance
Roland Hager
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ Devel mailing list Devel@owncloud.org http://mailman.owncloud.org/mailman/listinfo/devel
