Re: seafile client doesn't sync files
On 2020-11-18, avv. Nicola Dell'Uomo wrote: > Hi Stuart, > > Il 17/11/20 12:45, Stuart Henderson ha scritto: >> On 2020/11/14 21:02, avv. Nicola Dell'Uomo wrote: >>> Hi Stuart, >>> >>> thank you for your help! >>> >>> Now it works (almost). >>> >>> The matter here was that login.conf limits were not high enough. >>> >>> With my server configuration seafile client opens ~ 43.000 files: I really >>> didn't expect such >>> an high limit, so I set it to 8192 in login.conf. >>> >>> Now it works, but it definitely wastes an exaggerate amount of resources. >>> >>> When I close seafile client, kern.nfiles count drop form ~ 44.000 to ~ 890! >> If you have that many files in the monitored directory tree, it's expected. > > hope this helps: I do have this many files in the directory tree; \ > however, when I manually sync each library (autosync off), files \ > count never raises and everything definitely syncs in a few seconds (for \ > huge libraries). > However the situation changes drastically when I enable autosync. I guess "autosync" is what is telling it to use libinotify / kqueue. > Moreover the client behaves differently in linux and in MacOS (it does not > open all these files even with autosync enabled). yes, I have explained about this.
Re: seafile client doesn't sync files
Hi Stuart,
Il 17/11/20 12:45, Stuart Henderson ha scritto:
On 2020/11/14 21:02, avv. Nicola Dell'Uomo wrote:
Hi Stuart,
thank you for your help!
Now it works (almost).
The matter here was that login.conf limits were not high enough.
With my server configuration seafile client opens ~ 43.000 files: I really
didn't expect such
an high limit, so I set it to 8192 in login.conf.
Now it works, but it definitely wastes an exaggerate amount of resources.
When I close seafile client, kern.nfiles count drop form ~ 44.000 to ~ 890!
If you have that many files in the monitored directory tree, it's expected.
hope this helps: I do have this many files in the directory tree; \
however, when I manually sync each library (autosync off), files \
count never raises and everything definitely syncs in a few seconds (for \
huge libraries).
However the situation changes drastically when I enable autosync.
Moreover the client behaves differently in linux and in MacOS (it does not
open all these files even with autosync enabled).
Obviously I'm referring to syncing exactly the the same libraries...
That's what led me to think this could be due to a bug.
Moreover seafile client consume a huge amount of cpu time.
If you have the ports tree installed you can try updating libinotify
- change 20170711 to 20180201 in /usr/ports/devel/libinotify, "make
makesum" and "make update", there are some changes in this that should
reduce memory use and I think maybe also reduce cpu load. If there are
problems or it doesn't help then revert to the version in packages
("pkg_add -r -D downgrade libinotify")
I'll give it a try: I can use the client with no major problem at the
moment \
, but autosyncing would be welcomed (and I think also future users will \
appreciate it ...).
Thank you again for your help!!!
Nicola
Re: seafile client doesn't sync files
On 2020/11/14 21:02, avv. Nicola Dell'Uomo wrote:
> Hi Stuart,
>
> thank you for your help!
>
> Now it works (almost).
>
> The matter here was that login.conf limits were not high enough.
>
> With my server configuration seafile client opens ~ 43.000 files: I really
> didn't expect such
> an high limit, so I set it to 8192 in login.conf.
>
> Now it works, but it definitely wastes an exaggerate amount of resources.
>
> When I close seafile client, kern.nfiles count drop form ~ 44.000 to ~ 890!
If you have that many files in the monitored directory tree, it's expected.
> Moreover seafile client consume a huge amount of cpu time.
If you have the ports tree installed you can try updating libinotify
- change 20170711 to 20180201 in /usr/ports/devel/libinotify, "make
makesum" and "make update", there are some changes in this that should
reduce memory use and I think maybe also reduce cpu load. If there are
problems or it doesn't help then revert to the version in packages
("pkg_add -r -D downgrade libinotify").
Re: seafile client doesn't sync files
So, here is what I discovered so far. 1. private ca: apparently everything works fine if you append your private ca to /etc/ssl/cert.pem. If somebody has an alternative which does not involve modifying stock cert.pem, please let me know; 2. seafile client syncs files only if auto update is disabled. You still have to manually update each single library, but in this case everything works fine and file count from sysctl kern.nfiles won't go wild. 3. If you enable auto update, file count suddendly passes from ~800 - 2000 to ~45000; the client then uses a huge amount of cpu time and files don't sync (maybe because there are too many opened files and the system is not efficient?); 4. if I restrict /etc/login.conf openfiles param to lower numbers (i.e. ~8192), the client just won't work, reporting the same errors as in my first post (Too many open files); 5. I let the client sync for ~ half an hour and nothing changed: so I think this unlikely would change in more sync time. 6. on other OS (mac OS and Linux) file count never exceed 3600. Do you think this could be due to a bug? As I told you I've been having this problem since 6.6 and on different (client) hosts, but I never try to figure it out before because I didn't need seafile to sync my files at the time... Please let me know if I can help with some tests or data. Nicola Il 15/11/20 12:27, avv. Nicola Dell'Uomo ha scritto: Hi Stuart, thank you for your help! Now it works (almost). The matter here was that login.conf limits were not high enough. With my server configuration seafile client opens ~ 43000 files: I really didn't expect such an high limit, so I set it to 8192 in login.conf. Now it works, but it definitely wastes an exaggerate amount of resources. When I close seafile client, kern.nfiles count drops from ~ 44.000 to ~ 890! Moreover seafile client consumes a huge amount of cpu time. I'll monitor the app to see if it's just a momentary situation and if it can be fine tuned... I also have problems in making it understand I use a private ca. When I have news, I'll post them. If somebody has been using seafile client since longtime and already has experience, please join in! Nicola > On 2020-11-14, avv. Nicola Dell'Uomo wrote: > > Hi, > > thank you for answering. > > I raised class limit to 4096 current and 8192 max, but nothing changes: > the logs report exactly the same error scheme as before. > > Do you use this client? No, but the error messages are pretty clear, and I can see that it uses kqueue for file monitoring (via libinotify which emulates Linux's inotify interface using kqueue) so it's a fairly safe bet that's what you're running into. Did you re-login after cbanging login.conf? Check the new values show up with ulimit. > Does it work for you? > > I have many files in each directory, but it works out of the box with > other OS, so I was trying to understand why this is happening in OpenBSD. OpenBSD has pretty tight default limits, plus kqueue works differently than the native inotify on Linux causing it to use more file descriptors. (All dirs and files, not just dirs). > puffy$ ulimit -s > 4096 -s is stack. You want -n (or -a to show all values). > puffy$ sysctl kern.nfiles > kern.nfiles=708 > > I do agree kern.maxfiles is wildly high: this limit was obviously not > meant for production but just for testing purposes... It's not uncommon to set things "for testing" and forget that they're there until the system falls over badly. > Moreover I can't figure out why first sync works and all files are > regularly saved to local destination; then it stops saving to local, but > I can still use all other client function... I suppose if it just syncs all files then it doesn't need to monitor them for updates. >
Re: seafile client doesn't sync files
Hi Stuart, thank you for your help! Now it works (almost). The matter here was that login.conf limits were not high enough. With my server configuration seafile client opens ~ 43000 files: I really didn't expect such an high limit, so I set it to 8192 in login.conf. Now it works, but it definitely wastes an exaggerate amount of resources. When I close seafile client, kern.nfiles count drops from ~ 44.000 to ~ 890! Moreover seafile client consumes a huge amount of cpu time. I'll monitor the app to see if it's just a momentary situation and if it can be fine tuned... I also have problems in making it understand I use a private ca. When I have news, I'll post them. If somebody has been using seafile client since longtime and already has experience, please join in! Nicola > On 2020-11-14, avv. Nicola Dell'Uomo wrote: Hi, thank you for answering. I raised class limit to 4096 current and 8192 max, but nothing changes: the logs report exactly the same error scheme as before. Do you use this client? No, but the error messages are pretty clear, and I can see that it uses kqueue for file monitoring (via libinotify which emulates Linux's inotify interface using kqueue) so it's a fairly safe bet that's what you're running into. Did you re-login after cbanging login.conf? Check the new values show up with ulimit. Does it work for you? I have many files in each directory, but it works out of the box with other OS, so I was trying to understand why this is happening in OpenBSD. OpenBSD has pretty tight default limits, plus kqueue works differently than the native inotify on Linux causing it to use more file descriptors. (All dirs and files, not just dirs). puffy$ ulimit -s 4096 -s is stack. You want -n (or -a to show all values). puffy$ sysctl kern.nfiles kern.nfiles=708 I do agree kern.maxfiles is wildly high: this limit was obviously not meant for production but just for testing purposes... It's not uncommon to set things "for testing" and forget that they're there until the system falls over badly. Moreover I can't figure out why first sync works and all files are regularly saved to local destination; then it stops saving to local, but I can still use all other client function... I suppose if it just syncs all files then it doesn't need to monitor them for updates.
Re: seafile client doesn't sync files
Hi Stuart, thank you for your help! Now it works (almost). The matter here was that login.conf limits were not high enough. With my server configuration seafile client opens ~ 43000 files: I really didn't expect such an high limit, so I set it to 8192 in login.conf. Now it works, but it definitely wastes an exaggerate amount of resources. When I close seafile client, kern.nfiles count drops from ~ 44.000 to ~ 890! Moreover seafile client consumes a huge amount of cpu time. I'll monitor the app to see if it's just a momentary situation and if it can be fine tuned... I also have problems in making it understand I use a private ca. When I have news, I'll post them. If somebody has been using seafile client since longtime and already has experience, please join in! Nicola > On 2020-11-14, avv. Nicola Dell'Uomo wrote: Hi, thank you for answering. I raised class limit to 4096 current and 8192 max, but nothing changes: the logs report exactly the same error scheme as before. Do you use this client? No, but the error messages are pretty clear, and I can see that it uses kqueue for file monitoring (via libinotify which emulates Linux's inotify interface using kqueue) so it's a fairly safe bet that's what you're running into. Did you re-login after cbanging login.conf? Check the new values show up with ulimit. Does it work for you? I have many files in each directory, but it works out of the box with other OS, so I was trying to understand why this is happening in OpenBSD. OpenBSD has pretty tight default limits, plus kqueue works differently than the native inotify on Linux causing it to use more file descriptors. (All dirs and files, not just dirs). puffy$ ulimit -s 4096 -s is stack. You want -n (or -a to show all values). puffy$ sysctl kern.nfiles kern.nfiles=708 I do agree kern.maxfiles is wildly high: this limit was obviously not meant for production but just for testing purposes... It's not uncommon to set things "for testing" and forget that they're there until the system falls over badly. Moreover I can't figure out why first sync works and all files are regularly saved to local destination; then it stops saving to local, but I can still use all other client function... I suppose if it just syncs all files then it doesn't need to monitor them for updates.
Re: seafile client doesn't sync files
On 2020-11-14, avv. Nicola Dell'Uomo wrote: > > Hi, > > thank you for answering. > > I raised class limit to 4096 current and 8192 max, but nothing changes: > the logs report exactly the same error scheme as before. > > Do you use this client? No, but the error messages are pretty clear, and I can see that it uses kqueue for file monitoring (via libinotify which emulates Linux's inotify interface using kqueue) so it's a fairly safe bet that's what you're running into. Did you re-login after cbanging login.conf? Check the new values show up with ulimit. > Does it work for you? > > I have many files in each directory, but it works out of the box with > other OS, so I was trying to understand why this is happening in OpenBSD. OpenBSD has pretty tight default limits, plus kqueue works differently than the native inotify on Linux causing it to use more file descriptors. (All dirs and files, not just dirs). > puffy$ ulimit -s > 4096 -s is stack. You want -n (or -a to show all values). > puffy$ sysctl kern.nfiles > kern.nfiles=708 > > I do agree kern.maxfiles is wildly high: this limit was obviously not > meant for production but just for testing purposes... It's not uncommon to set things "for testing" and forget that they're there until the system falls over badly. > Moreover I can't figure out why first sync works and all files are > regularly saved to local destination; then it stops saving to local, but > I can still use all other client function... I suppose if it just syncs all files then it doesn't need to monitor them for updates. >
Re: seafile client doesn't sync files
Hi, thank you for answering. I raised class limit to 4096 current and 8192 max, but nothing changes: the logs report exactly the same error scheme as before. Do you use this client? Does it work for you? I have many files in each directory, but it works out of the box with other OS, so I was trying to understand why this is happening in OpenBSD. puffy$ ulimit -s 4096 puffy$ sysctl kern.nfiles kern.nfiles=708 I do agree kern.maxfiles is wildly high: this limit was obviously not meant for production but just for testing purposes... Moreover I can't figure out why first sync works and all files are regularly saved to local destination; then it stops saving to local, but I can still use all other client function... Mmmh
Re: seafile client doesn't sync files
On 2020-11-14, avv. Nicola Dell'Uomo wrote: > Hi, > > using seafile client on OpenBSD 6.8 GENERIC on amd64 and xfce (but also > on cwm) > > I'm not able to sync files after first general server sync. > > I observed the same problem with 6.6, 6.7 and on other laptops running > OpenBSD. > > When I first sync files from seafile server, everything goes fine and > files are downloaded and synced; after that, seafile does not sync files > anymore. How does ulimit -n look? You may need to raise the openfiles limit for your user login class (probably "staff" if you are the only user on the system). > I also tried raising kern.maxfiles=300 but nothing changed. Bumping kern.maxfiles is often needed for software doing file monitoring via kqueue/libinotify but that's, err, rather high..
