Re: [Development] QFileSystemWatcher and Recursive Monitoring
BH Alright I'll begin trying to implement some of this, possibly inside the QFileSystemWatcher class itself. But don't expect much, this is opensauce after all so I might come back in a few months and yawn Oh right, QFileSystemWatcher... forgot about that :D Though if some volunteers can help implement the Linux/X11 inotify and MacOSX FSEvents/Kqueue parts, that would help motivate a little (and get everything coded sooner) since I'm mainly familiar with the Windows ReadDirectoryChangesW end. Cheers -regedit ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
I've been thinking about it some more, and I think it would be a good idea to add another optional bool to the opt-in custom interface constructor. So it now would be: QFileSystemWatcher(IQFileSystemWatcherSpecialCoarseToFineGrainedNotificationsFigureOuter *userCustomSnapshotComparisonCode, bool alwaysUseCustomFigureOuterEvenIfPlatformDoesntNeedIt = false, QObject *parent = 0); The reason for the additional bool is that we cannot predict what criteria a _user_ will use to determine a file changed. They may ignore certain changes that the native platform would emit, which will lead to inconsistencies in their application on platforms with native support vs. those without. The bool allows them to opt to take control of every platform so there aren't any inconsistencies. It defaults to false because most of the time the user will want to use the much more efficient native platform notifications when available. We need a pretty beefy class description explaining all this. d3fault ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
Op 28-7-2012 11:04, d3fault schreef: I've been thinking about it some more, and I think it would be a good idea to add another optional bool to the opt-in custom interface constructor. Having (series of) bools in API's is usually a bad idea. See for instance this blog posting: http://ariya.ofilabs.com/2011/08/hall-of-api-shame-boolean-trap.html So it now would be: QFileSystemWatcher(IQFileSystemWatcherSpecialCoarseToFineGrainedNotificationsFigureOuter *userCustomSnapshotComparisonCode, bool alwaysUseCustomFigureOuterEvenIfPlatformDoesntNeedIt = false, QObject *parent = 0); The reason for the additional bool is that we cannot predict what criteria a _user_ will use to determine a file changed. They may ignore certain changes that the native platform would emit, which will lead to inconsistencies in their application on platforms with native support vs. those without. The bool allows them to opt to take control of every platform so there aren't any inconsistencies. It defaults to false because most of the time the user will want to use the much more efficient native platform notifications when available. We need a pretty beefy class description explaining all this. That's exactly the problem. In my view, these things should not be needed in the constructor at all. Instead, make the whole thing property based. That is: make the user-class that you plug into the QFileSystemWatcher something that you can just get and set. There are other examples of this in the Qt API, for instance in working with delegates in itemviews. By default, there will be a delegate set on an item view, but you can set your own if you want. The same goes for QItemSelectionModel: an instance is created by default, but you can just set another one if you want. The benefit is that: 1) it is all very explicit in the code. It is easy to understand what happens 2) it does not get in the way of those not using it 3) it allows initializing in any order that makes sense to the user It would be nice if the system default object doing the work, if there is such a thing, would be available from the API and you could get it from the QFileSytemModel. That would make it possible to use the default implementation from the custom one. So, instead of complicated constructors, I'd suggest some normal API methods instead: //say, QAbstractFileSystemBuffer is the class (interface) that takes care of tracking the file system changes void setFileSystemBuffer(QAbstractFileSystemBuffer* buffer); // QFileSystemModel takes ownership QAbstractFileSystemBuffer* getFileSystemBuffer(); // simply returns a pointer to the object QAbstractFileSystemBuffer* takeFileSystemBuffer(); // releases ownership of the object That makes this possible: QFileSystemWatcher* watcher = new QFileSystemWatcher(this); // MyFileSystemBuffer can use the standard implementation to do much of the work... MyFileSystemBuffer* buffer = new MyFileSystemBuffer(watcher-takeFileSystemBuffer()); watcher-setFileSystemBuffer(buffer); //watcher takes ownership of the buffer André ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
On Thu, Jul 26, 2012 at 9:55 AM, logic.cpp logic@gmail.com wrote:
So how about something like this (others have suggested something along these
lines):
In order to work recursively,
Recursive isn't the same problem. Recursive is easy. The only problem
with doing recursive before was that 256 fd limit on Mac... which I
think we should get around by using FSEvents.
The rest of your post was kind of confusing, but I came up with a
design where everyone wins. It's just a high level public API design,
but you should be able to easily figure out the internals from it (let
me know if you can't).
It's:
1) Cross platform without requiring the user to write custom code
2) Customizable/optimizable for the apps that need/want it
3) Efficient on platforms that support fine grained notifications,
inefficient (unless you do (2) above) on platforms without
Here you go, I used overly verbose wording to make it as clear as
possible. Also, I'm not sure how interfaces are supposed to be named
within Qt so I just guessed. And note the pure virtual public slot in
the interface... it should help explain.
class QFileSystemWatcher : public QObject
{
Q_OBJECT
public:
QFileSystemWatcher(QObject *parent = 0);
QFileSystemWatcher(IQFileSystemWatcherSpecialCoarseToFineGrainedNotificationsFigureOuter
*userCustomSnapshotComparisonCode, QObject *parent = 0);
void addPath(const QString path, bool recursive = false);
void addPaths(const QStringList paths, bool recursive = false);
QStringList directories() const;
QStringList files() const;
void removePath(const QString path, bool recursivelyRemove = false);
void removePaths(const QStringList paths, bool recursivelyRemove =
false);
signals:
void directoryChanged(const QString dir);
void fileChanged(const QString file);
};
class IQFileSystemWatcherSpecialCoarseToFineGrainedNotificationsFigureOuter
: public QObject
{
Q_OBJECT
public slots:
virtual void dirChangedSoFigureOutFileChangesAndEmitThem(const QString
dir)=0;
signals:
void manuallyDetectedFileChanged(const QString file);
};
class
QFileSystemWatcherDefaultSpecialCoarseToFineGrainedNotificationsFigureOuter
: public IQFileSystemWatcherSpecialCoarseToFineGrainedNotificationsFigureOuter
{
Q_OBJECT
public slots:
void dirChangedSoFigureOutFileChangesAndEmitThem(const QString dir)
{
//here is the default implementation using the most common
method
(snapshot comparison of file properties) for detecting fine grained
notifications
//after we detect a file in the dir has changed, we simply do:
emit manuallyDetectedFileChanged(filename);
}
};
using it:
//if you want cross-platform support and don't care that some
platforms will be very slow
QFileSystemWatcher *fsw = new QFileSystemWatcher();
//if you have a special app and want to do some optimized detection
(but this is only applicable to platforms that don't already provide
fine-grained. in those cases the interface is simply ignored)
IQFileSystemWatcherSpecialCoarseToFineGrainedNotificationsFigureOuter
*myOptimizedDetector = new MyOptimizedDetector();
QFileSystemWatcher *fsw = new QFileSystemWatcher(myOptimizedDetector);
d3fault
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
BH I think this is very much like what I was trying to describe. It's really annoying that Mac=Snow Leopard is forcing us into such corners, but if it works and it pleases all Qt evangelists :) then maybe this'll do. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
On Jul 27, 2012 6:02 AM, logic.cpp logic@gmail.com wrote: BH I think this is very much like what I was trying to describe. I think so too, it was immediately after reading your post that the design jumped out at me. The confusing part was your stating that the interface should be required to be implemented, then saying there's a default impl as a bonus (contradiction). Oh well, doesn't matter now :). Are you working on getting it implemented? It's really annoying that Mac=Snow Leopard is forcing us into such corners, but if it works and it pleases all Qt evangelists :) then maybe this'll do. It is annoying, but it's even worse to allow a platform to stall Qt like that. I'm glad to see Qfsw will hopefully finally progress... I'm pretty sure recursive has been in the plans for a while now (it just always leads to this same discussion about platforms not supporting fine grained notifications). d3fault ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
So how about something like this (others have suggested something along these lines): In order to work recursively, the file-system watcher class will require the user to implement a sort of snapshotting backend which will be interfaced/used for FS change comparisons. And only as a bonus, the class will also offer a default such backend on applicable OSs (i.e. all except Mac=Snow Leopard, which in fact don't even need any snapshotting at all). I think this wins because first off, it makes things nice easy for applications that DO involve FS snapshotting /anyway/ (which is possibly 70% of all use-cases that need recursive monitoring in the first place) you could simply interface your application's snapshotting functionalities straight into Qt's filesystem monitor! This is interesting for ALL OSs even those that already offer granular file change details. And at the same time, it's very clear to users of this class that recursive monitoring must have snapshotting - unless a default is kindly provided by Qt BUT not on all OSs. So in other words the question is: Is it acceptable for Qt to say Hey look; to use my QFoo facilities, you must implement bar on your end and plug it in to QFoo. Y'know - most use-cases QFoo was designed for have implemented a bar of their own anyway. But just because I'm so nice, I'll offer you a default bar implementation wherever I can ? ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
On 25/07/2012 00:29, ext Laszlo Papp wrote: On Tue, Jul 24, 2012 at 11:59 PM, marius.storm-ol...@nokia.com wrote: 4) put the functionality in an add-on, which has no requirement of supporting all platforms Really? Was a community decision agreed upon about that earlier? Perhaps I have just missed something, but as an official Qt Project add-on, I would expect a bit more as a developer using that add-on. Perhaps plugin architecture can help or so, with such cases? This important information is not documented here either: http://qt-project.org/wiki/Creating-a-new-module-or-tool-for-Qt If it is like that, perhaps it should be. I may not be the only person whom it makes wondering. :-) I'm not quite sure what information you are missing on that page? I didn't specify if it should be a playground project or not. No matter if it's playground or not, both versions are add-ons which other may include in their builds if they want the functionality. Neither playground nor non-playground add-ons need to support all platforms, though if they want to be in the Qt Essentials set of modules, they do. -- .marius ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
I'm not quite sure what information you are missing on that page? The information about expectations and requirements about an add-on, for sure. There should be mentionings there for new contributors joining the project through playground for instance, or at least a link to such a page from the aforementioned. Neither playground nor non-playground add-ons need to support all platforms, though if they want to be in the Qt Essentials set of modules, they do. Is that documented somewhere? I have not personally seen any such a discussion on this mailing list. Best Regards, Laszlo Papp ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
It predates the list. It's one of the grandfathered principles of the Qt Project: the essentials work everywhere, in all Reference Platforms. Any feature contributed to them must support all Reference Platforms before it can be considered for acceptance. That I am aware of, for sure. However, my default understanding was the same about the add-on modules, which is apparently not right. It would be nice to see this documented, if it is not yet done. Best Regards, Laszlo Papp ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
On 25/07/2012 05:52, ext Thiago Macieira wrote: On quarta-feira, 25 de julho de 2012 11.45.23, Laszlo Papp wrote: Neither playground nor non-playground add-ons need to support all platforms, though if they want to be in the Qt Essentials set of modules, they do. Is that documented somewhere? I have not personally seen any such a discussion on this mailing list. It predates the list. It's one of the grandfathered principles of the Qt Project: the essentials work everywhere, in all Reference Platforms. Any feature contributed to them must support all Reference Platforms before it can be considered for acceptance. It's also stated in a bit roundabout way. If you look at http://qt-project.org/wiki/Qt-Essentials-Modules it'll say The Qt Essentials modules are mandatory in all platforms. while the same is not stated on the http://qt-project.org/wiki/Qt-Add-ons-Modules it says Add-ons can be included optionally in Qt enabled platforms. So, it already says that that Essentials works on all platforms, while add-ons may only work on some. And many of the previous Qt Mobility modules only have backends that work on some platforms, for example. -- .marius ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
Nice. I have missed that. :-) Thank you for your help. Best Regards, Laszlo Papp On Wed, Jul 25, 2012 at 12:03 PM, marius.storm-ol...@nokia.com wrote: On 25/07/2012 05:52, ext Thiago Macieira wrote: On quarta-feira, 25 de julho de 2012 11.45.23, Laszlo Papp wrote: Neither playground nor non-playground add-ons need to support all platforms, though if they want to be in the Qt Essentials set of modules, they do. Is that documented somewhere? I have not personally seen any such a discussion on this mailing list. It predates the list. It's one of the grandfathered principles of the Qt Project: the essentials work everywhere, in all Reference Platforms. Any feature contributed to them must support all Reference Platforms before it can be considered for acceptance. It's also stated in a bit roundabout way. If you look at http://qt-project.org/wiki/Qt-Essentials-Modules it'll say The Qt Essentials modules are mandatory in all platforms. while the same is not stated on the http://qt-project.org/wiki/Qt-Add-ons-Modules it says Add-ons can be included optionally in Qt enabled platforms. So, it already says that that Essentials works on all platforms, while add-ons may only work on some. And many of the previous Qt Mobility modules only have backends that work on some platforms, for example. -- .marius ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
On Jul 23, 2012 7:51 PM, logic.cpp
logic@gmail.comjavascript:_e({}, 'cvml',
'logic@gmail.com');
wrote:
So - are you suggesting that Qt file-system monitoring facilities
should actually generate an in-memory snapshot of the necessary
directory tree(s) for platforms that need it (in this case Mac OSX =
Snow Leopard) ?
Hey I'm not against, it would be really easy to implement. But you
should really try testing something like this yourself on your
computer first. It's /really/ memory consuming as well as /really/
time consuming for large trees of stuff, fairly common nowadays in
anyone's photo/music/video/doc etc. collection. Sure we can alleviate
the time-consuming part with a worker thread, but I'm telling you this
is quite a hunk of resources you're signing off Qt to consume for a
possibly large percentage of use cases (many many people out there
have = Snow Leopard).
I really like the idea of 1 signal and use the best as possible of each OS.
I am against Qt to make a snapshot of the directory because it will not be
done efficiently, means that this code, to be efficient, depends on the
task to achieve, and there is probably a ton of optimisations that we can
apply in a particular context.
Maybe create a separate to provide a helper of creating a snapshot? In this
case I would suggest using a sqlite database and not the memory. Memory is
definitly a showstopper.
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
On Jul 23, 2012 11:49 PM, Sylvain Pointeau sylvain.point...@gmail.com wrote: I really like the idea of 1 signal and use the best as possible of each OS. I am against Qt to make a snapshot of the directory because it will not be done efficiently, means that this code, to be efficient, depends on the task to achieve, and there is probably a ton of optimisations that we can apply in a particular context. It will only be inefficient for the few platforms that don't support the functionality. The cost for platforms that already do provide the functionality is zero. Filesystem notifications are a pretty basic piece of functionality, and having them be 100% cross platform would be a huge win for Qt. Maybe create a separate to provide a helper of creating a snapshot? In this case I would suggest using a sqlite database and not the memory. Memory is definitly a showstopper. Theoretically it could be either (user chooses memory or sqlite) or both (fixed size memory cache with sqlite storing the rest) or neither (when the user wants to maintain the snapshot themselves (but remember, this is only applicable to the platforms that DON'T support fine grained fs event notifications. All these options are n/a to the platforms that do) because they can perform context sensitive optimizations as you mentioned)... but uhh I'm willing to bet sqlite would be a lot slower since now there's a hdd read/write. Would save memory though if that's your concern. It's just added design complexity which I'm guessing regedit doesn't want to deal with. It's already getting complex enough. Can any Qt devs comment on whether QFsw would even be allowed in Qt in a non cross-platform state (ignoring the fact that pretty much any solution is better than the current Qt release's solution lol)? regedit, you might find it doesn't pass code review... but maybe the capabilities() hack is enough, idk (i still wouldn't recommend it)... d3fault p.s. just thought of this: you could specify what properties in the snapshot you want to be compared as an optimization (enum flags filter). but then there's the cross platform breaking use case where you use the optimization just described to purposefully get less notifications (say... ignore filename changes but tell me about file size changes (all using some enum flag filter mechanism))... but when you run that code on a platform that doesn't need the snapshot logic, you'll still receive filename changed events (unless your setting of said enum flag filters activates snapshot comparison mode... which isn't there because it's #ifdef'd out on this platform (and so shouldn't be)). Bah, this optimization sucks but it's still better than forcing every _user_ app to do it manually every time. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
It will only be inefficient for the few platforms that don't support the functionality. The cost for platforms that already do provide the functionality is zero. Filesystem notifications are a pretty basic piece of functionality, and having them be 100% cross platform would be a huge win for Qt. having them inefficient is worst than not having them. make a separate lib or class and let the user decide if he can use it. don't force everybody to be penalized. As I said, a generic implementation cannot beat specialized code for the specific domain/case. Maybe create a separate to provide a helper of creating a snapshot? In this case I would suggest using a sqlite database and not the memory. Memory is definitly a showstopper. Theoretically it could be either (user chooses memory or sqlite) or both (fixed size memory cache with sqlite storing the rest) or neither (when the user wants to maintain the snapshot themselves (but remember, this is only applicable to the platforms that DON'T support fine grained fs event notifications. All these options are n/a to the platforms that do) because they can perform context sensitive optimizations as you mentioned)... but uhh I'm willing to bet sqlite would be a lot slower since now there's a hdd read/write. Would save memory though if that's your concern. It's just added design complexity which I'm guessing regedit doesn't want to deal with. It's already getting complex enough. Can any Qt devs comment on whether QFsw would even be allowed in Qt in a non cross-platform state (ignoring the fact that pretty much any solution is better than the current Qt release's solution lol)? regedit, you might find it doesn't pass code review... but maybe the capabilities() hack is enough, idk (i still wouldn't recommend it)... Please go for it as a separate lib/class, the users will decide if this is efficient enough to be usable. Anyway we can implement the single signal notification, and see if your implementation is robust, fast, and using only few memory, to be implemented directly in the FSW. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
On Jul 24, 2012 3:09 AM, Sylvain Pointeau sylvain.point...@gmail.com
wrote:
having them inefficient is worst than not having them.
Arguably. We have to choose from the following:
1) Make Qt only target Lion+, drop Leopard support as well as any platform
without fine grained fs notifications
2) Not have a cross platform public API (breaks Qt rules?)
3) Be inefficient for the few platforms that don't support fine grained fs
notifications
All 3 options suck (as does the current state of Qfsw), but I'm pretty sure
the last option sucks the least.
make a separate lib or class and let the user decide if he can use it.
don't force everybody to be penalized.
Similarly, don't force everyone to reinvent the wheel.
As I said, a generic implementation cannot beat specialized code for the
specific domain/case.
Yes, but as mentioned before, we could additionally (runtime switch? ex:
if(Qfsw::currentPlatformDoesntSupportFineGrainedFsNotifications()) {
m_Qfsw.setDontSimulateFineGrainedSupportBecauseWeWillDoItManually(true); /*
already true(ish) for platforms with fine grain notifications */ }) disable
the behind-the-scenes-state-comparison code so the user can optimize their
heart out. They'd have to again use that static fine grained detection
method in their slot connected to dirChanged to decide whether or not to
even use their own manual implementation, because most relevant platforms
already provide fine grained notifications and the user's manual code
should then be skipped. A bit complicated, but definitely doable.
d3fault
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
4) put the functionality in an add-on, which has no requirement of supporting
all platforms, and allow people who need it link to it. It can live its life
there until a decent algorithm and API has been developed, and then we can
bring it in if we want.
--
Sent from my Nokia N9
On 7/24/12 17:21 ext d3fault wrote:
On Jul 24, 2012 3:09 AM, Sylvain Pointeau sylvain.point...@gmail.com wrote:
having them inefficient is worst than not having them.
Arguably. We have to choose from the following:
1) Make Qt only target Lion+, drop Leopard support as well as any platform
without fine grained fs notifications
2) Not have a cross platform public API (breaks Qt rules?)
3) Be inefficient for the few platforms that don't support fine grained fs
notifications
All 3 options suck (as does the current state of Qfsw), but I'm pretty sure the
last option sucks the least.
make a separate lib or class and let the user decide if he can use it.
don't force everybody to be penalized.
Similarly, don't force everyone to reinvent the wheel.
As I said, a generic implementation cannot beat specialized code for the
specific domain/case.
Yes, but as mentioned before, we could additionally (runtime switch? ex:
if(Qfsw::currentPlatformDoesntSupportFineGrainedFsNotifications()) {
m_Qfsw.setDontSimulateFineGrainedSupportBecauseWeWillDoItManually(true); /*
already true(ish) for platforms with fine grain notifications */ }) disable the
behind-the-scenes-state-comparison code so the user can optimize their heart
out. They'd have to again use that static fine grained detection method in
their slot connected to dirChanged to decide whether or not to even use their
own manual implementation, because most relevant platforms already provide fine
grained notifications and the user's manual code should then be skipped. A bit
complicated, but definitely doable.
d3fault
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
On Tue, Jul 24, 2012 at 11:59 PM, marius.storm-ol...@nokia.com wrote: 4) put the functionality in an add-on, which has no requirement of supporting all platforms Really? Was a community decision agreed upon about that earlier? Perhaps I have just missed something, but as an official Qt Project add-on, I would expect a bit more as a developer using that add-on. Perhaps plugin architecture can help or so, with such cases? This important information is not documented here either: http://qt-project.org/wiki/Creating-a-new-module-or-tool-for-Qt If it is like that, perhaps it should be. I may not be the only person whom it makes wondering. :-) Bst Regards, Laszlo Papp ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
I don't like your single signal design. It's cross platform only in that it will compile/function on every platform. The application will behave differently in the slot connected to said signal on mac than on windows/linux. We should not expect every _user_ app to have separate codepaths for different platforms. A.k.a exactly what Qt isn't about. What else do you suggest then, should we make 3 separate classes one for each OS? Just because you attempted the perfect solution once and failed, doesn't mean it can't be done. At the end of the day we just need some #ifdef Q_MAC_OSX (or maybe this can/should be in QPA) with boring file property (filename,filesize,maybe-crc32?,etc) comparison stuff before determing the proper signal to emit. By all means, please do try to generate this in-memory snapshot of a sizable directory tree (say 20,000 folders 150,000 files) with all the necessary info you listed (filename,filesize etc.) and tell me how it goes. Last I tried it on my own hard drive full of stuff (40k folders 300k files) it 1) consumed too huge an amount of memory, and 2) took too long to complete. *** What I'm trying to achieve here is to have some kind of common cross platform interface for file-system monitoring (in contrast to, for example, making 3 separate classes QWinWatcher QNixWatcher QMacWatcher which will force users to have platform specific code) that works as similarly as possible on all OSs and gracefully degrades as necessary. Users will be made aware of this degradation and will know to check for it and program their code to cover the different eventualities (like with a capabilities() function or some such). In fact - as was noted in a previous post - even on Mac OS itself there can be different features available depending on which version (Snow Leopard(-) vs. Lion(+)) and not just any trivial feature difference, it's a huge important one; getting actual file-level details on what exactly changed, eliminating the need for an existing snapshot to compare against! Will Qt not take advantage of these features where available? Maybe not, I don't know... but if yes, it would only make sense to have some sort of capabilities() function or some-such which can dynamically inform users about what features are available and what isn't. Anyway, that's just one idea of mine. Dunno if it works into Qt's way of doing things. Please let me know. The questions remain as follows; - Should it all be in one class? If yes, how to go about differences in features/functionality. Should we emit many different kinds of signals, or just 1 general signal. Should we do this all in QFileSystemWatcher, or another new class. - Or should fs-watching exist in totally separate platform-specific classes? - Other? -regedit ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
On Jul 23, 2012 8:07 AM, logic.cpp logic@gmail.com wrote: What else do you suggest then, should we make 3 separate classes one for each OS? definitely not. qt is a cross platform toolkit after all. separate back-ends... on the other hand: yes. By all means, please do try to generate this in-memory snapshot of a sizable directory tree (say 20,000 folders 150,000 files) with all the necessary info you listed (filename,filesize etc.) and tell me how it goes. Last I tried it on my own hard drive full of stuff (40k folders 300k files) it 1) consumed too huge an amount of memory, and 2) took too long to complete. That's fine, so long as a warning is present in the class description for affected platforms. The argument i'm making is that EVERY _user_ app will be responsible for creating the in-memory snapshot and additional codepaths at least if they plan on targetting Mac OS. The overhead in terms of memory/performance is the exact same... but you're giving app developers the extra burden of doing it every time they use qfsw. It's better behind the scenes and hidden from the _user_. At the very least just so it only has to be coded/maintained once (instead of once per _user_ app). What I'm trying to achieve here is to have some kind of common cross platform interface for file-system monitoring (in contrast to, for example, making 3 separate classes QWinWatcher QNixWatcher QMacWatcher which will force users to have platform specific code) that works as similarly as possible on all OSs and gracefully degrades as necessary. Users will be made aware of this degradation Behind the scenes compile time progressive enhancement is better than userland runtime graceful degradation -- at least in this use case. In fact - as was noted in a previous post - even on Mac OS itself there can be different features available depending on which version (Snow Leopard(-) vs. Lion(+)) and not just any trivial feature difference, it's a huge important one; getting actual file-level details on what exactly changed, eliminating the need for an existing snapshot to compare against! Will Qt not take advantage of these features where available? Simple, the #ifdef Q_MAC_OSX now becomes #if (Q_MAC_OSX_VERSION = 69 /*or whatever the Leopard version is*/) So Lion+ uses it's native functionality, and Leopard and any other platform missing the functionality uses an expensive in-house snapshot comparison method. The questions remain as follows; - Should it all be in one class? Yes, making it two classes isn't necessary. It's still the same functionality, just with a recursive element. If yes, how to go about differences in features/functionality. Should we emit many different kinds of signals, or just 1 general signal. This doesn't matter so much, as long as the Qt code ensures that the same signal(s) are emitted for every platform. Using a single signal + capabilities() shifts the burden onto the _user_ developer... and one could even argue that qfsw isn't even cross platform at that point. d3fault ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
So - are you suggesting that Qt file-system monitoring facilities should actually generate an in-memory snapshot of the necessary directory tree(s) for platforms that need it (in this case Mac OSX = Snow Leopard) ? Hey I'm not against, it would be really easy to implement. But you should really try testing something like this yourself on your computer first. It's /really/ memory consuming as well as /really/ time consuming for large trees of stuff, fairly common nowadays in anyone's photo/music/video/doc etc. collection. Sure we can alleviate the time-consuming part with a worker thread, but I'm telling you this is quite a hunk of resources you're signing off Qt to consume for a possibly large percentage of use cases (many many people out there have = Snow Leopard). BTW d3fault are you on IRC? What's your handle timezone, maybe we can chat about this more. -regedit P.S. Just on a personal note, the reason I got caught up in all this to begin with was some application I wanted to develop. My particular project involves a snapshot of the file system anyway, so in my particular case on MacOSX=Snow Leopard I would definitely just obtain directory-level information from FSEvents and proceed to compare the directory's contents with that of my existing snapshot. In fact if Qt is gonna do a whole in-memory snapshot as d3fault suggests for this situation, I'm considering dropping this whole QFSW campaign and just roll out my own solution :) How about we offer a choice to the programmer whether they want Qt to do the internal snapshot (perhaps the default) or not? ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
On Jul 23, 2012 7:51 PM, logic.cpp logic@gmail.com wrote: So - are you suggesting that Qt file-system monitoring facilities should actually generate an in-memory snapshot of the necessary directory tree(s) for platforms that need it (in this case Mac OSX = Snow Leopard) ? Hey I'm not against, it would be really easy to implement. But you should really try testing something like this yourself on your computer first. It's /really/ memory consuming as well as /really/ time consuming for large trees of stuff, fairly common nowadays in anyone's photo/music/video/doc etc. collection. Sure we can alleviate the time-consuming part with a worker thread, but I'm telling you this is quite a hunk of resources you're signing off Qt to consume for a possibly large percentage of use cases (many many people out there have = Snow Leopard). The amount of resources consumed is irrelevant if the _user_ consumes the same amount with their custom implementation. The only value gained in doing the work up front in Qt / behind the scenes is that now every _user_ developer doesn't have to implement it manually. It will save a ton of man hours from having to reinvent the wheel. You mentioned giving the user a choice on whether Qt maintains the state comparison tree or not. This is definitely the optimal choice, but: a) adds design complexity b) would maybe be confusing on platforms where it's n/a both of those problems are solvable (obscure the (b) choice somewhat... i think i recall a Mac-specific QMainWindow function regarding the toolbar and OS integration. do something like that) BTW d3fault are you on IRC? What's your handle timezone, maybe we can chat about this more. I'm not on IRC lately but you can email me if you want. Irregular/shifting sleeping patterns so timezone won't help. P.S. Just on a personal note, the reason I got caught up in all this to begin with was some application I wanted to develop. My particular project involves a snapshot of the file system anyway, so in my particular case on MacOSX=Snow Leopard I would definitely just obtain directory-level information from FSEvents and proceed to compare the directory's contents with that of my existing snapshot. In fact if Qt is gonna do a whole in-memory snapshot as d3fault suggests for this situation, I'm considering dropping this whole QFSW campaign and just roll out my own solution :) Might as well contribute your own solution then. This way you don't have to maintain the code... everyone else (myself included) benefits... and you don't even have to write that much more code. You just organize it differently (and comply with Qt coding standards etc). d3fault ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring (Arvid E. Picciani)
BH Hey'all, Thanks for the feedback. I think we can still make everything work inside one single class, either QFileSystemWatcher or perhaps a new class if we don't want to break QFSW too much. We'll do this à la Graceful Degradation. Before I explain, consider also the fact that - as Arvid mentioned - QFSW itself doesn't (yet) offer fully granular notifications as to what precisely changed. All we have is fileChanged(), directoryChanged(), fileModified(), pathCreated() and pathDeleted() (the last three are w00t's patch in review https://codereview.qt-project.org/#change,19274 ). I propose that Qt should offer more awesomely useful notifications about file-system changes. This can be acheived either by creating even moar signals, or perhaps in a different way - as will be explained. Here's how it'll work; Windows: With ReadDirectoryChangesW we have single-item monitoring, recursive monitoring, and we can get this cool notification Renamed from + Renamed to which QFSW doesn't offer ATM. Linux/X11: With inotify we have single-item monitoring, recursive monitoring (gulp), and we have even awesomer signals than windows, like Moved from + Moved to. Mac OS X: We'll use kqueue for single-item monitoring (addPath() for A-category use cases) and we'll use FSEvents for recursive monitoring (addPathRecursive() for B use cases). From what I gather, in Qt we don't like to have actual functions/signals that behave differently (or are altogether ignored) on different platforms. So instead of having many different signals in the file-system monitor class which won't even be emitted on some platforms (Mac), how about we emit just ONE signal that will pass to the user 1) a path string and 2) an enum flag about what excly happened (akin to inotify events enum http://en.wikipedia.org/wiki/Inotify ). This ensures consistent behavior cross all platforms, there's always a path and enum passed with the signal. The only difference will be in the granularity of info that the path/enum-flag will have, offering the most info available on the platform gracefully degrading as needed. So for example, for a single file watch on foo/bar/baz.txt: Windows: C:\foo\bar\baz.txt QFSW_DELETED Linux/X11: /foo/bar/baz.txt QFSW_DELETED Mac (kqueue): /foo/bar/baz.txt QFSW_DELETED - no degradation And for a recursive watch on the foo/ directory: Windows: C:\foo\bar\baz.txt QFSW_DELETED Linux/X11: /foo/bar/baz.txt QFSW_DELETED Mac (FSEvents): /foo/bar/ QFSW_DIR_CHANGED - graceful degradation Notice how on Mac we get notified on the exact leaf-most folder where the event happened, and that the event is always just something changed here. This will all be very clearly documented in detail, so users will know that on Mac they need to do further investigations when using recursive. They'll also learn about the limits of single-item monitoring (256 kqueue file descriptors). Alternatively we can expose a capabilities() function which can dynamically tell the user about the API's platform limitations (granularity of notification info, file-descriptor count limitations on Mac, etc). Bottom line: Doing it like this will successfully exploit the capabilities of all platforms' file-system monitoring facilities to their max, and will safely cover most if not all use case scenarios. A.k.a. what Qt is all about :) -regedit P.S. I was made aware by a smart fellow in #qt-labs that FSEvents began offering file-level notifications since OS X Lion+. Seeing as Qt must support older versions, the idea of a capabilities() function sounds more interesting here (so Lion+ can already begin taking advantage of file-level events). ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring (Arvid E. Picciani)
Whoops I am so sorry, total noob here when it comes to mailing lists. This was meant to be a reply to the existing thread http://lists.qt-project.org/pipermail/development/2012-July/005279.html The last post of which was by Arvid E. Picciani Not sure if this means the discussion continues in this thread from now on, or whether we resume the previous thread... Someone halp please! -regedit ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
BH Hey'all, Thanks for the feedback. I think we can still make everything work inside one single class, either QFileSystemWatcher or perhaps a new class if we don't want to break QFSW too much. We'll do this à la Graceful Degradation. Before I explain, consider also the fact that - as Arvid mentioned - QFSW itself doesn't (yet) offer fully granular notifications as to what precisely changed. All we have is fileChanged(), directoryChanged(), fileModified(), pathCreated() and pathDeleted() (the last three are w00t's patch in review https://codereview.qt-project.org/#change,19274 ). I propose that Qt should offer more awesomely useful notifications about file-system changes. This can be acheived either by creating even moar signals, or perhaps in a different way - as will be explained. Here's how it'll work; Windows: With ReadDirectoryChangesW we have single-item monitoring, recursive monitoring, and we can get this cool notification Renamed from + Renamed to which QFSW doesn't offer ATM. Linux/X11: With inotify we have single-item monitoring, recursive monitoring (gulp), and we have even awesomer signals than windows, like Moved from + Moved to. Mac OS X: We'll use kqueue for single-item monitoring (addPath() for A-category use cases) and we'll use FSEvents for recursive monitoring (addPathRecursive() for B use cases). From what I gather, in Qt we don't like to have actual functions/signals that behave differently (or are altogether ignored) on different platforms. So instead of having many different signals in the file-system monitor class which won't even be emitted on some platforms (Mac), how about we emit just ONE signal that will pass to the user 1) a path string and 2) an enum flag about what excly happened (akin to inotify events enum http://en.wikipedia.org/wiki/Inotify ). This ensures consistent behavior cross all platforms, there's always a path and enum passed with the signal. The only difference will be in the granularity of info that the path/enum-flag will have, offering the most info available on the platform gracefully degrading as needed. So for example, for a single file watch on foo/bar/baz.txt: Windows: C:\foo\bar\baz.txt QFSW_DELETED Linux/X11: /foo/bar/baz.txt QFSW_DELETED Mac (kqueue): /foo/bar/baz.txt QFSW_DELETED - no degradation And for a recursive watch on the foo/ directory: Windows: C:\foo\bar\baz.txt QFSW_DELETED Linux/X11: /foo/bar/baz.txt QFSW_DELETED Mac (FSEvents): /foo/bar/ QFSW_DIR_CHANGED - graceful degradation Notice how on Mac we get notified on the exact leaf-most folder where the event happened, and that the event is always just something changed here. This will all be very clearly documented in detail, so users will know that on Mac they need to do further investigations when using recursive. They'll also learn about the limits of single-item monitoring (256 kqueue file descriptors). Alternatively we can expose a capabilities() function which can dynamically tell the user about the API's platform limitations (granularity of notification info, file-descriptor count limitations on Mac, etc). Bottom line: Doing it like this will successfully exploit the capabilities of all platforms' file-system monitoring facilities to their max, and will safely cover most if not all use case scenarios. A.k.a. what Qt is all about :) -regedit P.S. I was made aware by a smart fellow in #qt-labs that FSEvents began offering file-level notifications since OS X Lion+. Seeing as Qt must support older versions, the idea of a capabilities() function sounds more interesting here (so Lion+ can already begin taking advantage of file-level events). ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
Darn, second attempt to fix my mess-up here in the mailing list only made things worse. Shoot me. -regedit On Fri, Jul 20, 2012 at 1:21 AM, logic.cpp logic@gmail.com wrote: BH Hello, Hope you don't mind my less-waste-your-time more-get-to-the-point English. Begin ramble: Wanted to fix QFileSystemWatcher for Windows as per w00t's article: http://blog.rburchell.com/2012/03/qt-51-aka-when-qfilesystemwatcher-might.html (also see his fix: https://codereview.qt-project.org/#change,19274 ) Realized that what I really needed was *recursive* monitoring, entire trees, which the WinAPI function ReadDirectoryChangesW http://msdn.microsoft.com/en-us/library/windows/desktop/aa365465(v=vs.85).aspx that was mentioned in w00t's article offers right out of the box. Others in #qt-labs also believe recursive file-system monitoring to be a useful feature to offer in Qt. Here is my research on the topic: 1. Status of file-system monitoring on all major platforms. Windows --- API: ReadDirectoryChangesW Recursive: yes Granular info about what changed: yes Linux/X11 - API: inotify Recursive: no, but can be achieved manually* Granular info about what changed: yes *I like to refer to the Gnome Tracker project (and probably also the KDE Nepomuk project), they manually traverse spawn another inotify monitor for each every single dir in entire trees. They also claim that it doesn't hurt performance too much either. It's the best you can currently get, there's currently no other way of doing it. MacOSX -- API: Kqueue Recursive: no, and cannot be achieved manually* Granular info about what changed: yes * Max 256 file descriptors anyway, so out of question for recursive purposes. -- API: FSEvents Recursive: yes Granular info about what changed: no, but you are notified about exactly where the change happened (leaf-most directory, direct parent of where change happened) so at most you need to manually poll contents of just 1 directory. Also multiple changes are compounded into 1 event, configurable. MUST read: http://arstechnica.com/apple/2007/10/mac-os-x-10-5/7/ 2. Problems with making QFileSystemWatcher recursive - Windows ReadDirectoryCHangesW is the winner here. Would be easiest to implement, we could even drop all the threading currently there now. - inotify gives us all the chills at the thought of manually generating watches recursively for entire directory trees, but it can be done satisfactorially enough behind the scenes inside QFileSystemWatcher. - Mac OS is the troublemaker here; Kqueue is out of question, as we established. With FSEvents you cannot emit signals such as fileChanged(), fileModified(), pathCreated(), or pathDeleted() if all you're notified about is that something happened in this folder here, you MUST have some sort of snapshot of the folder to compare against, and QFileSystemWatcher generating an in-memory snapshot of the user's requested directories to be watched can be crazy time consuming and memory draining. Try doing this with an entire drive of some 40,000 folders and 300K files. 3. What are the use cases anyway? Reevaluate: what is file-system watching good for anyway? There are two general use-case categories I can think of: A) Your everyday lightweight necessities, like has the config file changed. QFileSystemWarcher is just fine for this with inotify kqueue (though Windows can still use some ReadDirectoryChangesW love instead of the current use of Find*ChangeNotification APIs). B) Heavy duty monitoring of entire hierachies, like monitoring dir trees in an image library, backup applications, file crawling / indexing, virus scanning etc. If you think about it, I suspect all B-style applications can be expected to have their own snapshot of the file system stored anyway. Any app that needs to monitor changes to THAT much stuff probably has it's own view of the file system that it just wants to keep updated. If this assumption is acceptable, we discover that granular info about what exactly changed is (useful but) not too urgent after all. So maybe FSEvents on mac isn't so bad. The application will only need to compare 1 folder contents with it's own snapshot to determine what changed. So now what; If we care to offer recursive FS monitoring in Qt, we can either try to stuff it into QFileSystemWatcher, or make a new class for it (perhaps platform specific API classes if necessary). 4. The first option: Enhanced QFileSystemWatcher If we do it this way, we'll one single awesome class that does it all. We just may have to make peace with varying usage/behavior on different platforms. If this is acceptable, we're in business. For example, here are some things we might need to mention in the docs; QFileSystemWatcher::fileModified() NOTE: This signal is not emitted on Mac OS X. Instead, directoryChanged() is emitted and the user is
Re: [Development] QFileSystemWatcher and Recursive Monitoring
Notice how on Mac we get notified on the exact leaf-most folder where the event happened, and that the event is always just something changed here. Are you sure about that? The documentation specifies an kFSEventStreamEventFlagMustScanSubDirs flag for events which indicates that a problem happened either in the kernel or user space and events were coalesced. how about we emit just ONE signal that will pass to the user 1) a path string and 2) an enum flag about what exactly happened (akin to inotify events enum http://en.wikipedia.org/wiki/Inotify ). Sounds sensible. It looks as though this would map better to the underlying APIs for FSEvents and ReadDirectoryChangesW. How will you handle renames? It looks like ReadDirectoryChanges uses two separate events, leaving it up to the user to match them up. With inotify we have single-item monitoring, recursive monitoring (gulp), But do document the issues with watch limits. A related issue is that scanning the file system in order to set up the notifications on a sub-tree could take an arbitrary amount of time with inotify. You could potentially defer that work to a background thread, or simply document the issue and leave it up to the app developer - if the app is going to scan a whole dir tree at some point, it will most likely not want to do that from the UI thread anyway. Regards, Rob. On 22 July 2012 11:15, logic.cpp logic@gmail.com wrote: Darn, second attempt to fix my mess-up here in the mailing list only made things worse. Shoot me. -regedit On Fri, Jul 20, 2012 at 1:21 AM, logic.cpp logic@gmail.com wrote: BH Hello, Hope you don't mind my less-waste-your-time more-get-to-the-point English. Begin ramble: Wanted to fix QFileSystemWatcher for Windows as per w00t's article: http://blog.rburchell.com/2012/03/qt-51-aka-when-qfilesystemwatcher-might.html (also see his fix: https://codereview.qt-project.org/#change,19274 ) Realized that what I really needed was *recursive* monitoring, entire trees, which the WinAPI function ReadDirectoryChangesW http://msdn.microsoft.com/en-us/library/windows/desktop/aa365465(v=vs.85).aspx that was mentioned in w00t's article offers right out of the box. Others in #qt-labs also believe recursive file-system monitoring to be a useful feature to offer in Qt. Here is my research on the topic: 1. Status of file-system monitoring on all major platforms. Windows --- API: ReadDirectoryChangesW Recursive: yes Granular info about what changed: yes Linux/X11 - API: inotify Recursive: no, but can be achieved manually* Granular info about what changed: yes *I like to refer to the Gnome Tracker project (and probably also the KDE Nepomuk project), they manually traverse spawn another inotify monitor for each every single dir in entire trees. They also claim that it doesn't hurt performance too much either. It's the best you can currently get, there's currently no other way of doing it. MacOSX -- API: Kqueue Recursive: no, and cannot be achieved manually* Granular info about what changed: yes * Max 256 file descriptors anyway, so out of question for recursive purposes. -- API: FSEvents Recursive: yes Granular info about what changed: no, but you are notified about exactly where the change happened (leaf-most directory, direct parent of where change happened) so at most you need to manually poll contents of just 1 directory. Also multiple changes are compounded into 1 event, configurable. MUST read: http://arstechnica.com/apple/2007/10/mac-os-x-10-5/7/ 2. Problems with making QFileSystemWatcher recursive - Windows ReadDirectoryCHangesW is the winner here. Would be easiest to implement, we could even drop all the threading currently there now. - inotify gives us all the chills at the thought of manually generating watches recursively for entire directory trees, but it can be done satisfactorially enough behind the scenes inside QFileSystemWatcher. - Mac OS is the troublemaker here; Kqueue is out of question, as we established. With FSEvents you cannot emit signals such as fileChanged(), fileModified(), pathCreated(), or pathDeleted() if all you're notified about is that something happened in this folder here, you MUST have some sort of snapshot of the folder to compare against, and QFileSystemWatcher generating an in-memory snapshot of the user's requested directories to be watched can be crazy time consuming and memory draining. Try doing this with an entire drive of some 40,000 folders and 300K files. 3. What are the use cases anyway? Reevaluate: what is file-system watching good for anyway? There are two general use-case categories I can think of: A) Your everyday lightweight necessities, like has the config file changed. QFileSystemWarcher is just fine for this with inotify kqueue (though Windows can still use some ReadDirectoryChangesW love instead of the current
Re: [Development] QFileSystemWatcher and Recursive Monitoring
Are you sure about that? The documentation specifies an kFSEventStreamEventFlagMustScanSubDirs flag for events which indicates that a problem happened either in the kernel or user space and events were coalesced. Events being coalesced is another unrelated thing (a feature in fact - not an issue). The 'kFSEventStreamEventFlagMustScanSubDirs' thing is an important event that lets the user know that Whoa- too much has happened and buffers have overflowed, assume that some events are not recorded/missing so it's time for you to to a bit of manual scanning. The same is available on Windows RDCW and in inotify. Indeed this is yet another file-system event we should offer in Qt that QFSW does not currently account for. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
How will you handle renames? It looks like ReadDirectoryChanges uses two separate events, leaving it up to the user to match them up. As for renames (and moves on Linux/X11) we'd need to come up with some mechanism to sebd 2 pieces of information in one signal. This can be either quick-n-dirty passing a concatenated string (semicolon separated or whatever) as the path value, or getting fancy with passing pointers to structures that may have several pieces of info packed in depending on whether it's a regular event or a rename/move event. Or I dunno, any other ideas? It's reallys just a technicality that can definitely be figured out easily. -regedit ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
This can be either quick-n-dirty
passing a concatenated string (semicolon separated or whatever) as the path
value, or getting fancy with passing pointers to structures that may have
several pieces of info packed in
I'd just add an extra parameter to the signal which is empty for some events:
pathChanged(ChangeType type, const QString path, const QString originalPath);
or define a simple struct and pass by const-ref:
struct PathChange
{
Type event;
QString path;
QString originalPath;
};
Regards,
Rob.
On 22 July 2012 21:29, logic.cpp logic@gmail.com wrote:
How will you handle renames? It looks like ReadDirectoryChanges uses
two separate events, leaving it up to the user to match them up.
As for renames (and moves on Linux/X11) we'd need to come up with some
mechanism
to sebd 2 pieces of information in one signal. This can be either
quick-n-dirty
passing a concatenated string (semicolon separated or whatever) as the path
value, or getting fancy with passing pointers to structures that may have
several pieces of info packed in depending on whether it's a regular event or
a
rename/move event.
Or I dunno, any other ideas? It's reallys just a technicality that can
definitely be figured out easily.
-regedit
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
I don't like your single signal design. It's cross platform only in that it will compile/function on every platform. The application will behave differently in the slot connected to said signal on mac than on windows/linux. We should not expect every _user_ app to have separate codepaths for different platforms. A.k.a exactly what Qt isn't about. Just because you attempted the perfect solution once and failed, doesn't mean it can't be done. At the end of the day we just need some #ifdef Q_MAC_OSX (or maybe this can/should be in QPA) with boring file property (filename,filesize,maybe-crc32?,etc) comparison stuff before determing the proper signal to emit. I also think we should remove the 256 limit even for non-recursive monitoring on the Mac (always use FSEvents). 256 is a ridiculously small limit for any app wanting to do any slightly serious fs event monitoring. Aside: perhaps QObject::connectNotify can be cleverly used to skip the string comparison stage for an event if there is no slot connected to that particular signal. So if they only connect to dirChanged, the internal code never uses the 'expensive' #ifdef'd out file change detected code. d3fault ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
If Windows and Linux provide the capabilities, we shouldn't let specific functionality absent in Mac (or any platform for that matter) define Qt. I think modifying the addPath methods to include an optional bool watchRecursively = false as the second parameter is the correct solution. Any platforms that don't have a suitable solution can/should use an #ifdef'd out Qt-internal solution (or a QPA default implementation? something along those lines idk). So as you said you attempted and failed, the Mac OSX implementation should internally keep a copy of the dir structure and the public API should remain cross-platform. The only thing needed is a this class consumes more memory/cpu on Mac than on the other platforms. That warning definitely beats a oh by the way this public API performs differently on the Mac message, creating a nightmare for application developers wanting to target multiple platforms. I would compare this to Windows Metro (a specific platform) not including/allowing OpenGL (analogous to efficient recursive directory monitoring), and Qt solving this by using ANGLE (platform-specific workaround). d3fault ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
On Fri, 20 Jul 2012 21:39:32 +0300, Konstantin Ritt wrote:
I would prefer that QFSW was simple and reliable and doesn't attempt
to impose a heavy cost or internal complexity (which translates into
bugs) for things that the underlying APIs do not really support
Same opinion here.
Since we could get a good solution for windows only, the
QRecursiveFileSystemWatcher class goes to be not really
cross-platform.
not much user code needed, if you just follow the signal suggestions
from Robin.
void onWatcherPathCreated(const QString path) {
watcher.addPath(path);}
the rest can be default implementations without performance cost (e.g.
unwatch on delete)
the real culprit is
http://doc-snapshot.qt-project.org/5.0/qfilesystemwatcher.html#directoryChanged
which leads to a lot of ugly code in the signal handler, crying for a
better api.
It's also wrong, sometimes firing at pretty bad timing because the
semantics of what actually changed are lost.
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
With FSEvents you cannot emit signals such as fileChanged(), fileModified(), pathCreated(), or pathDeleted() if all you're notified about is that something happened in this folder here, From my experience building an app that 'watches' a folder and imports all files of a certain type added there, I would prefer that QFSW was simple and reliable and doesn't attempt to impose a heavy cost or internal complexity (which translates into bugs) for things that the underlying APIs do not really support - leaving it up to the app to decide how best to handle these limitations. As you say, applications which are watching (possibly large) directory trees will likely have their own database of content which they will compare any changes to. The app can optimize this in domain-specfic ways, such as not recording any information about types of files it doesn't care about. Here is the FSEvents wrapper we use on Mac for example: https://gist.github.com/3149811 If the underlying APIs for fine-grained notifications and broad file system watches are quite different (as is the case for Mac), I think it makes sense not to try and 'paper over' that in the Qt APIs, so providing different classes would be better. Would be easiest to implement, we could even drop all the threading currently there now. There is a lot to be said for internal simplicity, especially where threading is concerned. Regards, Rob. On 20 July 2012 06:28, logic.cpp logic@gmail.com wrote: #1 Clarification of original post: When I say Making QFileSystemWatcher Recursive I mean *adding* new functions for recursive monitoring, such as addPathRecursively(). I don't mean to replace the current non-recursive functinality, which is very useful in its own right (for A category use cases mentioned). ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
I would prefer that QFSW was simple and reliable and doesn't attempt to impose a heavy cost or internal complexity (which translates into bugs) for things that the underlying APIs do not really support Same opinion here. Since we could get a good solution for windows only, the QRecursiveFileSystemWatcher class goes to be not really cross-platform. Where the amount of nested levels is relatively small, the QFileSystemWatcher could be used for recursive watching manually instead - in a cross-platform way. P.S. I'm still interested in improving the QFSW windows backend by using ReadDirectoryChanges WinAPI instead of thread pool of waiters/listeners. regards, Konstantin 2012/7/20 Robert Knight robertkni...@gmail.com: With FSEvents you cannot emit signals such as fileChanged(), fileModified(), pathCreated(), or pathDeleted() if all you're notified about is that something happened in this folder here, From my experience building an app that 'watches' a folder and imports all files of a certain type added there, I would prefer that QFSW was simple and reliable and doesn't attempt to impose a heavy cost or internal complexity (which translates into bugs) for things that the underlying APIs do not really support - leaving it up to the app to decide how best to handle these limitations. As you say, applications which are watching (possibly large) directory trees will likely have their own database of content which they will compare any changes to. The app can optimize this in domain-specfic ways, such as not recording any information about types of files it doesn't care about. Here is the FSEvents wrapper we use on Mac for example: https://gist.github.com/3149811 If the underlying APIs for fine-grained notifications and broad file system watches are quite different (as is the case for Mac), I think it makes sense not to try and 'paper over' that in the Qt APIs, so providing different classes would be better. Would be easiest to implement, we could even drop all the threading currently there now. There is a lot to be said for internal simplicity, especially where threading is concerned. Regards, Rob. On 20 July 2012 06:28, logic.cpp logic@gmail.com wrote: #1 Clarification of original post: When I say Making QFileSystemWatcher Recursive I mean *adding* new functions for recursive monitoring, such as addPathRecursively(). I don't mean to replace the current non-recursive functinality, which is very useful in its own right (for A category use cases mentioned). ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] QFileSystemWatcher and Recursive Monitoring
snip, please share thoughts on QFileSystemWatcher (maybe monitor-recursive) As you say, applications which are watching (possibly large) directory trees will likely have their own database of content which they will compare any changes to. The app can optimize this in domain-specfic ways, such as not recording any information about types of files it doesn't care about. I'd like to kind-of-disagree here: IDEs like QtCreator or any-other-developer-IDE would need to monitor large directory trees, and may not rely upon their own databases. It would be nice to have this recursive-monitoring for things like build-system-daemons. The goal is central-notification-from-many-large-file-system-roots, and that seems like a fairly useful abstraction that could be used by many application-specific tools. Really good work, IMHO, comparing about the cross-platform issues. Bummer about the Mac. It would be curious if that were to change going-forward (as the future is larger hard drives, and more reliance upon files as everybody goes SSD-fast). I'd *prefer* a QFileSystemWatcherRecursive implementation, even if it could not be cross-platform (e.g., problems on the Mac). I understand why that's not preferred -- we want to get away from system-specific anomaly. Mostly, I'd hope that over time, there would be a technical solution to get it working on the Mac, but at least let people could start using it on Windows/Linux. I could be talked out of this, though. Just my $0.02. --charley ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
[Development] QFileSystemWatcher and Recursive Monitoring
BH Hello, Hope you don't mind my less-waste-your-time more-get-to-the-point English. Begin ramble: Wanted to fix QFileSystemWatcher for Windows as per w00t's article: http://blog.rburchell.com/2012/03/qt-51-aka-when-qfilesystemwatcher-might.html (also see his fix: https://codereview.qt-project.org/#change,19274 ) Realized that what I really needed was *recursive* monitoring, entire trees, which the WinAPI function ReadDirectoryChangesW http://msdn.microsoft.com/en-us/library/windows/desktop/aa365465(v=vs.85).aspx that was mentioned in w00t's article offers right out of the box. Others in #qt-labs also believe recursive file-system monitoring to be a useful feature to offer in Qt. Here is my research on the topic: 1. Status of file-system monitoring on all major platforms. Windows --- API: ReadDirectoryChangesW Recursive: yes Granular info about what changed: yes Linux/X11 - API: inotify Recursive: no, but can be achieved manually* Granular info about what changed: yes *I like to refer to the Gnome Tracker project (and probably also the KDE Nepomuk project), they manually traverse spawn another inotify monitor for each every single dir in entire trees. They also claim that it doesn't hurt performance too much either. It's the best you can currently get, there's currently no other way of doing it. MacOSX -- API: Kqueue Recursive: no, and cannot be achieved manually* Granular info about what changed: yes * Max 256 file descriptors anyway, so out of question for recursive purposes. -- API: FSEvents Recursive: yes Granular info about what changed: no, but you are notified about exactly where the change happened (leaf-most directory, direct parent of where change happened) so at most you need to manually poll contents of just 1 directory. Also multiple changes are compounded into 1 event, configurable. MUST read: http://arstechnica.com/apple/2007/10/mac-os-x-10-5/7/ 2. Problems with making QFileSystemWatcher recursive - Windows ReadDirectoryCHangesW is the winner here. Would be easiest to implement, we could even drop all the threading currently there now. - inotify gives us all the chills at the thought of manually generating watches recursively for entire directory trees, but it can be done satisfactorially enough behind the scenes inside QFileSystemWatcher. - Mac OS is the troublemaker here; Kqueue is out of question, as we established. With FSEvents you cannot emit signals such as fileChanged(), fileModified(), pathCreated(), or pathDeleted() if all you're notified about is that something happened in this folder here, you MUST have some sort of snapshot of the folder to compare against, and QFileSystemWatcher generating an in-memory snapshot of the user's requested directories to be watched can be crazy time consuming and memory draining. Try doing this with an entire drive of some 40,000 folders and 300K files. 3. What are the use cases anyway? Reevaluate: what is file-system watching good for anyway? There are two general use-case categories I can think of: A) Your everyday lightweight necessities, like has the config file changed. QFileSystemWarcher is just fine for this with inotify kqueue (though Windows can still use some ReadDirectoryChangesW love instead of the current use of Find*ChangeNotification APIs). B) Heavy duty monitoring of entire hierachies, like monitoring dir trees in an image library, backup applications, file crawling / indexing, virus scanning etc. If you think about it, I suspect all B-style applications can be expected to have their own snapshot of the file system stored anyway. Any app that needs to monitor changes to THAT much stuff probably has it's own view of the file system that it just wants to keep updated. If this assumption is acceptable, we discover that granular info about what exactly changed is (useful but) not too urgent after all. So maybe FSEvents on mac isn't so bad. The application will only need to compare 1 folder contents with it's own snapshot to determine what changed. So now what; If we care to offer recursive FS monitoring in Qt, we can either try to stuff it into QFileSystemWatcher, or make a new class for it (perhaps platform specific API classes if necessary). 4. The first option: Enhanced QFileSystemWatcher If we do it this way, we'll one single awesome class that does it all. We just may have to make peace with varying usage/behavior on different platforms. If this is acceptable, we're in business. For example, here are some things we might need to mention in the docs; QFileSystemWatcher::fileModified() NOTE: This signal is not emitted on Mac OS X. Instead, directoryChanged() is emitted and the user is expected to investigate further about what exactly changed. (Believe me I tried developing a solution where even on Mac with FSEvents QFileSystemWatcher would do this investigation behind the scenes. I failed). QFileSystemWatcher::addPathRecursively() NOTE: this can take some time on
Re: [Development] QFileSystemWatcher and Recursive Monitoring
#1 Clarification of original post: When I say Making QFileSystemWatcher Recursive I mean *adding* new functions for recursive monitoring, such as addPathRecursively(). I don't mean to replace the current non-recursive functinality, which is very useful in its own right (for A category use cases mentioned). ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
