No, I don't. The program should be for analysing logs (collected by fluentd) - should be open source and on github, however there isn't much done yet: https://github.com/logTank/
The index rebuilding shouldn't be a problem as CouchDB will be only used for general stats and the user actually won't see the up to date data, but always with a delay - another advantage of CouchDB, I can read the queries without bothering the system, and once the data is outdated, I can update the index. At least so far the theory, I'll need to run some performance tests if that actually works, once I'll have a MVP. The other option is to use MongoDB for ad-hoc queries, but I was thinking that CouchDB will be more efficient as storage is so cheap. As I'm learning every time I look up info about CouchDB something new, and something becomes more clear, I'm also glad about feedback on the idea in general, how I want to use CouchDB. However I'd be also very happy if I could somehow solve the problem with the possible DoS attacks :). Maybe there is something in CouchDB or evalcx which I can configure - maximal runtime of a map/reduce function? (shouldn't be more than 1ms). Or there are some logged data by CouchDB about the resources required by views (CPU Time + HDD Space)? Cheers Peter On Fri Nov 28 2014 at 2:54:51 AM Alexander Gabriel <[email protected]> wrote: > sorry for being off-topic > Alex > > > 2014-11-28 2:52 GMT+01:00 Alexander Gabriel <[email protected]>: > > > sounds like a very interesting application > > > > seems like you dont care if the user has to wait for an index to be built > > when the user creates a query > > > > Alex > > > > > > 2014-11-28 2:23 GMT+01:00 Peter Grman <[email protected]>: > > > >> Hi Alex, > >> > >> Yes, the users would be able to import different sets of data, which > isn't > >> relational, and use the platform to analyse it. The analysed data would > be > >> in 99% of the cases append only (+ removing old data) and the data can > be > >> defined by the user, as well as be hierarchical. > >> > >> When I thought about the system in the beginning, CouchDB seemed like an > >> awesome choice as there would be only a couple of well defined queries > and > >> storage is generally cheap, I thought that CouchDB views and their > caching > >> are what I'm looking for. > >> > >> The problem is again only with people who want to trick the system. I > >> would > >> be also happy with a solution which would detect bad views ones they > have > >> been deployed (uses too much space, takes too long to compute) and > >> deactivates and marks them for me to check. This way I could check those > >> few people who try a DoS attack and ban them from the service. > >> > >> The additional main problem was, if it is really impossible to get data > >> from a different database inside the view and if the user won't be able > to > >> access the underlying system, ..., or if it is just very difficult => > >> possible, if someone wants to do it they'll find a way. But after > reading > >> more and understanding more, how the views are executed using evalcx I > >> think the other problems aren't a big concern for me anymore, is that > >> correct?. > >> > >> Although I've found in the code "if possible, use evalcx (not always > >> available)" - how can I check that evalcx is available on my system? Or > is > >> it just a note for older distributions, nothing to be concerned about > >> anymore? > >> > >> Thank you > >> > >> Cheers > >> Peter > >> > >> On Fri Nov 28 2014 at 1:37:57 AM Alexander Gabriel <[email protected]> > >> wrote: > >> > >> > Hi Peter > >> > > >> > Will the users create their own datastructures too? > >> > If not this sounds like sql on relational tables might be a better > tool > >> for > >> > the problem. > >> > It seems to me you're hitting exactly the weak point of most nosql > >> > solutions. > >> > > >> > Alex > >> > > >> > > >> > 2014-11-28 0:49 GMT+01:00 Peter Grman <[email protected]>: > >> > > >> > > Hi, > >> > > > >> > > this might sound like a terrible idea to someone who knows CouchDB, > >> and > >> > if > >> > > that's the case, please just take a minute or two, to explain why, > >> > > otherwise, if the idea isn't so crazy after all, I hope I'll get > some > >> > > solutions to my problem: > >> > > > >> > > I'm thinking of creating a platform based on CouchDB, where each set > >> of > >> > > users (group, customer, ...) would get their own CouchDB Database, > to > >> > store > >> > > and query data. I've heard in a podcast, roughly a year ago, that > >> this is > >> > > how CouchDB was meant to be - many smaller databases. > >> > > > >> > > To query the data, I want to allow them, to define their own custom > >> > > queries. Now I could (and want to) create a form which allows to > >> build a > >> > > query and translates it to a JS view, but I was thinking about > >> > > additionally, on top of that, allowing them to define their custom > >> views > >> > > directly in JS. They would basically be allowed to define their > custom > >> > > Map/Reduce functions. > >> > > > >> > > There is a lot which can go wrong with this the worst ones I came up > >> > with: > >> > > - DoS attack with endless loops inside the function > >> > > - DoS attack by emitting too much data (potentially in a loop again) > >> > > > >> > > As far as I've understood, it's not possible to access other > Databases > >> > from > >> > > within the view, is this understanding of mine correct? > >> > > > >> > > Is it possible to access the filesystem or network services in any > way > >> > from > >> > > the CouchDB view or is the JavaScript engine, which is running the > >> code, > >> > > limiting enough? > >> > > > >> > > Are there any other things which could go wrong? - or did actually > >> > somebody > >> > > already use CouchDB like this, and it's perfectly normal? > >> > > > >> > > Is there any way I could prevent the problem with endless loops and > >> data > >> > > emitting from happening? - I can run JSLint, which maybe will detect > >> an > >> > > endless loop, but that won't help against a loop with a million > >> > iterations, > >> > > which will be called for every item inside CouchDB - still quite > >> endless. > >> > > > >> > > Thank you for your help! > >> > > > >> > > Cheers, > >> > > Peter > >> > > > >> > > >> > > > > >
