Le sam. 16 nov. 2019 à 11:06, Amirouche Boubekki <[email protected]> a écrit : > > I restarted working on my personal search engine. > > It used to be called culturia [0] with too many planned features. At > some point, I called it asylum [1] and focused on personal knowledge > base aspects and the last iteration was called gotofish [2] > > [0] https://framagit.org/a-guile-mind/culturia > [1] https://framagit.org/a-guile-mind/culturia.next > [2] https://git.sr.ht/~amz3/guile-gotofish > > I learned much from all this projects. In particular, I learned that > it will be a long long long project, even if I focus only on "personal > search engine" line of work. > > The last iteration, gotofish, was not too bad even if it has bitrot. > Based on my research and practical experiment, it seems very clear > that there is no workaround the use of map-reduce, that might be known > as n-par-for-each [3]. > > [3] > https://www.gnu.org/software/guile/manual/html_node/Parallel-Forms.html#index-n_002dpar_002dfor_002deach > > I made a prototype similar to that n-par-for-each, except it works > with guile-fibers, is asynchronous and works with a shared pool of > threads instead of spawning N threads for each incoming query like > gotofish does. > > Related blog post: https://hyper.dev/blog/on-the-road-to-babelia.html > > If you want to help or discuss those matters, do not hesitate to reply > to this message.
I forgot to add that there is several big-ish tasks that can be tackled in parallel (see the above blog post). In particular, a parser for wet or warc files, see https://en.wikipedia.org/wiki/Web_ARChive. This is the most common format of the output of crawlers e.g. http://commoncrawl.org/
