Héllo,
I made some progress regarding culturia. Now there is a web
interface available at hypermove.net [0].
[0] http://hypermove.net/?query=guile+algorithms+-wingolog
Now you can:
- search a bunch of website related to Guile I quickly selected
- use minus char "-" to exclude some keyword from the result set
- index small-ish domains
There is no UI yet to add your website to the index, if you want
your website to be part of the experiment reach me.
I tried two search backends. One based on a graph database which
is more versatile but much slower and another based on inverted
index. The latter is the one that is used currently. It's fast
enough right now.
What will be done next:
- Add wikipedia, wiktionary, stackoverflow and hackernews to the
database
- Improve how results are displayed
- Remove feeds from results
- Index pdf
- Remove duplicate entries from the results
- Paginate
- Make use of fibers to run the http server
- Better crawling algorithm that support updating the index
At some point there will be a 0.1 release, not sure when.
The code is also available at framagit [1] the web interface
is implemented in a single module [2].
[1] https://framagit.org/a-guile-mind/culturia
[2] https://framagit.org/a-guile-mind/culturia/blob/master/src/web.scm
Happy hacking!
On 2016-08-13 17:25, Amirouche Boubekki wrote:
Héllo,
The goal of Culturia is to create a framework that makes it easy
to tape into Natural Language Understanding algorithms (and NLP)
and provide an interface for common tasks.
Culturia is an intelligence augmentation software.
It's primary interface is a search engine. Another important aspect
of the project is that it wants to be useable offline as such it will
come with infrastructure to dump, load and store dataset for offline
use.
The current state of the project can be described as a big ball of mud.
There is a tiny search engine with crawling skills and that's basically
all of it.
The immediate changes that should happen are in order of preference:
- offline stackoverflow (cf. sotoki.scm) and use the generated
website to create a zim for kiwix [0]. This is great occasion to
show how great GNU Guile is!
- port whoosh/lucene to guile to improve text search
- offline hackernews, wikidata, wikipedia, wiktionary
- implement BM25f
Culturia is a reference to _Culture and Empire_ by Pieter Hintjens.
It has a sparse documentation is available online [1].
It's hosted on github [2] (This can change, if contributors
don't want to use github).
The TODO list is big, here is some stuff that needs to be done:
- finish GrammarLink bindings
- create sophia [3] bindings
- implement TextRank
- implement PageRank
- create a GUI using sly or html
- explore ways to easily share database among several processus
And many other things! Newbies are accepted obviously!
Send me a mail or use #guile @ irc.freenode.net, I am amz3.
Happy hacking!
[0] http://www.kiwix.org/wiki/Main_Page
[1] https://amirouche.github.io/Culturia/doc/
[2] https://github.com/amirouche/Culturia
[3] http://sophia.systems/
--
Amirouche ~ amz3 ~ http://www.hyperdev.fr
