Good morning all,
In my experience, in almost each Syncope project I worked on, I had to
face sooner or later with the huge size of the tables storing
propagation tasks and their executions (TASK and TASKEXEC). Despite the
possibility to tune execution storage and verbosity, in the first phases
of the projects (in test or even in production mode) it is useful to set
the most verborse trace level for CRUD operations and sometimes also for
pull/push ones. This because information stored (mainly) into the
propagation tasks executions are very useful for debugging purposes and
to understand what happened in case of errors, but also in case of
successful operations.
Thus most of the times, due to the too large size of TASK and TASKEXEC
tables, we are forced to clean up them (usually by date) to speed up
queries on propagation tasks and to make the console section that shows
resource propagation tasks usable.
Moreover in some cases it is very useful to analyze execution details,
that for propagation tasks, for example, contain the payload (in JSON
format) sent to the resource. At the moment there isn't the possibility
to parse and to make granular queries on task executions.
This said, I would suggest to add to Syncope the possibility to manage
such information, at least the ones into task executions (TASKEXEC
table) as big data or, generally speaking, as no SQL information. The
nearest solution could be the use of Elasticsearch as a possible
storage, mainly because it has successfuly been embedded into Syncope
[1] and is easily queriable (also through a free console as Kibana [2]).
Obviously no-SQL world is wide, so we can consider usign another
technology as MongoDB, a graph database, ecc. But, sincerely speaking, I
do not see the need to add another technology and dependency(ies) to the
project.
Lastly, I'm not suggesting to implement it right now, but at least to
discuss about to add this feature to the roadmap or not.
WDYT?
Best regards,
Andrea
[1] https://syncope.apache.org/docs/2.1/reference-guide.html#elasticsearch
[2] https://www.elastic.co/kibana
--
Dott. Andrea Patricelli
Tel. +39 3204524292
Engineer @ Tirasa S.r.l.
Viale Vittoria Colonna 97 - 65127 Pescara
Tel +39 0859116307 / FAX +39 0859111173
http://www.tirasa.net
Apache Syncope PMC Member
