On 26/07/24 11:19, Bergmann, Clemens wrote:
Hi, I am uncertain if this is the right group to ask this question. The connid-users [1] group seems better fitting but is not very responsive so I will ask here. I have a problem with removing entries that the source does no longer has on record We have a source system that we import anyObjects from. That source will deliver CSV file with all entries daily. That file does only contain the entries that they currently have on record. I configured the connector with the following Capabilities: DELETE, UPDATE, SYNC, SEARCH, CREATE, UPDATE_DELTA Then I added a resource with all default parameters and configured the mapping. Next I added a pull task with the following settings: “Pull Mode” set to INCREMENTAL “Matching Rule” set to UPDATE “Unmatching Rule” set to ASSIGN Checked the boxes of “Allow create”, “Allow update”, "Allow delete” and “Sync status” When I execute the pull task all the entries in the file are imported and respective anyObjects are created. I then remove one entry from the file an execute the pull task again. In the result status of the execution I see “UPDATE SUCCESS” for all other entries. The removed entry is not listed any more (as expected) but I see no DELETE for the respective anyObject. I looked into the Source code of the CSVDir bundle [2] and it seems that the DELETE DeltaType is only ever selected when the “Delete Column” is “true”. As we don’t have a delete column in the file that will never happen. Is there something that I am overlooking? How does connID handle the situation of removed entires from resources in general? That seems to be a common use case also for example for database and other connector bundles.
This is expected for all ConnId connectors, because it is implied by the way how the ConnId framework works. This issue should be looked both from ConnId's and Syncope's side. From ConnId connector's point of view, there should be something in the storage (CSV file, database table, changelog if available, etc) which can enable the connector to produce DELETE events as response to the sync() call. For database-like storage (as CSV), the DELETE column seems to be the most sensitive choice: when the sync() call arrives, the CSV file is read and the rows with DELETE column set to true will generate DELETE events. Otherwise said: what is tracking the fact that your source CSV file used to have a row that you have at some point deleted? Other connectors, as AD or LDAP for certain server implementations, when the sync() call arrives will query the server changelog and generate DELETE events for the corresponding changelog entries. On Syncope side, as reported by [3], the incremental pull mode is the only condition which can lead to deleting internal entries, because this is the only way that Syncope can receive DELETE events from a ConnId connector.
If indeed a delete column is needed the only solution I can think of is to prepare the file with a script before importing it with connID. That script could add the delete column with “false” on all existing entries and also add entries that where there in the last file and are no longer there in the current file. These entries could have the delete column set to “true”.
This looks like a wise solution. Regards.
[1] https://groups.google.com/g/connid-users [2] https://github.com/Tirasa/ConnIdCSVDirBundle/blob/master/src/main/java/net/tirasa/connid/bundles/csvdir/methods/CSVDirSync.java#L131
[3] https://syncope.apache.org/docs/3.0/reference-guide.html#pull-mode -- Francesco Chicchiriccò Tirasa - Open Source Excellence http://www.tirasa.net/ Member at The Apache Software Foundation Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail http://home.apache.org/~ilgrosso/
