Sounds like the only way to remove these will be to export, process yourself to remove all events for deleted items and re-import them.
On Jun 25, 2017, at 8:08 AM, Pat Ferrel <[email protected]> wrote: File a bug on PredictionIO with this info. the db-cleanger template just uses internal PIO APIs. On Jun 25, 2017, at 12:07 AM, [email protected] <mailto:[email protected]> wrote: I just tested it and found it wasn't as good as expected. Maybe it's a bug. TEST: [input-events] (python-SDK create_event / delete_item api) [ { "eventId": "jMame_rld_XelHmDsvlZQwAAAVza1NQAk7Kw94kNGnE", "event": "$set", "entityType": "item", "entityId": "item-1", "properties": { "gender": "male" }, "eventTime": "2017-06-25T00:00:00.000+08:00", "creationTime": "2017-06-25T06:41:23.174Z" }, { "eventId": "jMame_rld_XelHmDsvlZQwAAAVza1NQAn_o4ammNV0Y", "event": "$set", "entityType": "item", "entityId": "item-1", "properties": { "name": "alice", "id": "123" }, "eventTime": "2017-06-25T00:00:00.000+08:00", "creationTime": "2017-06-25T06:30:25.362Z" }, { "eventId": "jMame_rld_XelHmDsvlZQwAAAVzd9vAtgElY5zCAd68", "event": "$delete", "entityType": "item", "entityId": "item-1", "properties": {}, "eventTime": "2017-06-25T06:36:07.085Z", "creationTime": "2017-06-25T06:36:07.093Z" } ] >> then run "pio train -v test.json" under db-cleaner-master directory PS: compressProperties:true, removeDuplicates:true [after-train] [ { "eventId": "jMame_rld_XelHmDsvlZQwAAAVza1NQAk7Kw94kNGnE", "event": "$set", "entityType": "item", "entityId": "item-1", "properties": { "gender": "male", "name": "alice", "id": "123" }, "eventTime": "2017-06-25T00:00:00.000+08:00", "creationTime": "2017-06-25T00:00:00.000+08:00" }, { "eventId": "jMame_rld_XelHmDsvlZQwAAAVzd9vAtgElY5zCAd68", "event": "$delete", "entityType": "item", "entityId": "item-1", "properties": {}, "eventTime": "2017-06-25T06:36:07.085Z", "creationTime": "2017-06-25T06:36:07.085Z" } ] the props compressed but the item-1 and its '$delete' event is still there. 在 2017年6月25日星期日 UTC+8上午12:31:18，pat写道： Hmm, actually not sure it that is true. It would take an experiment, create an item with $set, $delete it and both events will be in the PIO EventServer. Then run db-cleaner with compression turned on and see if the events are still there. If they are, this is a bug in PIO, so you should create a bug report there. On Jun 22, 2017, at 8:33 AM, Pat Ferrel <[email protected] <javascript:>> wrote: The slow accumulation of your $set events (actually the objects they create—items) can be reversed by sending $delete when you know the item is not longer available. On Jun 21, 2017, at 7:09 PM, [email protected] <javascript:> wrote: thank you for your reply, pat. I have used db-cleaner to control the scale of the data, but since reserved event(e.g. $set) will not be deleted, so those item-ids associated with reserved event will always exist in the ES model (even those item-ids will never become the target of any users). will those "meaningless" item-ids (from reserved event) affect the resource requirements when training URModel (e.g. consturct BiMap) ? or does it just index those item-ids (from reserved event) to ES model if there are no named events(e.g purchace) on it ？ 在 2017年6月22日星期四 UTC+8上午12:25:50，pat写道： Yes, but the db-cleaner template and the SelfCleaningDatasource have an option to compress duplicates and create one aggregate $set from many changes. For highly constrained environments I’d run db-cleaner periodically to keep training time constant. https://github.com/actionml/db-cleaner <https://github.com/actionml/db-cleaner> On Jun 21, 2017, at 3:57 AM, [email protected] <> wrote: Hi~ As far as I know, the number of user-ids, #item-ids and #event-type will affect the training model of memory requirements, so I control the scale of the data through the eventwindow, but because "$set property change events are never dropped", does it mean that #item-ids will continue to increase , the memory requirements will be more and more ? (in my case, items expired after 3 days) -- You received this message because you are subscribed to the Google Groups "actionml-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected] <>. To post to this group, send email to [email protected] <>. To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/fac288e9-d9ff-4074-83a4-96364eb57154%40googlegroups.com <https://groups.google.com/d/msgid/actionml-user/fac288e9-d9ff-4074-83a4-96364eb57154%40googlegroups.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "actionml-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected] <javascript:>. To post to this group, send email to [email protected] <javascript:>. To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/1bd5f580-92e8-42a2-a484-7ce439cddb84%40googlegroups.com <https://groups.google.com/d/msgid/actionml-user/1bd5f580-92e8-42a2-a484-7ce439cddb84%40googlegroups.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "actionml-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected] <javascript:>. To post to this group, send email to [email protected] <javascript:>. To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/5A7D400F-749B-46A7-8F69-FA85B38272CE%40occamsmachete.com <https://groups.google.com/d/msgid/actionml-user/5A7D400F-749B-46A7-8F69-FA85B38272CE%40occamsmachete.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "actionml-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected] <mailto:[email protected]>. To post to this group, send email to [email protected] <mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/a00ad70d-1783-402f-972d-f75541636017%40googlegroups.com <https://groups.google.com/d/msgid/actionml-user/a00ad70d-1783-402f-972d-f75541636017%40googlegroups.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "actionml-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected] <mailto:[email protected]>. To post to this group, send email to [email protected] <mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/B177BA03-62B3-478B-BAC2-21A33CFCBA6F%40occamsmachete.com <https://groups.google.com/d/msgid/actionml-user/B177BA03-62B3-478B-BAC2-21A33CFCBA6F%40occamsmachete.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
