Hi Dominik, Ok. We also need a strategy on deleting data from the data lake. I think it makes sense to define a retention time per persisted stream.
Additionally we should discuss when the data is deleted. Currently we only have a beta feature to clean all data in the database, which is not sufficient. One of the main questions is: Should the data be kept in the database when the pipeline / source is deleted? From a user point of view it might make sense to remove all corresponding data when deleting a pipeline. But I often use this feature to collect some (high frequency) data for some time and remove the pipeline, but I want to keep the data to analyze it later (e.g. train a machine learning model). Any thoughts on that? Philipp On 2021/01/18 21:56:21, Dominik Riemer <[email protected]> wrote: > Hi Philipp, > > to be honest, I don't know yet - but I'd say there will be some data model > that links a data stream to the corresponding index. This is probably similar > to how we currently handle "visualizable pipelines". > > Dominik > > On 2021/01/18 21:35:20, Philipp Zehnder <[email protected]> wrote: > > Hi, > > very cool, the API is now much more clear. > > > > How will we handle the read of data from the data lake in the future? > > Currently the index name is used, but I guess we will change this to the > > stream id, > > right? How will this look like? > > > > Philipp > > > > Am 18.01.2021 um 08:33 schrieb Dominik Riemer <[email protected]>: > > > > Hi all, > > > > > > > > I did some changes to the data explorer management and extracted all queries > > for historic data to an individual module (streampipes-data-explorer). While > > doing this, I also split the previously existing DataLakeManagementv3 class > > into several smaller units to make it more readable. > > > > The change is currently in the STREAMPIPES-272 branch [1], but I think we > > can merge this soon for the next release. > > > > > > > > Dominik > > > > > > > > [1] > > https://github.com/apache/incubator-streampipes/tree/STREAMPIPES-272/streamp > > ipes-data-explorer/src/main/java/org/apache/streampipes/dataexplorer > > > > > > > > >
