tenthe commented on issue #856:
URL: https://github.com/apache/streampipes/issues/856#issuecomment-1341434055
Hi Stefan,
thanks for opening the Issue.
I think this is a great question. I guess we have to destiguish between the
'guess schema' phase and 'runtime' phase:
- Guess schema:
- If a value is null we have the problem that the data type can not be
inferred. So I would say we have two options, either remove the property, or
notify the user to select a data type manually. (I think I would prefer the
second option, because the user can decide what should happen. This would
require changes in the API and the UI)
- Runtime:
- Here I see three options to deal with missing data. Either remove the
whole event, remove the missing value or provide a default value. I would
prefere the second option. We recently implemented something similar for the
data lake. The problem with this solution is that processing elements expect
that events are complete (have values for all properties). If we start to
remove property values from events, then we need a mechanism to deal with
incomplete events. Therefore, we have to adapt the API for processing elements.
What are your thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
