sheepdreamofandroids commented on issue #22840: URL: https://github.com/apache/beam/issues/22840#issuecomment-1236635253
We're processing JSON events for an online marketplace like creation, modification, publishing, (de-)activating features etc. Lots of private info so I can't share actual info. Depending on type these events go to different indexes and most go to 2 different indexes: one for full history and one where we update the latest version using a scripted upsert. This is to facilitate different styles of searching. While it is possible to determine what the type is from the json itself, it's easier to do it from metadata sent in pubsub. When sending the same json to different indexes though, that is obviously not possible. It is solved now by using the type from the metadata and dispatching each json to one or more subsequent pipelines, one for each permutation of index and upsert. Much nicer would be to simply attach the index and upsert to the json and keep everything in one simple pipeline. The most flexible way would be to be able to send my own type into Write and define a jsonFn and upsertFn to extract from that type. Maybe a GenericWrite (with type parameter) could be defined as superclass of Write, which now uses String for that parameter and defines a default jsonFn. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
