[jira] [Comment Edited] (FLINK-3679) DeserializationSchema should handle zero or more outputs for every input
[ https://issues.apache.org/jira/browse/FLINK-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849633#comment-15849633 ] Tzu-Li (Gordon) Tai edited comment on FLINK-3679 at 2/2/17 8:22 AM: Hi [~wheat9]! Thank you for picking this JIRA up. How are you doing with this work? -Since the previous discussion didn't really come to a conclusion on the API changes for this feature yet, can you briefly describe how you plan to add this? We might need to be extra careful in how the new API works with the state update locking in the consumer.- (Sorry, I just realized you have some proposals already in FLINK-5583. I'll move the API discussion there.) Please also feel free to call out to us if you want to jump around some ideas or bump into any problems for this. was (Author: tzulitai): Hi [~wheat9]! -Thank you for picking this JIRA up. How are you doing with this work? Since the previous discussion didn't really come to a conclusion on the API changes for this feature yet, can you briefly describe how you plan to add this? We might need to be extra careful in how the new API works with the state update locking in the consumer.- (Sorry, I just realized you have some proposals already in FLINK-5583. I'll move the API discussion there.) Please also feel free to call out to us if you want to jump around some ideas or bump into any problems for this. > DeserializationSchema should handle zero or more outputs for every input > > > Key: FLINK-3679 > URL: https://issues.apache.org/jira/browse/FLINK-3679 > Project: Flink > Issue Type: Bug > Components: DataStream API, Kafka Connector >Reporter: Jamie Grier >Assignee: Haohui Mai > > There are a couple of issues with the DeserializationSchema API that I think > should be improved. This request has come to me via an existing Flink user. > The main issue is simply that the API assumes that there is a one-to-one > mapping between input and outputs. In reality there are scenarios where one > input message (say from Kafka) might actually map to zero or more logical > elements in the pipeline. > Particularly important here is the case where you receive a message from a > source (such as Kafka) and say the raw bytes don't deserialize properly. > Right now the only recourse is to throw IOException and therefore fail the > job. > This is definitely not good since bad data is a reality and failing the job > is not the right option. If the job fails we'll just end up replaying the > bad data and the whole thing will start again. > Instead in this case it would be best if the user could just return the empty > set. > The other case is where one input message should logically be multiple output > messages. This case is probably less important since there are other ways to > do this but in general it might be good to make the > DeserializationSchema.deserialize() method return a collection rather than a > single element. > Maybe we need to support a DeserializationSchema variant that has semantics > more like that of FlatMap. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (FLINK-3679) DeserializationSchema should handle zero or more outputs for every input
[ https://issues.apache.org/jira/browse/FLINK-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849633#comment-15849633 ] Tzu-Li (Gordon) Tai edited comment on FLINK-3679 at 2/2/17 8:22 AM: Hi [~wheat9]! -Thank you for picking this JIRA up. How are you doing with this work? Since the previous discussion didn't really come to a conclusion on the API changes for this feature yet, can you briefly describe how you plan to add this? We might need to be extra careful in how the new API works with the state update locking in the consumer.- (Sorry, I just realized you have some proposals already in FLINK-5583. I'll move the API discussion there.) Please also feel free to call out to us if you want to jump around some ideas or bump into any problems for this. was (Author: tzulitai): Hi [~wheat9]! -Thank you for picking this JIRA up. How are you doing with this work? Since the previous discussion didn't really come to a conclusion on the API changes for this feature yet, can you briefly describe how you plan to add this? We might need to be extra careful in how the new API works with the state update locking in the consumer.- (Sorry, I just realized you have some proposals already in ) Please also feel free to call out to us if you want to jump around some ideas or bump into any problems for this. > DeserializationSchema should handle zero or more outputs for every input > > > Key: FLINK-3679 > URL: https://issues.apache.org/jira/browse/FLINK-3679 > Project: Flink > Issue Type: Bug > Components: DataStream API, Kafka Connector >Reporter: Jamie Grier >Assignee: Haohui Mai > > There are a couple of issues with the DeserializationSchema API that I think > should be improved. This request has come to me via an existing Flink user. > The main issue is simply that the API assumes that there is a one-to-one > mapping between input and outputs. In reality there are scenarios where one > input message (say from Kafka) might actually map to zero or more logical > elements in the pipeline. > Particularly important here is the case where you receive a message from a > source (such as Kafka) and say the raw bytes don't deserialize properly. > Right now the only recourse is to throw IOException and therefore fail the > job. > This is definitely not good since bad data is a reality and failing the job > is not the right option. If the job fails we'll just end up replaying the > bad data and the whole thing will start again. > Instead in this case it would be best if the user could just return the empty > set. > The other case is where one input message should logically be multiple output > messages. This case is probably less important since there are other ways to > do this but in general it might be good to make the > DeserializationSchema.deserialize() method return a collection rather than a > single element. > Maybe we need to support a DeserializationSchema variant that has semantics > more like that of FlatMap. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (FLINK-3679) DeserializationSchema should handle zero or more outputs for every input
[ https://issues.apache.org/jira/browse/FLINK-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849633#comment-15849633 ] Tzu-Li (Gordon) Tai edited comment on FLINK-3679 at 2/2/17 8:22 AM: Hi [~wheat9]! -Thank you for picking this JIRA up. How are you doing with this work? Since the previous discussion didn't really come to a conclusion on the API changes for this feature yet, can you briefly describe how you plan to add this? We might need to be extra careful in how the new API works with the state update locking in the consumer.- (Sorry, I just realized you have some proposals already in ) Please also feel free to call out to us if you want to jump around some ideas or bump into any problems for this. was (Author: tzulitai): Hi [~wheat9]! Thank you for picking this JIRA up. How are you doing with this work? Since the previous discussion didn't really come to a conclusion on the API changes for this feature yet, can you briefly describe how you plan to add this? We might need to be extra careful in how the new API works with the state update locking in the consumer. Please also feel free to call out to us if you want to jump around some ideas or bump into any problems for this. > DeserializationSchema should handle zero or more outputs for every input > > > Key: FLINK-3679 > URL: https://issues.apache.org/jira/browse/FLINK-3679 > Project: Flink > Issue Type: Bug > Components: DataStream API, Kafka Connector >Reporter: Jamie Grier >Assignee: Haohui Mai > > There are a couple of issues with the DeserializationSchema API that I think > should be improved. This request has come to me via an existing Flink user. > The main issue is simply that the API assumes that there is a one-to-one > mapping between input and outputs. In reality there are scenarios where one > input message (say from Kafka) might actually map to zero or more logical > elements in the pipeline. > Particularly important here is the case where you receive a message from a > source (such as Kafka) and say the raw bytes don't deserialize properly. > Right now the only recourse is to throw IOException and therefore fail the > job. > This is definitely not good since bad data is a reality and failing the job > is not the right option. If the job fails we'll just end up replaying the > bad data and the whole thing will start again. > Instead in this case it would be best if the user could just return the empty > set. > The other case is where one input message should logically be multiple output > messages. This case is probably less important since there are other ways to > do this but in general it might be good to make the > DeserializationSchema.deserialize() method return a collection rather than a > single element. > Maybe we need to support a DeserializationSchema variant that has semantics > more like that of FlatMap. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (FLINK-3679) DeserializationSchema should handle zero or more outputs for every input
[ https://issues.apache.org/jira/browse/FLINK-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849633#comment-15849633 ] Tzu-Li (Gordon) Tai edited comment on FLINK-3679 at 2/2/17 8:16 AM: Hi [~wheat9]! Thank you for picking this JIRA up. How are you doing with this work? Since the previous discussion didn't really come to a conclusion on the API changes for this feature yet, can you briefly describe how you plan to add this? We might need to be extra careful in how the new API works with the state update locking in the consumer. Please also feel free to call out to us if you want to jump around some ideas or bump into any problems for this. was (Author: tzulitai): Hi [~wheat9]! Thank you for picking this JIRA up. How are you doing with this work? Since the previous discussion didn't really come to a conclusion on the API changes for this feature yet, can you briefly describe how you plan to add this? Please also feel free to call out to us if you want to jump around some ideas or bump into any problems for this. > DeserializationSchema should handle zero or more outputs for every input > > > Key: FLINK-3679 > URL: https://issues.apache.org/jira/browse/FLINK-3679 > Project: Flink > Issue Type: Bug > Components: DataStream API, Kafka Connector >Reporter: Jamie Grier >Assignee: Haohui Mai > > There are a couple of issues with the DeserializationSchema API that I think > should be improved. This request has come to me via an existing Flink user. > The main issue is simply that the API assumes that there is a one-to-one > mapping between input and outputs. In reality there are scenarios where one > input message (say from Kafka) might actually map to zero or more logical > elements in the pipeline. > Particularly important here is the case where you receive a message from a > source (such as Kafka) and say the raw bytes don't deserialize properly. > Right now the only recourse is to throw IOException and therefore fail the > job. > This is definitely not good since bad data is a reality and failing the job > is not the right option. If the job fails we'll just end up replaying the > bad data and the whole thing will start again. > Instead in this case it would be best if the user could just return the empty > set. > The other case is where one input message should logically be multiple output > messages. This case is probably less important since there are other ways to > do this but in general it might be good to make the > DeserializationSchema.deserialize() method return a collection rather than a > single element. > Maybe we need to support a DeserializationSchema variant that has semantics > more like that of FlatMap. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (FLINK-3679) DeserializationSchema should handle zero or more outputs for every input
[ https://issues.apache.org/jira/browse/FLINK-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224692#comment-15224692 ] Jamie Grier edited comment on FLINK-3679 at 4/4/16 6:12 PM: I'm not sure about the locking and operator chaining issues so I would say if that's unduly complicated because of this change maybe it's not worth it. However, a DeserializationSchema with more flatMap() like semantics would certainly be the better API given that bad data issues are a reality. It also seems we could provide this without breaking existing code, but certainly it would add a bit more complexity to the API (having multiple variants for this). Anyway, I agree you can work around this issue my making a special "sentinel" value and dealing with all of this is in a chained flatMap() operator. I imagine that's exactly the approach that people are already using. was (Author: jgrier): I'm not sure about the locking and operator chaining issues so I would say if that's unduly complicated because of this change maybe it's not worth it. However, a DeserializationSchema with more flatMap() like semantics would certainly the better API given that bad data issues are a reality. It also seems we could provide this without breaking existing code, but certainly it would add a bit more complexity to the API (having multiple variants for this). Anyway, I agree you can work around this issue my making a special "sentinel" value and dealing with all of this is in a chained flatMap() operator. I imagine that's exactly the approach that people are already using. > DeserializationSchema should handle zero or more outputs for every input > > > Key: FLINK-3679 > URL: https://issues.apache.org/jira/browse/FLINK-3679 > Project: Flink > Issue Type: Bug > Components: DataStream API >Reporter: Jamie Grier > > There are a couple of issues with the DeserializationSchema API that I think > should be improved. This request has come to me via an existing Flink user. > The main issue is simply that the API assumes that there is a one-to-one > mapping between input and outputs. In reality there are scenarios where one > input message (say from Kafka) might actually map to zero or more logical > elements in the pipeline. > Particularly important here is the case where you receive a message from a > source (such as Kafka) and say the raw bytes don't deserialize properly. > Right now the only recourse is to throw IOException and therefore fail the > job. > This is definitely not good since bad data is a reality and failing the job > is not the right option. If the job fails we'll just end up replaying the > bad data and the whole thing will start again. > Instead in this case it would be best if the user could just return the empty > set. > The other case is where one input message should logically be multiple output > messages. This case is probably less important since there are other ways to > do this but in general it might be good to make the > DeserializationSchema.deserialize() method return a collection rather than a > single element. > Maybe we need to support a DeserializationSchema variant that has semantics > more like that of FlatMap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
