[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449014#comment-15449014
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2305


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15448334#comment-15448334
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2305
  
Looks good, thanks!

Merging this...


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15448012#comment-15448012
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/2305
  
Hi @aljoscha  @StephanEwen , this issue has been blocked for some days. It 
would be great, if you can  have a look  again ? 


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438733#comment-15438733
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/2305
  
Hi @StephanEwen , comments addressed.  

I'm not sure whether the comments are correct. Let me know if I'm wrong.


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438680#comment-15438680
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2305
  
The methods look good. They need a `@PublicEvolving` annotation and I think 
we should mark them with `@Deprecated` and comment that they are a temporary 
workaround while the `apply()` method has the wrong return type.


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438529#comment-15438529
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/2305
  
Hi @StephanEwen @aljoscha , I have updated this PR with the `with(...)` 
approach , could you have a look at it?

And log FLINK-4503 JIRA for Flink 2.0 breaking changes.


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434875#comment-15434875
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/2305
  
Yes, I would like.  

> 在 2016年8月24日，20:38，Stephan Ewen  写道：
> 
> @wuchong Do you want to update this PR with the with(...) approach and 
file the JIRA for Flink 2.0 breaking changes?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
> 



> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434849#comment-15434849
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2305
  
@wuchong Do you want to update this PR with the `with(...)` approach and 
file the JIRA for Flink 2.0 breaking changes?


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432906#comment-15432906
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2305
  
It seems we are go for the `with(...)` approach.
Pending decision is whether we want `with` to be the long-run solution, or 
stay with `apply`.

The reason why `DataStream` has `apply()` and `DataSet` has `with()` is 
that different people wrote the API functions and everyone has their favorite 
name and style that they stick to ;-)

I agree that consistency should be key in the future. The `DataStream` API 
has more traction right now, and should long-term subsume the DataSet API, so I 
have a slight bias to keep the DataStream style for now (many people will not 
even use the `with(...)` variant because they don't set individual parallelism).


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430385#comment-15430385
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/2305
  
But then we would have to change it for all windowed operators.

Furthermore the advantage of `apply` is that it is consistent with the 
Scala API and Scala's `apply` function treatment where you don't have to 
specify the method name explicitly. 


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428402#comment-15428402
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2305
  
Are there strong reasons to use `apply` for `DataStream` and `with` for 
`DataSet`? Could we deprecate the `apply` so that users having switched to 
`with` will not lose API compatibility with 2.0?


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428190#comment-15428190
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/2305
  
I'm also okay with `with()`


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426214#comment-15426214
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/2305
  
+1 for this approach.

On Thu, Aug 18, 2016 at 6:59 AM, Jark  wrote:

> I'm ok with the with(...) approach. In addition, we should create a JIRA
> under [FLINK-3957]
> "Breaking API changes for Flink 2.0", to keep track of deprecating and
> changing the apply method return type.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425903#comment-15425903
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/2305
  
I'm ok with the `with(...)` approach. In addition, we should create a JIRA 
under [FLINK-3957]
"Breaking API changes for Flink 2.0", to keep track of deprecating  and 
changing the `apply` method return type.


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425006#comment-15425006
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2305
  
I would like to come to an agreement with everyone how to proceed.

From the discussion on the mailing list, I see that quite some people are 
opposed to breaking the compatibility unless we name the next release 2.0. 
Since that has not been decided upon, we need a different fix.

How about we introduce (and immediately deprecate) the `with(...)` method. 
We should describe there that this is a temporary workaround. The `with(...) 
method only differs in the return type and calls `apply(...)`, casting the 
returned value.

Would everyone be okay with that?


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413665#comment-15413665
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2305
  
We could "deprecate" the `apply()` method and redirect to `with(...)`. That 
would be easier to find than a comment about casting.


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413649#comment-15413649
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2305
  
Wouldn't this suffer from the same problem as the "casting solution"? 
People would use `apply` and then wonder why there is no `setParallelism`, not 
bothering to read the Javadoc to find out that there is also `with`.


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413641#comment-15413641
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2305
  
I just thought of a different trick: We could add a second variant of the 
`apply(..)` function (for example called `with(...)` as in the DataSet API) and 
have the proper return type there (calling apply() and cast).

We can then immediately deprecate the `with()` function to indicate that it 
is a temporary workaround and is to be replaced by `apply(...)` in Flink 2.0.


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411852#comment-15411852
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2305
  
I am at the point where I would suggest to actually break the API and apply 
a proper fix. Tough step, but the alternatives seem even worse.

I wrote a [discuss] mail to the dev list, let's see if someone objects, 
otherwise I would merge this in the next days (adding the japicmp exception 
rule).


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408106#comment-15408106
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2305
  
@zentol I think that works only in theory. In practice, probably no user is 
going to realize that this trick is possible (even if we put it into the 
JavaDocs). It also seems pretty mean thing to force a user to do.

I am quite torn at the moment whether this is an incident to break the API 
- I am leaning towards yes.



> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408008#comment-15408008
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2305
  
I'm not so sure about breaking compatibility. Let's keep in mind that users 
can work around this by casting the returned `DataStream` to a 
`SingleOutputStreamOperator`.


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407910#comment-15407910
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2305
  
As I said, I think it would be very good to change the return type to 
`SingleOutputStreamOperator`. However, breaking binary compatibility should 
also not be taken lightly.


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407086#comment-15407086
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/2305
  
Hi @aljoscha @tillrohrmann , what do you think about this?



> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405827#comment-15405827
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2305
  
True, this may warrant a case of breaking APIs.


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405657#comment-15405657
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/2305
  
I think, specify parallelism between `window(...)` and `apply(...)` is not 
nice. We not just need `setParallelism` function but also `name`, `uid`, 
`slotSharingGroup` and many other functions in `SingleOutputStreamOperator`. If 
we copy these to `WithWindow`, it will be very duplicate.


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405623#comment-15405623
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2305
  
We should not change the checker sensitivity. It is very important that it 
checks for binary compatibility - that was the promise of Flink 1.0.

If we want to break compatibility, we can define an exception rule to the 
checker.

I think we should explore other options before. We can specify the 
parallelism between `window(...)` and `apply(...)` without breaking the 
compatibility. It would be a bit less nice, because it is different from how 
the other functions do it, but it allows us to maintain the promise of 
compatibility.

What do you think about that?


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405156#comment-15405156
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/2305
  
I agree with @aljoscha . Can we reduce the checker's sensitivity to pass 
this change ? How do we do when we need to break compatibility?


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404228#comment-15404228
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2305
  
I think it breaks binary compatibility, not source compatibility.
The specific return type is part of the method signature used by the JVM 
for linking.


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404193#comment-15404193
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2305
  
@StephanEwen I think technically it doesn't break the API because 
`SingleOutputStreamOperator` is a subclass of `DataStream`, right? (It might 
break binary compatibility though, because the signature of the method changes) 
It might only be the checker that is to sensitive.


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404149#comment-15404149
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2305
  
@wuchong The CI fails because this change breaks backwards compatibility. I 
understand though that this is a pretty critical change and may actually 
warrant to break the backwards compatibility.

Let me think what we can do there...


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397346#comment-15397346
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/2305
  
The CI failed because of japicmp, as we changed the public API. 

Have no idea how to fix this... 


> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397229#comment-15397229
 ] 

ASF GitHub Bot commented on FLINK-4271:
---

GitHub user wuchong opened a pull request:

https://github.com/apache/flink/pull/2305

[FLINK-4271] [DataStreamAPI] Enable CoGroupedStreams and JoinedStreams to 
set parallellism

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [ ] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed

The CoGroupStream will construct the following graph. 

```
source -> MAP ---
|->  WindowOp -> Sink
source -> MAP ---
```

By now , the MAP and WindowOp can not set parallelism.  We can keep the MAP 
has same parallelism as previous operator (chaining). And we can change 
{{CoGroupedStreams.apply}} to return a  {{SingleOutputStreamOperator}} instead 
of {{DataStream}}, so that we can set WindowOp's parallelism.  The same thing 
has be done to {{JoinedStream}}.

So that we can do the following things:

```
DataStream result = one.coGroup(two)
 .where(new MyFirstKeySelector())
 .equalTo(new MyFirstKeySelector())
 .window(TumblingEventTimeWindows.of(Time.of(5, TimeUnit.SECONDS)))
 .apply(new MyCoGroupFunction());
 .setParallelism(10)
 .name("MyCoGroupWindow")
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wuchong/flink CoGroupStreams

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2305.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2305


commit 7b9594a175f33e62826a0cb51380f33dec5857b6
Author: Jark Wu 
Date:   2016-07-28T06:32:13Z

[FLINK-4271] [DataStreamAPI] Enable CoGroupedStreams and JoinedStreams to 
set parallelism.




> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[Jark Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397187#comment-15397187
 ] 

Jark Wu commented on FLINK-4271:


Yea, SingleOutputStreamOperator can set parallelism, uid, name and so on.  

But Where and EqualsTo will not reflect to physical operators, it is just used 
to get the keys to co-group. So it has no attributes to set.

> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[Wenlong Lyu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397140#comment-15397140
 ] 

Wenlong Lyu commented on FLINK-4271:


I think the solution can be better. Not only the parallelism but also other 
attributes of the operator such as uid, name, etc. should be configurable.
Maybe the ```Where``` and ```EqualsTo``` in the ```CoGroupedStreams``` can 
provide some interface wrappering the same functions provided in the 
```SingleOutputStreamOperator```, I am not sure yet~

> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4271) There is no way to set parallelism of operators produced by CoGroupedStreams

[Jark Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397004#comment-15397004
 ] 

Jark Wu commented on FLINK-4271:


The CoGroupStream will construct the following graph. 

{code}
source -> MAP ---
|->  WindowOp -> Sink
source -> MAP ---
{code}

By now , the MAP and WindowOp can not set parallelism.  We can keep the MAP has 
same parallelism as previous operator. And we can change 
{{WindowedStream.apply}} to return a  {{SingleOutputStreamOperator}} instead of 
{{DataStream}}, so that we can change WindowOp's parallelism.

> There is no way to set parallelism of operators produced by CoGroupedStreams
> 
>
> Key: FLINK-4271
> URL: https://issues.apache.org/jira/browse/FLINK-4271
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: Wenlong Lyu
>Assignee: Jark Wu
>
> Currently, CoGroupStreams package the map/keyBy/window operators with a human 
> friendly interface, like: 
> dataStreamA.cogroup(streamB).where(...).equalsTo().window().apply(), both the 
> intermediate operators and final window operators can not be accessed by 
> users, and we cannot set attributes of the operators, which make co-group 
> hard to use in production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)