[GitHub] flink issue #5545: [FLINK-8718][DataStream] Set maxParallelism on non-parall...

[StefanRRichter]

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/5545
  
I agree with @aljoscha, there is a sanity check that `maxParallelism` of 
operators did not change when restoring.


---

[GitHub] flink issue #5545: [FLINK-8718][DataStream] Set maxParallelism on non-parall...

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5545
  
Yes, no source can use keyed state. 😄 But the `StateAssignmentOperation` 
will still fail a restore if the `maxParallelism` of an operator changes. Maybe 
@StefanRRichter can chime in on that but I think we need to at least touch 
those parts.


---

[GitHub] flink issue #5545: [FLINK-8718][DataStream] Set maxParallelism on non-parall...

[tillrohrmann]

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/5545
  
Shouldn't it always be possible to merge n key groups into a single key 
group? Maybe this could be the backwards compatibility path.

Moreover, which sources use keyed state? Don't almost all of the sources 
use operator state if they are stateful?


---

[GitHub] flink issue #5545: [FLINK-8718][DataStream] Set maxParallelism on non-parall...

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5545
  
I'm not sure we can do this change since it breaks compatibility for 
savepoints because previous versions of Flink hat some `maxParallelism` setting 
for sources which would be incompatible with the new `maxParallelism` of 1. 
Thoughts?


---

[GitHub] flink issue #5545: [FLINK-8718][DataStream] Set maxParallelism on non-parall...

[GJL]

Github user GJL commented on the issue:

https://github.com/apache/flink/pull/5545
  
The test relies on being able to set setParallelism on a non-parallel 
source:
`DataStream input1 = env.fromElements(1, 2, 3, 
4).setMaxParallelism(128);`


---

[GitHub] flink issue #5545: [FLINK-8718][DataStream] Set maxParallelism on non-parall...

[tillrohrmann]

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/5545
  
Could that be a faulty test?


---

[GitHub] flink issue #5545: [FLINK-8718][DataStream] Set maxParallelism on non-parall...

[GJL]

Github user GJL commented on the issue:

https://github.com/apache/flink/pull/5545
  
I have test failures, e.g., ` 
StreamGraphGeneratorTest.testMaxParallelismWithConnectedKeyedStream` 😢 


---