Github user sachingoel0101 commented on the pull request:
https://github.com/apache/flink/pull/1338#issuecomment-155064766
This can be fixed, but it requires breaking the existing Netty pipeline.
The `HttpObjectAggregator` requires a max content length, which I think cannot
be set to too high; 10 MB seemed reasonable. [I tried setting it to 100 MB and
the JVM ran out of heap space surprisingly. Does the web monitor also use heap
space out of Job Manager? ]
I am not at all familiar with Netty however, and all I know is what I
learnt while working on this. If anyone has an idea how to handle large
uploads, in the existing pipeline, please let me know. :)
Otherwise I think replacing the `HttpObjectAggregator` with something that
handles large payloads on disk might be a good idea. An example is here:
https://github.com/netty/netty/blob/master/example/src/main/java/io/netty/example/http/upload/HttpUploadServerHandler.java
It should be able to handle everything cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
