dpcollins-google commented on pull request #14122: URL: https://github.com/apache/beam/pull/14122#issuecomment-788329463
R: @pabloem I don't know if this would affect anything in the dataflow backend? But AFAICT this would result in more compressed JSON and fewer user jobs over the limit. @Fokko, I don't know that this solves your issue. Even after this change, your job graph size is still over the limit (20 MiB IIUC?) https://cloud.google.com/dataflow/docs/guides/common-errors#job-graph-too-large It might be good to look at what is in the serialized JSON payload for your job, see if there's any multi-MiB blobs of data that you shouldn't be serializing? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
