[
https://issues.apache.org/jira/browse/BEAM-3455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16332985#comment-16332985
]
Chamikara Jayalath commented on BEAM-3455:
------------------------------------------
This issue was already discussed in the mailing list.
[https://lists.apache.org/thread.html/a4346e5bbef22fc311e76ccd13568bd9a4e0b4e83a137fc1d63ed3b8@%3Cuser.beam.apache.org%3E]
Unais, do you need additional info regarding this ?
> Request payload size exceeds the limit: 10485760 bytes
> ------------------------------------------------------
>
> Key: BEAM-3455
> URL: https://issues.apache.org/jira/browse/BEAM-3455
> Project: Beam
> Issue Type: Bug
> Components: beam-model, build-system, runner-dataflow
> Reporter: Unais
> Assignee: Chamikara Jayalath
> Priority: Major
>
> I wrote a python dataflow job to read data from Bigquery and do some
> transform and save the result as bq table..
> I tested with 8 days data it works fine - when I scaled to 180 days I’m
> getting the below error
> ```"message": "Request payload size exceeds the limit: 10485760 bytes.",```
> ```pitools.base.py.exceptions.HttpError: HttpError accessing
> <https://dataflow.googleapis.com/v1b3/projects/careem-mktg-dwh/locations/us-central1/jobs?alt=json>:
> response: <{'status': '400', 'content-length': '145', 'x-xss-protection':
> '1; mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding':
> 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF',
> '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Wed, 10 Jan
> 2018 22:49:32 GMT', 'x-frame-options': 'SAMEORIGIN', 'alt-svc': 'hq=":443";
> ma=2592000; quic=51303431; quic=51303339; quic=51303338; quic=51303337;
> quic=51303335,quic=":443"; ma=2592000; v="41,39,38,37,35"', 'content-type':
> 'application/json; charset=UTF-8'}>, content <{
> "error": {
> "code": 400,
> "message": "Request payload size exceeds the limit: 10485760 bytes.",
> "status": "INVALID_ARGUMENT"
> }
> ```
> In short, this is what I’m doing
> 1 - Reading data from bigquery table using
> ```beam.io.BigQuerySource ```
> 2 - Partitioning each days using
> ``` beam.Partition ```
> 3- Applying transforms each partition and combining some output P-Collections.
> 4- After the transforms, the results are saved to a biqquery date partitioned
> table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
