date:20191205

[jira] [Work logged] (BEAM-8895) BigQueryIO streaming test on Java is flaky

2019-12-05 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/BEAM-8895?focusedWorklogId=354985=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354985
 ]

ASF GitHub Bot logged work on BEAM-8895:


Author: ASF GitHub Bot
Created on: 06/Dec/19 07:22
Start Date: 06/Dec/19 07:22
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #10294: [BEAM-8895] Add 
BigQuery table name sanitization to BigQueryIOIT
URL: https://github.com/apache/beam/pull/10294#issuecomment-562463303
 
 
   Run BigQueryIO Streaming Performance Test Java
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 354985)
Time Spent: 1h 40m  (was: 1.5h)

> BigQueryIO streaming test on Java is flaky
> --
>
> Key: BEAM-8895
> URL: https://issues.apache.org/jira/browse/BEAM-8895
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Kamil Wasilewski
>Assignee: Michal Walenia
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> {code:java}
> SEVERE: 2019-12-05T06:57:31.089Z: java.lang.RuntimeException: 
> com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad 
> Request
> 07:57:32 {
> 07:57:32   "code" : 400,
> 07:57:32   "errors" : [ {
> 07:57:32 "domain" : "global",
> 07:57:32 "message" : "Invalid table ID 
> \"bqio_write_10GB_java_e27dc010-6896-41ac-90f3-25b5adc58617\". Table IDs must 
> be alphanumeric (plus underscores) and must be at most 1024 characters long. 
> Also, Table decorators cannot be used.",
> 07:57:32 "reason" : "invalid"
> 07:57:32   } ],
> 07:57:32   "message" : "Invalid table ID 
> \"bqio_write_10GB_java_e27dc010-6896-41ac-90f3-25b5adc58617\". Table IDs must 
> be alphanumeric (plus underscores) and must be at most 1024 characters long. 
> Also, Table decorators cannot be used.",
> 07:57:32   "status" : "INVALID_ARGUMENT"
> 07:57:32 }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8882) Allow Dataflow to automatically choose portability or not.

2019-12-05 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/BEAM-8882?focusedWorklogId=354958=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354958
 ]

ASF GitHub Bot logged work on BEAM-8882:


Author: ASF GitHub Bot
Created on: 06/Dec/19 06:31
Start Date: 06/Dec/19 06:31
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #10292: [BEAM-8882] Fully 
populate log messages.
URL: https://github.com/apache/beam/pull/10292#issuecomment-562451084
 
 
   Run Python PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 354958)
Time Spent: 4h 20m  (was: 4h 10m)

> Allow Dataflow to automatically choose portability or not.
> --
>
> Key: BEAM-8882
> URL: https://issues.apache.org/jira/browse/BEAM-8882
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Critical
> Fix For: 2.18.0
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> We would like the Dataflow service to be able to automatically choose whether 
> to run pipelines in a portable way. In order to do this, we need to provide 
> more information even if portability is not explicitly requested. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-8906) Long BigQuery dry runs cause avalanche delay

2019-12-05 Thread June Oh (Jira)

June Oh created BEAM-8906:
-

Summary: Long BigQuery dry runs cause avalanche delay
Key: BEAM-8906
URL: https://issues.apache.org/jira/browse/BEAM-8906
Project: Beam
Issue Type: Bug
Components: io-java-gcp
Affects Versions: 2.16.0
Environment: Google Cloud Platform
Reporter: June Oh

Reproduction Steps:

1. Compose a BigQuery SELECT query that will take over 80 seconds for a dry run.
2. Run the query with Beam SDK's BigQueryIO.
3. Observe the 10+ minute delay before the actual query job is created.

When running readTableRows(), BigQueryIO attempts to estimate the query size by
performing a dry run, even if withoutValidation() is set. If the request takes
over 80 seconds (RetryHttpRequestInitializer.HANGING_GET_TIMEOUT_SEC),
RetryHttpRequestInitializer will time out and retry, up to 9 times
(BigQueryServicesImpl.MAX_RPC_RETRIES). Hence, once a dry run duration crosses
the 80 second tipping point, it causes an inevitable avalanche of a 720-second
delay. Considering the fact that size estimation is not a requirement in
running the query [1], BigQueryIO should provide a way to circumvent the
redundant delay, especially in consideration of time-critical enterprise
workloads.

There can be several ways to address this:
- increasing the timeout threshold (which will still create a tipping point);
- preventing the dry run requests from retrying; or
- adding an option to skip the size estimation within serializeToCloudSource().

[1]
https://github.com/apache/beam/blob/2ec3b0495c191597c9a88830d25a2c360b3277e0/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/CustomSources.java#L75

1 2 3 4 >

1 - 100 of 332 matches

Mail list logo