rokadepiyush49-rgb commented on issue #18123:
URL: https://github.com/apache/beam/issues/18123#issuecomment-6565202847
I looked at where this check could go. `_stage_resources` in
`runners/dataflow/internal/apiclient.py` already validates that
`--staging_location` and `--temp_location` are set, and `GcsIO.get_bucket()`
returns the bucket object, so its lifecycle rules can be read without adding a
dependency.
A couple of details I'd want to get right:
- reading bucket metadata needs `storage.buckets.get`, which many
job-submitting service accounts don't have. A 403 (or any error) should count
as "unknown" and only log at debug level. It should never fail or slow down
submission;
- only rules with a Delete action and an `age` condition matter here. If one
is found, a single warning naming the bucket and the age seems right, and the
temp location probably deserves the same check.
Since this was filed a long time ago: is a warning still the behavior you'd
want, or would a docs note be preferred? If a warning is fine, I'd like to work
on it.
```mermaid
flowchart TD
subgraph P1["1. Before coding"]
A1["Re-run grep for utcfromtimestamp and utcnow"] --> A2["Post the
comment proposing the datetime slice"]
A2 --> A3{"Maintainer OK with a scoped PR?"}
A3 -- yes --> A4["Comment .take-issue, branch from master"]
end
subgraph P2["2. Implement, one call site at a time"]
R["Rule: use datetime.timezone.utc,<br/>not datetime.UTC, because
Python 3.10 is supported"]
B1["typehints/schemas.py:<br/>combine datetime.date(1970, 1, 1) with
the time value"] --> B2["io/gcp/bigquery_tools.py:<br/>fromtimestamp(value,
tz=timezone.utc), same strftime format"]
B2 -->
B3["interactive/display/pcoll_visualization.py:<br/>fromtimestamp(x,
tz=timezone.utc), then astimezone"]
B3 --> B4["examples/complete/game/hourly_team_score.py:<br/>keep a
naive epoch, datetime(1970, 1, 1)"]
B4 --> B5["dataframe/io_test.py and datastore v1new
types_test.py:<br/>update and re-check their assertions"]
end
subgraph P3["3. Test"]
C1["Add a test pinning the BigQuery TIMESTAMP string<br/>for one
known timestamp"] --> C2["pytest each touched module with<br/>-W
error::DeprecationWarning"]
C2 --> C3["Format with yapf 0.43.0, lint the changed files"]
end
subgraph P4["4. Pull request"]
D1["Title: Replace deprecated datetime.utcfromtimestamp"] -->
D2["Body: Part of 18303,<br/>so the broad issue stays open"]
D2 --> D3["Generated-by trailer, answer review"]
end
A4 --> B1
R -.-> B2
B5 --> C1
C3 --> D1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
