shunping opened a new issue, #39235:
URL: https://github.com/apache/beam/issues/39235
### What happened?
To reproduce, start a fresh virtual environment and install
google-cloud-bigquery>=3.42.0 and apache_beam.
```
pip install 'apache_beam[gcp]==2.74'
python -m apache_beam.examples.wordcount --output a.txt
```
```
INFO:root:Missing pipeline option (runner). Executing pipeline using the
default runner: DirectRunner.
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File
"/Users/shunping/Projects/v2/venv-2.75rc1/lib/python3.12/site-packages/apache_beam/examples/wordcount.py",
line 116, in <module>
run()
File
"/Users/shunping/Projects/v2/venv-2.75rc1/lib/python3.12/site-packages/apache_beam/examples/wordcount.py",
line 90, in run
lines = pipeline | 'Read' >> ReadFromText(known_args.input)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/shunping/Projects/v2/venv-2.75rc1/lib/python3.12/site-packages/apache_beam/io/textio.py",
line 807, in __init__
self._source = self._source_class(
^^^^^^^^^^^^^^^^^^^
File
"/Users/shunping/Projects/v2/venv-2.75rc1/lib/python3.12/site-packages/apache_beam/io/textio.py",
line 143, in __init__
super().__init__(
File
"/Users/shunping/Projects/v2/venv-2.75rc1/lib/python3.12/site-packages/apache_beam/io/filebasedsource.py",
line 126, in __init__
self._validate()
File
"/Users/shunping/Projects/v2/venv-2.75rc1/lib/python3.12/site-packages/apache_beam/options/value_provider.py",
line 192, in _f
return fnc(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/shunping/Projects/v2/venv-2.75rc1/lib/python3.12/site-packages/apache_beam/io/filebasedsource.py",
line 189, in _validate
match_result = FileSystems.match([pattern], limits=[1])[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/shunping/Projects/v2/venv-2.75rc1/lib/python3.12/site-packages/apache_beam/io/filesystems.py",
line 240, in match
return filesystem.match(patterns, limits)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/shunping/Projects/v2/venv-2.75rc1/lib/python3.12/site-packages/apache_beam/io/filesystem.py",
line 777, in match
raise BeamIOError("Match operation failed", exceptions)
apache_beam.io.filesystem.BeamIOError: Match operation failed with
exceptions {'gs://dataflow-samples/shakespeare/kinglear.txt':
ValueError('Context has already been used to create a Connection, it cannot be
mutated again')}
```
This happens when environment variable
CLOUDSDK_CONTEXT_AWARE_USE_CLIENT_CERTIFICATE is set to `true`. If we set it to
`false` (e.g. `CLOUDSDK_CONTEXT_AWARE_USE_CLIENT_CERTIFICATE=false python -m
apache_beam.examples.wordcount --output e.txt`), the pipeline can run
successfully.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
