[
https://issues.apache.org/jira/browse/BEAM-14014?focusedWorklogId=753974&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-753974
]
ASF GitHub Bot logged work on BEAM-14014:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 07/Apr/22 10:58
Start Date: 07/Apr/22 10:58
Worklog Time Spent: 10m
Work Description: lwyszomi commented on PR #17244:
URL: https://github.com/apache/beam/pull/17244#issuecomment-1091590168
@ryanthompson591 I tried to test this solution and I have some problems. I
build the SDK locally using BEAM wiki
https://cwiki.apache.org/confluence/display/BEAM/Python+Tips. When I run command
```bash
python -m apache_beam.examples.wordcount --runner DataflowRunner
--num_workers 1 --project airflow-operator-tests-1 --output
gs://airflow-system-tests-data-bucket/output --temp_location
gs://airflow-system-tests-data-bucket/temp/ --worker_harness_container_image
gcr.io/airflow-operator-tests-1/wyszomirski/beam/beam_python3.6_sdk:2.39.0.dev
--experiment beam_fn_api --sdk_location build/apache-beam-2.39.0.dev0.tar.gz
--debug --target_principal <SA> --delegate_accounts <SA2>
```
I got an error:
```tryceback
Traceback (most recent call last):
File
"/usr/local/google/home/wyszomirski/.pyenv/versions/3.8.9/lib/python3.8/runpy.py",
line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File
"/usr/local/google/home/wyszomirski/.pyenv/versions/3.8.9/lib/python3.8/runpy.py",
line 87, in _run_code
exec(code, run_globals)
File
"/usr/local/google/home/wyszomirski/beam/sdks/python/apache_beam/examples/wordcount.py",
line 105, in <module>
run()
File
"/usr/local/google/home/wyszomirski/beam/sdks/python/apache_beam/examples/wordcount.py",
line 84, in run
lines = p | 'Read' >> ReadFromText(known_args.input)
File
"/usr/local/google/home/wyszomirski/beam/sdks/python/apache_beam/io/textio.py",
line 666, in __init__
self._source = self._source_class(
File
"/usr/local/google/home/wyszomirski/beam/sdks/python/apache_beam/io/textio.py",
line 128, in __init__
super().__init__(
File
"/usr/local/google/home/wyszomirski/beam/sdks/python/apache_beam/io/filebasedsource.py",
line 124, in __init__
self._validate()
File
"/usr/local/google/home/wyszomirski/beam/sdks/python/apache_beam/options/value_provider.py",
line 193, in _f
return fnc(self, *args, **kwargs)
File
"/usr/local/google/home/wyszomirski/beam/sdks/python/apache_beam/io/filebasedsource.py",
line 185, in _validate
match_result = FileSystems.match([pattern], limits=[1])[0]
File
"/usr/local/google/home/wyszomirski/beam/sdks/python/apache_beam/io/filesystems.py",
line 204, in match
return filesystem.match(patterns, limits)
File
"/usr/local/google/home/wyszomirski/beam/sdks/python/apache_beam/io/filesystem.py",
line 749, in match
raise BeamIOError("Match operation failed", exceptions)
apache_beam.io.filesystem.BeamIOError: Match operation failed with
exceptions {'gs://dataflow-samples/shakespeare/kinglear.txt':
AttributeError("module 'google.auth' has no attribute
'impersonated_credentials'")}
```
Also I noticed when I'm trying to execute the same command without new
parameters `target_principal` and `delegate_accounts` I also have error:
```tryceback
Traceback (most recent call last):
File
"/usr/local/google/home/wyszomirski/.pyenv/versions/3.8.9/lib/python3.8/runpy.py",
line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File
"/usr/local/google/home/wyszomirski/.pyenv/versions/3.8.9/lib/python3.8/runpy.py",
line 87, in _run_code
exec(code, run_globals)
File
"/usr/local/google/home/wyszomirski/beam/sdks/python/apache_beam/examples/wordcount.py",
line 105, in <module>
run()
File
"/usr/local/google/home/wyszomirski/beam/sdks/python/apache_beam/examples/wordcount.py",
line 100, in run
output | 'Write' >> WriteToText(known_args.output)
File
"/usr/local/google/home/wyszomirski/beam/sdks/python/apache_beam/pipeline.py",
line 596, in __exit__
self.result = self.run()
File
"/usr/local/google/home/wyszomirski/beam/sdks/python/apache_beam/pipeline.py",
line 546, in run
return Pipeline.from_runner_api(
File
"/usr/local/google/home/wyszomirski/beam/sdks/python/apache_beam/pipeline.py",
line 573, in run
return self.runner.run_pipeline(self, self._options)
File
"/usr/local/google/home/wyszomirski/beam/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
line 587, in run_pipeline
self.dataflow_client = apiclient.DataflowApplicationClient(
File
"/usr/local/google/home/wyszomirski/beam/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",
line 558, in __init__
self._google_cloud_options.target_principal,
AttributeError: 'DataflowApplicationClient' object has no attribute
'_google_cloud_options'
```
I think the second case should be executed without any error.
Issue Time Tracking
-------------------
Worklog Id: (was: 753974)
Time Spent: 1h (was: 50m)
> Support impersonation credentials in Dataflow runner.
> -----------------------------------------------------
>
> Key: BEAM-14014
> URL: https://issues.apache.org/jira/browse/BEAM-14014
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Reporter: Valentyn Tymofieiev
> Assignee: Ryan Thompson
> Priority: P2
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
