rahuliyer95 commented on pull request #17078: URL: https://github.com/apache/beam/pull/17078#issuecomment-1065458981
I ran the integration test suite by making some changes to the build files to allow them to run with our GCP setup. ``` ./gradlew -Pdocker-pull-licenses -PpythonVersion=3.7 -PdataflowProject=twttr-udp-e2e-jobs-prod :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava ``` The job ran with the following beam plugins, ``` --beam_plugins='apache_beam.io.filesystem.FileSystem, apache_beam.io.hadoopfilesystem.HadoopFileSystem, apache_beam.io.localfilesystem.LocalFileSystem, apache_beam.io.gcp.gcsfilesystem.GCSFileSystem, apache_beam.io.aws.s3filesystem.S3FileSystem, apache_beam.io.azure.blobstoragefilesystem.BlobStorageFileSystem, apache_beam.io.filesystem_test.TestingFileSystem, apache_beam.runners.portability.sdk_container_builder.SdkContainerImageBuilder, apache_beam.runners.portability.sdk_container_builder._SdkContainerImageLocalBuilder, apache_beam.runners.portability.sdk_container_builder._SdkContainerImageCloudBuilder, apache_beam.runners.interactive.display.pipeline_graph_renderer.PipelineGraphRenderer, apache_beam.runners.interactive.display.pipeline_graph_renderer.MuteRenderer, apache_beam.runners.interactive.display.pipeline_graph_renderer.TextRenderer, apache_beam.runners.interactive.display.pipeline_graph_renderer.PydotRenderer ``` and failed with the following error, ``` "2022/03/11 03:53:04 Traceback (most recent call last): " "2022/03/11 03:53:04 File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py", line 55, in _import_beam_plugins " "2022/03/11 03:53:04 importlib.import_module(plugin) " "2022/03/11 03:53:04 File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module " "2022/03/11 03:53:04 return _bootstrap._gcd_import(name[level:], package, level) " "2022/03/11 03:53:04 File "<frozen importlib._bootstrap>", line 1006, in _gcd_import " "2022/03/11 03:53:04 File "<frozen importlib._bootstrap>", line 983, in _find_and_load " "2022/03/11 03:53:04 File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked " "2022/03/11 03:53:04 File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed " "2022/03/11 03:53:04 File "<frozen importlib._bootstrap>", line 1006, in _gcd_import " "2022/03/11 03:53:04 File "<frozen importlib._bootstrap>", line 983, in _find_and_load " "2022/03/11 03:53:04 File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked " "2022/03/11 03:53:04 File "<frozen importlib._bootstrap>", line 677, in _load_unlocked " "2022/03/11 03:53:04 File "<frozen importlib._bootstrap_external>", line 728, in exec_module " "2022/03/11 03:53:04 File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed " "2022/03/11 03:53:04 File "/usr/local/lib/python3.7/site-packages/apache_beam/io/filesystem_test.py", line 34, in <module> " "2022/03/11 03:53:04 from parameterized import param " "2022/03/11 03:53:04 ModuleNotFoundError: No module named 'parameterized' " ``` Apache Beam SDK does not ship with test packages aka `parameterized` and hence the import fails. With the fix of ignoring the module if the import fails with a warning message seems to pass the integration tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
