max-lepikhin opened a new issue, #24537: URL: https://github.com/apache/beam/issues/24537
### What happened? 1. Python [flink_runner starts](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/flink_runner.py#L109) beam flink job-server with these parameters: ```python def java_arguments( self, job_port, artifact_port, expansion_port, artifacts_dir): return [ '--flink-master', self._master_url, '--artifacts-dir', artifacts_dir, '--job-port', job_port, '--artifact-port', artifact_port, '--expansion-port', expansion_port ] ``` 2. When **artifacts_dir** above is path to a container in Azure blob storage (e.g. "azfs://storage-account/container), the [AzureBlobStoreFileSystem](https://github.com/apache/beam/blob/4ffeae4d2b800f2df36d2ea2eab549f2204d5691/sdks/java/io/azure/src/main/java/org/apache/beam/sdk/io/azure/blobstore/AzureBlobStoreFileSystem.java#L271) attempts to create BlobServiceClient and fails with: ``` I1206 02:34:29.021346 139656344459008 subprocess_server.py:126] Caused by: java.lang.IllegalArgumentException: Invalid URL format. URL: null I1206 02:34:29.021379 139656344459008 subprocess_server.py:126] at com.azure.storage.blob.BlobUrlParts.parse(BlobUrlParts.java:349) I1206 02:34:29.021412 139656344459008 subprocess_server.py:126] at com.azure.storage.blob.implementation.util.BuilderHelper.httpsValidation(BuilderHelper.java:170) I1206 02:34:29.021589 139656344459008 subprocess_server.py:126] at com.azure.storage.blob.implementation.util.BuilderHelper.buildPipeline(BuilderHelper.java:106) I1206 02:34:29.024176 139656344459008 subprocess_server.py:126] at com.azure.storage.blob.BlobServiceClientBuilder.buildAsyncClient(BlobServiceClientBuilder.java:113) I1206 02:34:29.024224 139656344459008 subprocess_server.py:126] at com.azure.storage.blob.BlobServiceClientBuilder.buildClient(BlobServiceClientBuilder.java:89) I1206 02:34:29.024258 139656344459008 subprocess_server.py:126] at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:167) I1206 02:34:29.024288 139656344459008 subprocess_server.py:126] at org.apache.beam.sdk.io.azure.blobstore.AzureBlobStoreFileSystem.create(AzureBlobStoreFileSystem.java:271) I1206 02:34:29.024316 139656344459008 subprocess_server.py:126] at org.apache.beam.sdk.io.azure.blobstore.AzureBlobStoreFileSystem.create(AzureBlobStoreFileSystem.java:68) I1206 02:34:29.024344 139656344459008 subprocess_server.py:126] at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:243) I1206 02:34:29.024372 139656344459008 subprocess_server.py:126] at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:230) I1206 02:34:29.024399 139656344459008 subprocess_server.py:126] at org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$ArtifactDestination.fromFile(ArtifactStagingService.java:140) ``` The null reference exception occurs because the endpoint to blob service is not configured. It is expected to be passed via "blob_service_endpoint" [option](https://github.com/apache/beam/blob/243128a8fc52798e1b58b0cf1a271d95ee7aa241/sdks/java/io/azure/src/main/java/org/apache/beam/sdk/io/azure/options/BlobstoreOptions.java#L85). Neither the option is passed from flink_runner.py nor it is inferred from the "azfs://storage-account/container-name" file path in AzureBlobStoreFileSystem.java. ### Issue Priority Priority: 1 ### Issue Component Component: beam-community -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
