unsubscribe On Fri, Aug 6, 2021 at 10:36 AM Jeremy Lewi <jeremy.l...@primer.ai> wrote:
> Hi Folks, > > I'm running Beam Python (2.31.0) on Flink on Kubernetes. I'm using the > PortableRunner and a beam job server. > I'm using S3 for the artifacts dir. The job server is throwing exceptions > like the one below complaining that filesystem scheme S3 isn't registered. > > I'm using the stock job server > container apache/beam_flink1.13_job_server:2.31.0. It doesn't look like the > jar that ships in that server includes the AWS Filesystem classes. > > So I tried adding the > jar beam-sdks-java-io-amazon-web-services-2.31.0.jar. Now when the job > server loads I get a ClassNotFound exception for AWSCredentialsProvider > because > I'm missing some of the dependencies for that. > > Does anyone have recommendations on the easiest path to obtaining the > jar(s) needed to support S3 for the artifact service? Is there an uber jar > published somewhere or will I have to build it myself? > > Thanks. > J > > > > Aug 06, 2021 1:52:51 AM > org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2 > finishStaging > > SEVERE: Error staging artifacts > > java.util.concurrent.ExecutionException: > java.lang.IllegalArgumentException: No filesystem found for scheme s3 > > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > > at > org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.finishStaging(ArtifactStagingService.java:461) > > at > org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.resolveNextEnvironment(ArtifactStagingService.java:439) > > at > org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.onNext(ArtifactStagingService.java:417) > > at > org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.onNext(ArtifactStagingService.java:303) > > at > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:255) > > at > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33) > > at > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76) > > at > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:309) > > at > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:292) > > at > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:765) > > at > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) > > at > org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > Caused by: java.lang.IllegalArgumentException: No filesystem found for > scheme s3 > > at > org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:497) > > at > org.apache.beam.sdk.io.FileSystems.matchNewResource(FileSystems.java:567) > > at > org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$1.stagingDir(ArtifactStagingService.java:193) > > at > org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$1.getDestination(ArtifactStagingService.java:169) > > at > org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:271) > > at > org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:247) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > ... 3 more > > > > > --- > > > Here's the exception after adding the amazon web services Jar > > > Module could not be instantiated > > at java.util.ServiceLoader.fail(ServiceLoader.java:232) > > at java.util.ServiceLoader.access$100(ServiceLoader.java:185) > > at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) > > at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) > > at java.util.ServiceLoader$1.next(ServiceLoader.java:480) > > at > com.fasterxml.jackson.databind.ObjectMapper.findModules(ObjectMapper.java:1081) > > at > org.apache.beam.sdk.options.PipelineOptionsFactory.<clinit>(PipelineOptionsFactory.java:478) > > at > org.apache.beam.runners.flink.FlinkJobServerDriver.main(FlinkJobServerDriver.java:72) > > Caused by: java.lang.NoClassDefFoundError: > com/amazonaws/auth/AWSCredentialsProvider > > at org.apache.beam.sdk.io.aws.options.AwsModule.<init>(AwsModule.java:86) > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > > at java.lang.Class.newInstance(Class.java:442) > > at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) > > ... 5 more > > Caused by: java.lang.ClassNotFoundException: > com.amazonaws.auth.AWSCredentialsProvider > > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > > ... 12 more > > >
