+user <[email protected]> to see if anyone else can provide guidance.
On Fri, Oct 11, 2019 at 5:31 PM Jitendra kumavat <[email protected]> wrote: > I have added the plugin and checked my, jar it contains the my service > registered. Still unable to run my JvmInitializer class. > > <plugin> > <groupId>org.apache.maven.plugins</groupId> > <artifactId>maven-shade-plugin</artifactId> > <version>3.2.1</version> > <executions> > <execution> > <goals> > <goal>shade</goal> > </goals> > <configuration> > <transformers> > <transformer > implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/> > </transformers> > </configuration> > </execution> > </executions> > </plugin> > > > On Fri, Oct 11, 2019 at 2:56 PM Luke Cwik <[email protected]> wrote: > >> It is a common issue for users build processes to use the Maven shade >> plugin or jarjar and to lose the META-INF/services files. >> Can you verify that the jar's you submitted to Dataflow contain the >> META-INF/services files? >> >> See this stackoverflow question >> https://stackoverflow.com/questions/44365545/apache-beam-unable-to-find-registrar-for-gs >> for >> a similar problem for more details. >> >> >> On Fri, Oct 11, 2019 at 2:38 PM Jitendra kumavat <[email protected]> >> wrote: >> >>> Hi Luke, >>> >>> I added @Autoservice for my custom intializer/processor and i can see >>> the entries in META-INF/Services folder in my build class. Please see the >>> attached screenshot. Still while running the dataflow job on google cloud i >>> can not see the my System.out statements in logs. It is not picking up >>> the GcmJvmInitializer, i believe so. >>> >>> >>> Steps which i follow to run the job: >>> >>> 1. mvn clean install >>> 2. create dataflow template in gcs bucket using mvn compile command. >>> 3. run the job using gcloud dataflow jobs <jobname> >>> >>> Can you please tell me what's wrong i am doing here. Thanks. >>> >>> Below is my classes structure: >>> >>> @AutoService(Processor.class) >>> @SupportedAnnotationTypes( >>> "org.springframework.context.annotation.*") >>> @SupportedSourceVersion(SourceVersion.RELEASE_8) >>> public class MyProcessor extends AbstractProcessor { >>> @Override >>> public boolean process(Set<? extends TypeElement> annotations, >>> RoundEnvironment roundEnv) { >>> return false; >>> } >>> } >>> >>> >>> >>> >>> @AutoService(GcmJvmInitializer.class) >>> public class GcmJvmInitializer implements JvmInitializer { >>> >>> @Override >>> public void onStartup() { >>> System.out.println("Starting Custom GcmJvmInitializer"); >>> ApplicationContext applicationContext = new >>> AnnotationConfigApplicationContext( >>> AppConfig.class); >>> for (String beanName : applicationContext.getBeanDefinitionNames()) { >>> System.out.println(beanName); >>> } >>> System.out.println("Stopping Custom GcmJvmInitializer"); >>> } >>> >>> @Override >>> public void beforeProcessing(PipelineOptions options) { >>> System.out.println("Starting Custom GcmJvmInitializer"); >>> ApplicationContext applicationContext = new >>> AnnotationConfigApplicationContext( >>> AppConfig.class); >>> for (String beanName : applicationContext.getBeanDefinitionNames()) { >>> System.out.println(beanName); >>> } >>> System.out.println("Stopping Custom GcmJvmInitializer"); >>> } >>> >>> } >>> >>> >>> Thanks, >>> >>> Jitendra >>> >>> >>> On Thu, Oct 10, 2019 at 6:35 PM Luke Cwik <[email protected]> wrote: >>> >>>> You shouldn't need to call it before running the pipeline as you are >>>> doing (you can if you want but its not necessary). >>>> >>>> Have you created a service META-INF entry for the JvmInitializer you >>>> have created or are using @AutoService? >>>> This is the relevant bit of the documentation[1]. Here is some good >>>> docs for how to use @AutoService[2]. >>>> >>>> 1: >>>> https://github.com/apache/beam/blob/f3ce8669b50837d48ab0d0ee9a1298ce3b5bc61c/sdks/java/core/src/main/java/org/apache/beam/sdk/harness/JvmInitializer.java#L30 >>>> 2: https://github.com/google/auto/tree/master/service >>>> >>>> >>>> On Thu, Oct 10, 2019 at 5:29 PM Jitendra kumavat < >>>> [email protected]> wrote: >>>> >>>>> Hi Luke, >>>>> >>>>> I tried the JvmIntializer.beforeProccssing method to initialize the >>>>> spring application context. But it seems not working. >>>>> >>>>> >>>>> * Below is my class definitions. * >>>>> >>>>> JvmInitializer class with context initialization. >>>>> >>>>> public class GcmJvmInitializer implements JvmInitializer { >>>>> @Override >>>>> public void beforeProcessing(PipelineOptions options){ >>>>> System.out.println("Starting Custom GcmJvmInitializer"); >>>>> ApplicationContext applicationContext = new >>>>> AnnotationConfigApplicationContext( >>>>> AppConfig.class); >>>>> for (String beanName : applicationContext.getBeanDefinitionNames()) { >>>>> System.out.println(beanName); >>>>> } >>>>> System.out.println("Stopping Custom GcmJvmInitializer"); >>>>> } >>>>> } >>>>> >>>>> >>>>> *Appconfig* >>>>> >>>>> @Configuration >>>>> @PropertySource("classpath:gcm.properties") >>>>> @ComponentScan(basePackages = {"com.liveramp.intl.gcm"}) >>>>> public class AppConfig { >>>>> >>>>> // Bean definitions with @Bean annotation. >>>>> >>>>> } >>>>> >>>>> >>>>> And i am using JvmInitializers to inject the same. Below is the my main >>>>> method for the same. >>>>> >>>>> >>>>> >>>>> public static void main(String[] args) { >>>>> >>>>> PCollection<String> lines = pipeline.apply("Read Lines", >>>>> TextIO.read().from(inputPathProvider)); >>>>> >>>>> //.. follows other pipeline transformations. >>>>> >>>>> //run pipeline >>>>> JvmInitializers.runBeforeProcessing(options); >>>>> pipeline.run().waitUntilFinish(); >>>>> >>>>> } >>>>> >>>>> >>>>> Is this the right way to use it? Or is there other way round? Please let >>>>> me know. >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Jitendra >>>>> >>>>> >>>>> On Wed, Oct 9, 2019 at 1:32 PM Luke Cwik <[email protected]> wrote: >>>>> >>>>>> 1. won't work since it is happening at pipeline construction time and >>>>>> not pipeline execution time. >>>>>> 2. only works if your application context is scoped to the DoFn >>>>>> instance and doesn't have things you want to possibly share across DoFn >>>>>> instances. >>>>>> >>>>>> You could also try and make it a PipelineOption that is tagged >>>>>> with @JsonIgnore and also has a @Default.InstanceFactory like this[1]. >>>>>> This >>>>>> way when it is accessed by your DoFn it will be initialized for the first >>>>>> time and shared within your process. Making it a PipelineOption would >>>>>> also >>>>>> allow you to pass in preinitialized versions for testing. >>>>>> >>>>>> 1: >>>>>> https://github.com/apache/beam/blob/8267c223425bc201be700babbe596d133b79686e/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java#L127 >>>>>> >>>>>> On Wed, Oct 9, 2019 at 1:10 PM Jitendra kumavat < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Luke, >>>>>>> >>>>>>> Thanks a lot for your reply. >>>>>>> I tried couple of options which is as follows. >>>>>>> >>>>>>> 1. Initialise the context in main method only. and use it. Creating >>>>>>> the context: >>>>>>> new AnnotationConfigApplicationContext(AppConfig.class); >>>>>>> 2. Creating the context on DoFn.Startup method. >>>>>>> >>>>>>> Unfortunately none of the worked perfectly, later works but it has >>>>>>> issue with @ComponentScan. >>>>>>> Please let me know your comments for the same. >>>>>>> >>>>>>> I will also try this JvmInitializer for context initialisation. >>>>>>> >>>>>>> Thanks, >>>>>>> Jitendra >>>>>>> >>>>>>> On Wed, Oct 9, 2019 at 12:48 PM Luke Cwik <[email protected]> wrote: >>>>>>> >>>>>>>> [email protected], [email protected] >>>>>>>> >>>>>>>> How are you trying to inject your application context? >>>>>>>> Have you looked at the JvmInitializer.beforeProcessing[1] to create >>>>>>>> your application context? >>>>>>>> >>>>>>>> 1: >>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/harness/JvmInitializer.java >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Oct 4, 2019 at 12:32 PM Jitendra kumavat < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I want to add Spring framework in my apache beam project. Somehow >>>>>>>>> i am unable to inject the Spring Application context to executing >>>>>>>>> ParDo >>>>>>>>> functions. I couldn't find the way to do so? Can you please let me >>>>>>>>> know how >>>>>>>>> to integrate Spring runtime application context with Apache Beam >>>>>>>>> pipeline. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Jitendra >>>>>>>>> >>>>>>>>
