Hello Robert, I'm sorry for the late reply. Also, thank you for your reply.
I forgot to send it to user as well. It should now be added to the thread. When I access the repositories / orgapachebeam-1254 that I told you, the 404 not found page is displayed. Does this mean that I had another solution by the time I answered? Thanks & Regards Abe 2022年3月3日(木) 3:55 Robert Burke <[email protected]>: > Turns out this is exactly possible, as the Jar download paths have a > common suffix > https://issues.apache.org/jira/browse/BEAM-14029 has been filed for the > convenience override. > It's not a 2.37.0 release blocker due to the newness of the auto download > and start functionality, it's not a regression, but the convenience > override should be handled for 2.38.0 > Rather than another flag that will clutter a user's help, we'll be having > a variable or a Set function on the appropriate package for users to call > (or populated electively with their own flag), and the > https://beam.apache.org/blog/validate-beam-release/ will be updated to > address this ability for Cross Language transforms. > > On Wed, Mar 2, 2022 at 9:59 AM Robert Burke <[email protected]> wrote: > >> In replying to an off thread question from Abe, I realized that my >> suggestion doesn't work since the SDK doesn't know how to source/grab the >> RC versions of the Cross Language Jars. They end up getting a 404 response >> as a result. >> >> > panic: tried cross-language for >> beam:transform:org.apache.beam:schemaio_jdbc_read:v1 against >> autojava::sdks:java:extensions:schemaio-expansion-service:runExpansionService >> and failed >> > expanding external transform >> > received non 200 response code, got 404 >> >> I'll see if we can't improve that kind of error message for future >> releases. >> >> What is the best way for a user to access RC jars? >> >> I guess one would need to download them from the staging repo for the >> RC: >> https://repository.apache.org/content/repositories/orgapachebeam-1254/ >> >> In this case, it's likely one of the jars in here: >> >> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/ >> And specifically >> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/beam-sdks-java-extensions-schemaio-expansion-service-2.37.0.jar >> >> >> One should be able to start up the service with the jar, then using the >> address and port, provide an `jdbcio.ExpansionServiceRead` option [1] to >> the call, so it queries the service you started up instead. >> >> [1] >> https://pkg.go.dev/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/xlang/jdbcio#ExpansionAddrRead >> >> I've asked the team to look into simplifying this. We should in principle >> should get it to work against a specific RC repo with a flag, rather than >> forcing the full manual startup. >> >> Cheers, >> Robert Burke >> Beam Go Busybody >> >> >> >> On Mon, Feb 28, 2022 at 11:09 AM Robert Burke <[email protected]> wrote: >> >>> Hi Abe! >>> >>> Thank you for your interest in the Go SDK! I have two responses for you. >>> >>> First, the current native databaseio and bigqueryio transform in the Go >>> SDK aren't production ready at this time. They support scaling, which may >>> lead to the limits that you're seeing. Outside of the native TextIO with >>> the ReadSDF* calls, those old transforms don't scale well. They were >>> originally community contributions, but additional maintenance or >>> improvements haven't occurred in some time. This is a known issue and will >>> be resolved in time. >>> >>> Second, you are in luck! >>> In the in progress v2.37.0-RC2 release, there are several Cross >>> Language IOs >>> <https://pkg.go.dev/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/xlang/> >>> [1] related >>> to databases, including a JDBCIO >>> <https://pkg.go.dev/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/xlang/jdbcio> >>> [2] >>> and a BigQueryIO >>> <https://pkg.go.dev/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/xlang/bigqueryio> >>> [3]. >>> These should automatically incorporate the Java versions of these >>> transforms into your pipeline, just call and configure the "wrapper" Read >>> and Write calls to use them in your pipeline. Using these does require >>> that a recent edition of Java be installed on your system, but should >>> otherwise not require manual set up of Java jars and similar. >>> >>> Cross language transforms require a runner that supports them, which >>> currently include Flink, Spark, Google Cloud Dataflow, and the Python >>> Portable runner. The default Go Direct runner doesn't support them at this >>> time, and will fail. >>> >>> Robert Burke >>> Beam Go Busybody >>> >>> [1] >>> https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/[email protected] >>> [2] >>> https://pkg.go.dev/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/xlang/jdbcio >>> [3] >>> https://pkg.go.dev/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/xlang/bigqueryio >>> >>> On Sun, Feb 27, 2022 at 10:05 PM Ahmet Altay <[email protected]> wrote: >>> >>>> Forwarding to user@ list. And adding relevant folks who could reply @Pablo >>>> Estrada <[email protected]> @Robert Burke <[email protected]> >>>> >>>> (Folks, if you respond please reply to the list and Abe together. Abe >>>> is having problems with replying back to the user lists. >>>> >>>> ---------- Forwarded message --------- >>>> From: 阿部智大 <[email protected]> >>>> Date: Sun, Feb 27, 2022 at 10:00 PM >>>> Subject: Re: Beam Website Feedback >>>> To: Ahmet Altay <[email protected]> >>>> >>>> >>>> Thank you for your reply. >>>> >>>> The content I tried to send is as follows. >>>> >>>> >>>> ーーー >>>> In the following steps, we are creating a process to write the data on >>>> MySQL to the BQ table. >>>> >>>> 1. Get data from MySQL with "SELECT * FROM table" with Query function >>>> of databaseio package >>>> 2. Write to the BQ table specified by the Write function of the >>>> bigquery package >>>> >>>> However, there is a difference between the number of data that can be >>>> actually acquired by MySQL and the number that is inserted into the BQ >>>> table. >>>> Is this a bug? >>>> >>>> I have tried both Apache Beam versions 2.30.0 and 2.36.0 with the same >>>> results. >>>> >>>> 2022年2月28日(月) 14:58 Ahmet Altay <[email protected]>: >>>> >>>>> Hi, >>>>> >>>>> What is your question? >>>>> >>>>> Ahmet >>>>> >>>>> On Sun, Feb 27, 2022 at 9:56 PM 阿部智大 <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Beam Users, >>>>>> >>>>>> >>>>>> I'm a Japanese developer. >>>>>> >>>>>> I have a question about using the Go SDK, so I want to send an email >>>>>> to [email protected], but I can't because MAILER-DAEMON is >>>>>> returned. >>>>>> Could you tell me how to send it? >>>>>> >>>>>> Thanks & Regards >>>>> >>>>>
