Sorry. I upgraded the version to 2.7.0 and still get the same error in jdbcio.
2022年3月15日(火) 4:58 Kyle Weaver <[email protected]>: > > When I access the repositories / orgapachebeam-1254 that I told you, the > 404 not found page is displayed. > > Beam 2.37.0 is now released, so you can use it the normal way. > > On Sun, Mar 6, 2022 at 6:12 PM 阿部智大 <[email protected]> > wrote: > >> Hello Robert, >> >> I'm sorry for the late reply. >> Also, thank you for your reply. >> >> I forgot to send it to user as well. >> It should now be added to the thread. >> >> When I access the repositories / orgapachebeam-1254 that I told you, the >> 404 not found page is displayed. >> >> Does this mean that I had another solution by the time I answered? >> >> Thanks & Regards >> Abe >> >> 2022年3月3日(木) 3:55 Robert Burke <[email protected]>: >> >>> Turns out this is exactly possible, as the Jar download paths have a >>> common suffix >>> https://issues.apache.org/jira/browse/BEAM-14029 has been filed for the >>> convenience override. >>> It's not a 2.37.0 release blocker due to the newness of the auto >>> download and start functionality, it's not a regression, but the >>> convenience override should be handled for 2.38.0 >>> Rather than another flag that will clutter a user's help, we'll be >>> having a variable or a Set function on the appropriate package for users to >>> call (or populated electively with their own flag), and the >>> https://beam.apache.org/blog/validate-beam-release/ will be updated to >>> address this ability for Cross Language transforms. >>> >>> On Wed, Mar 2, 2022 at 9:59 AM Robert Burke <[email protected]> wrote: >>> >>>> In replying to an off thread question from Abe, I realized that my >>>> suggestion doesn't work since the SDK doesn't know how to source/grab the >>>> RC versions of the Cross Language Jars. They end up getting a 404 response >>>> as a result. >>>> >>>> > panic: tried cross-language for >>>> beam:transform:org.apache.beam:schemaio_jdbc_read:v1 against >>>> autojava::sdks:java:extensions:schemaio-expansion-service:runExpansionService >>>> and failed >>>> > expanding external transform >>>> > received non 200 response code, got 404 >>>> >>>> I'll see if we can't improve that kind of error message for future >>>> releases. >>>> >>>> What is the best way for a user to access RC jars? >>>> >>>> I guess one would need to download them from the staging repo for the >>>> RC: >>>> https://repository.apache.org/content/repositories/orgapachebeam-1254/ >>>> >>>> In this case, it's likely one of the jars in here: >>>> >>>> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/ >>>> And specifically >>>> https://repository.apache.org/content/repositories/orgapachebeam-1254/org/apache/beam/beam-sdks-java-extensions-schemaio-expansion-service/2.37.0/beam-sdks-java-extensions-schemaio-expansion-service-2.37.0.jar >>>> >>>> >>>> One should be able to start up the service with the jar, then using the >>>> address and port, provide an `jdbcio.ExpansionServiceRead` option [1] to >>>> the call, so it queries the service you started up instead. >>>> >>>> [1] >>>> https://pkg.go.dev/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/xlang/jdbcio#ExpansionAddrRead >>>> >>>> I've asked the team to look into simplifying this. We should in >>>> principle should get it to work against a specific RC repo with a flag, >>>> rather than forcing the full manual startup. >>>> >>>> Cheers, >>>> Robert Burke >>>> Beam Go Busybody >>>> >>>> >>>> >>>> On Mon, Feb 28, 2022 at 11:09 AM Robert Burke <[email protected]> wrote: >>>> >>>>> Hi Abe! >>>>> >>>>> Thank you for your interest in the Go SDK! I have two responses for >>>>> you. >>>>> >>>>> First, the current native databaseio and bigqueryio transform in the >>>>> Go SDK aren't production ready at this time. They support scaling, which >>>>> may lead to the limits that you're seeing. Outside of the native TextIO >>>>> with the ReadSDF* calls, those old transforms don't scale well. They were >>>>> originally community contributions, but additional maintenance or >>>>> improvements haven't occurred in some time. This is a known issue and will >>>>> be resolved in time. >>>>> >>>>> Second, you are in luck! >>>>> In the in progress v2.37.0-RC2 release, there are several Cross >>>>> Language IOs >>>>> <https://pkg.go.dev/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/xlang/> >>>>> [1] related >>>>> to databases, including a JDBCIO >>>>> <https://pkg.go.dev/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/xlang/jdbcio> >>>>> [2] >>>>> and a BigQueryIO >>>>> <https://pkg.go.dev/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/xlang/bigqueryio> >>>>> [3]. >>>>> These should automatically incorporate the Java versions of these >>>>> transforms into your pipeline, just call and configure the "wrapper" Read >>>>> and Write calls to use them in your pipeline. Using these does require >>>>> that a recent edition of Java be installed on your system, but should >>>>> otherwise not require manual set up of Java jars and similar. >>>>> >>>>> Cross language transforms require a runner that supports them, which >>>>> currently include Flink, Spark, Google Cloud Dataflow, and the Python >>>>> Portable runner. The default Go Direct runner doesn't support them at this >>>>> time, and will fail. >>>>> >>>>> Robert Burke >>>>> Beam Go Busybody >>>>> >>>>> [1] >>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/[email protected] >>>>> [2] >>>>> https://pkg.go.dev/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/xlang/jdbcio >>>>> [3] >>>>> https://pkg.go.dev/github.com/apache/beam/sdks/[email protected]/go/pkg/beam/io/xlang/bigqueryio >>>>> >>>>> On Sun, Feb 27, 2022 at 10:05 PM Ahmet Altay <[email protected]> wrote: >>>>> >>>>>> Forwarding to user@ list. And adding relevant folks who could reply >>>>>> @Pablo >>>>>> Estrada <[email protected]> @Robert Burke <[email protected]> >>>>>> >>>>>> (Folks, if you respond please reply to the list and Abe together. Abe >>>>>> is having problems with replying back to the user lists. >>>>>> >>>>>> ---------- Forwarded message --------- >>>>>> From: 阿部智大 <[email protected]> >>>>>> Date: Sun, Feb 27, 2022 at 10:00 PM >>>>>> Subject: Re: Beam Website Feedback >>>>>> To: Ahmet Altay <[email protected]> >>>>>> >>>>>> >>>>>> Thank you for your reply. >>>>>> >>>>>> The content I tried to send is as follows. >>>>>> >>>>>> >>>>>> ーーー >>>>>> In the following steps, we are creating a process to write the data >>>>>> on MySQL to the BQ table. >>>>>> >>>>>> 1. Get data from MySQL with "SELECT * FROM table" with Query function >>>>>> of databaseio package >>>>>> 2. Write to the BQ table specified by the Write function of the >>>>>> bigquery package >>>>>> >>>>>> However, there is a difference between the number of data that can be >>>>>> actually acquired by MySQL and the number that is inserted into the BQ >>>>>> table. >>>>>> Is this a bug? >>>>>> >>>>>> I have tried both Apache Beam versions 2.30.0 and 2.36.0 with the >>>>>> same results. >>>>>> >>>>>> 2022年2月28日(月) 14:58 Ahmet Altay <[email protected]>: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> What is your question? >>>>>>> >>>>>>> Ahmet >>>>>>> >>>>>>> On Sun, Feb 27, 2022 at 9:56 PM 阿部智大 <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Beam Users, >>>>>>>> >>>>>>>> >>>>>>>> I'm a Japanese developer. >>>>>>>> >>>>>>>> I have a question about using the Go SDK, so I want to send an >>>>>>>> email to [email protected], but I can't because MAILER-DAEMON >>>>>>>> is returned. >>>>>>>> Could you tell me how to send it? >>>>>>>> >>>>>>>> Thanks & Regards >>>>>>> >>>>>>>
