Hi, XQ
Thank you for your reply.
I tried to implement your suggestion by running `mvn compile exec:java
-Dexec.args="--add-opens=java.base/java.lang=ALL-UNNAMED"` or by editing a
plugin in the pom.xml as shown below.
However, I was unable to resolve this issue. As I am very new to Beam and
the OSS community, I am unsure of what the next step should be. Should I
create an issue on Github, or could you point out where I might be missing
something?
I appreciate your help.
Best regards,
Kazuha
```pom.xml
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<goals>
<goal>java</goal>
</goals>
</execution>
</executions>
<configuration>
<mainClass>com.example.MainClass</mainClass>
<arguments>
<argument>-Dexec.args="--add-opens=java.base/java.lang=ALL-UNNAMED"</argument>
</arguments>
</configuration>
</plugin>
```
On Tue, Jun 4, 2024 at 8:14 AM XQ Hu via user <[email protected]> wrote:
> Probably related to the strict encapsulation that is enforced with Java
> 21.
> Use `--add-opens=java.base/java.lang=ALL-UNNAMED` as the JVM flag could be
> a temporary workaround.
>
> On Mon, Jun 3, 2024 at 3:04 AM 田中万葉 <[email protected]> wrote:
>
>> Hi all,
>>
>> I encountered an UnsupportedOperationException when using Java 21 and the
>> BigQuery Storage Write API in a Beam pipeline by using
>> ".withMethod(BigQueryIO.Write.Method.STORAGE_WRITE_API));"
>>
>> Having read issue #28120[1] and understanding that Beam version 2.52.0 or
>> later supports Java 21 as a runtime, I wonder why such an error happens.
>>
>> I found there are two workarounds, but the Storage Write API is a more
>> preferable way to insert data into BigQuery, so I'd like to find a
>> solution.
>>
>> 1. One workaround is to switch from Java 21 to Java 17(openjdk version
>> "17.0.10" 2024-01-16). By changing the <maven.compiler.source> and
>> <maven.compiler.target> in the pom.xml file (i.e., without modifying
>> App.java itself), the pipeline successfully writes data to my destination
>> table on BigQuery. It seems Java 17 and BigQuery Storage Write API works
>> fine.
>> 2. The other workaround is to change insert method. I tried the BigQuery
>> legacy streaming API(
>> https://cloud.google.com/bigquery/docs/streaming-data-into-bigquery )
>> instead of the Storage Write API. Even though I still used Java 21, when I
>> changed my code to
>> .withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS));, I did not
>> encounter the error.
>>
>> So I faced the error only when using Java 21 and BigQuery Storage Write
>> API.
>>
>> I uploaded the code below to reproduce. Could you please inform me how to
>> handle this issue?
>> https://github.com/cloud-ace/min-reproduce
>>
>> My Environment
>> - OS
>> - Ubuntu 22.04
>> - Mac OS Sonoma(14.3.1)
>> - beam 2.53.0, 2.54.0
>> - openjdk version "21.0.2" 2024-01-16
>> - maven 3.9.6
>> - DirectRunner
>>
>> Thanks,
>>
>> Kazuha
>>
>> [1]: https://github.com/apache/beam/issues/28120
>>
>> Here is the detailed error message.
>>
>> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
>> java.lang.UnsupportedOperationException: Cannot define class using
>> reflection: Unable to make protected java.lang.Package
>> java.lang.ClassLoader.getPackage(java.lang.String) accessible: module
>> java.base does not "opens java.lang" to unnamed module @116d5dff
>>
>> Caused by: java.lang.UnsupportedOperationException: Cannot define class
>> using reflection: Unable to make protected java.lang.Package
>> java.lang.ClassLoader.getPackage(java.lang.String) accessible: module
>> java.base does not "opens java.lang" to unnamed module @116d5dff
>> at
>> net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection$Dispatcher$Initializable$Unavailable.defineClass
>> (ClassInjector.java:472)
>> at
>> net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection.injectRaw
>> (ClassInjector.java:284)
>> at net.bytebuddy.dynamic.loading.ClassInjector$AbstractBase.inject
>> (ClassInjector.java:118)
>> at
>> net.bytebuddy.dynamic.loading.ClassLoadingStrategy$Default$InjectionDispatcher.load
>> (ClassLoadingStrategy.java:241)
>> at net.bytebuddy.dynamic.loading.ClassLoadingStrategy$Default.load
>> (ClassLoadingStrategy.java:148)
>> at net.bytebuddy.dynamic.TypeResolutionStrategy$Passive.initialize
>> (TypeResolutionStrategy.java:101)
>> at net.bytebuddy.dynamic.DynamicType$Default$Unloaded.load
>> (DynamicType.java:6317)
>> at
>> org.apache.beam.sdk.schemas.utils.AutoValueUtils.createBuilderCreator
>> (AutoValueUtils.java:247)
>> at org.apache.beam.sdk.schemas.utils.AutoValueUtils.getBuilderCreator
>> (AutoValueUtils.java:225)
>> at org.apache.beam.sdk.schemas.AutoValueSchema.schemaTypeCreator
>> (AutoValueSchema.java:122)
>> at org.apache.beam.sdk.schemas.CachingFactory.create
>> (CachingFactory.java:56)
>> at org.apache.beam.sdk.schemas.FromRowUsingCreator.apply
>> (FromRowUsingCreator.java:94)
>> at org.apache.beam.sdk.schemas.FromRowUsingCreator.apply
>> (FromRowUsingCreator.java:45)
>> at org.apache.beam.sdk.schemas.SchemaCoder.decode
>> (SchemaCoder.java:126)
>> at org.apache.beam.sdk.coders.Coder.decode (Coder.java:159)
>> at org.apache.beam.sdk.coders.KvCoder.decode (KvCoder.java:84)
>> at org.apache.beam.sdk.coders.KvCoder.decode (KvCoder.java:37)
>> at org.apache.beam.sdk.util.CoderUtils.decodeFromSafeStream
>> (CoderUtils.java:142)
>> at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray
>> (CoderUtils.java:102)
>> at org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray
>> (CoderUtils.java:96)
>> at org.apache.beam.sdk.util.CoderUtils.clone (CoderUtils.java:168)
>> at
>> org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.<init>
>> (MutationDetectors.java:118)
>> at org.apache.beam.sdk.util.MutationDetectors.forValueWithCoder
>> (MutationDetectors.java:49)
>> at
>> org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.add
>> (ImmutabilityCheckingBundleFactory.java:115)
>> at
>> org.apache.beam.runners.direct.ParDoEvaluator$BundleOutputManager.output
>> (ParDoEvaluator.java:305)
>> at
>> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.outputWindowedValue
>> (SimpleDoFnRunner.java:275)
>> at
>> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.access$900
>> (SimpleDoFnRunner.java:85)
>> at
>> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output
>> (SimpleDoFnRunner.java:423)
>> at
>> org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output
>> (DoFnOutputReceivers.java:76)
>> at
>> org.apache.beam.sdk.io.gcp.bigquery.StorageApiConvertMessages$ConvertMessagesDoFn.processElement
>> (StorageApiConvertMessages.java:161)
>>
>> ＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝
>> Cloud Ace, Inc.
>> Engineering Division/System Development Section/Data Solution Department
>> TANAKA Kazuha 田中 万葉
>>
>> 〒100-0004 東京都千代田区大手町1丁目7番2号 東京サンケイビル26階
>>
>> TEL: 03-6280-5939 FAX: 03-6800-3954
>> Email: [email protected]
>> URL: https://www.cloud-ace.jp/
>> ＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝
>>
>
