[jira] [Commented] (FLINK-19398) Hive connector fails with IllegalAccessError if submitted as usercode
[ https://issues.apache.org/jira/browse/FLINK-19398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242014#comment-17242014 ] Yun Gao commented on FLINK-19398: - OK, I'll open the PR, very sorry for not noticed the message. > Hive connector fails with IllegalAccessError if submitted as usercode > - > > Key: FLINK-19398 > URL: https://issues.apache.org/jira/browse/FLINK-19398 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.12.0, 1.11.2 >Reporter: Fabian Hueske >Assignee: Yun Gao >Priority: Blocker > Labels: pull-request-available > Fix For: 1.12.0, 1.11.4 > > > Using Flink's Hive connector fails if the dependency is loaded with the user > code classloader with the following exception. > {code:java} > java.lang.IllegalAccessError: tried to access method > org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.(Lorg/apache/flink/core/fs/Path;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketAssigner;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketFactory;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketWriter;Lorg/apache/flink/streaming/api/functions/sink/filesystem/RollingPolicy;ILorg/apache/flink/streaming/api/functions/sink/filesystem/OutputFileConfig;)V > from class > org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder > at > org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder.createBuckets(HadoopPathBasedBulkFormatBuilder.java:127) > > ~[flink-connector-hive_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.table.filesystem.stream.StreamingFileWriter.initializeState(StreamingFileWriter.java:81) > ~[flink-table-blink_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) > [flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) > [flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > {code} > The problem is the constructor of {{Buckets}} with default visibility which > is called from {{HadoopPathBasedBulkFormatBuilder}} . This works as long as > both classes are loaded with the same classloader but when they are loaded in > different classloaders, the access fails. > {{Buckets}} is loaded with the system CL because it is part of > flink-streaming-java. > > To solve this issue, we should change the visibility of the {{Buckets}} > constructor to {{public}}. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19398) Hive connector fails with IllegalAccessError if submitted as usercode
[ https://issues.apache.org/jira/browse/FLINK-19398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241647#comment-17241647 ] Leonard Xu commented on FLINK-19398: [~gaoyunhaii] Could you raise a PR for 1.11 branch and then we can close this clocker after fixed in 1.11 branch ? > Hive connector fails with IllegalAccessError if submitted as usercode > - > > Key: FLINK-19398 > URL: https://issues.apache.org/jira/browse/FLINK-19398 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.12.0, 1.11.2 >Reporter: Fabian Hueske >Assignee: Yun Gao >Priority: Blocker > Labels: pull-request-available > Fix For: 1.12.0, 1.11.4 > > > Using Flink's Hive connector fails if the dependency is loaded with the user > code classloader with the following exception. > {code:java} > java.lang.IllegalAccessError: tried to access method > org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.(Lorg/apache/flink/core/fs/Path;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketAssigner;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketFactory;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketWriter;Lorg/apache/flink/streaming/api/functions/sink/filesystem/RollingPolicy;ILorg/apache/flink/streaming/api/functions/sink/filesystem/OutputFileConfig;)V > from class > org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder > at > org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder.createBuckets(HadoopPathBasedBulkFormatBuilder.java:127) > > ~[flink-connector-hive_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.table.filesystem.stream.StreamingFileWriter.initializeState(StreamingFileWriter.java:81) > ~[flink-table-blink_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) > [flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) > [flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > {code} > The problem is the constructor of {{Buckets}} with default visibility which > is called from {{HadoopPathBasedBulkFormatBuilder}} . This works as long as > both classes are loaded with the same classloader but when they are loaded in > different classloaders, the access fails. > {{Buckets}} is loaded with the system CL because it is part of > flink-streaming-java. > > To solve this issue, we should change the visibility of the {{Buckets}} > constructor to {{public}}. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19398) Hive connector fails with IllegalAccessError if submitted as usercode
[ https://issues.apache.org/jira/browse/FLINK-19398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238457#comment-17238457 ] Jingsong Lee commented on FLINK-19398: -- master (1.12): cb79fd88c85480e1932218c82f2532e9b0dd652c Needs PR for 1.11. > Hive connector fails with IllegalAccessError if submitted as usercode > - > > Key: FLINK-19398 > URL: https://issues.apache.org/jira/browse/FLINK-19398 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.12.0, 1.11.2 >Reporter: Fabian Hueske >Assignee: Yun Gao >Priority: Blocker > Labels: pull-request-available > Fix For: 1.12.0, 1.11.4 > > > Using Flink's Hive connector fails if the dependency is loaded with the user > code classloader with the following exception. > {code:java} > java.lang.IllegalAccessError: tried to access method > org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.(Lorg/apache/flink/core/fs/Path;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketAssigner;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketFactory;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketWriter;Lorg/apache/flink/streaming/api/functions/sink/filesystem/RollingPolicy;ILorg/apache/flink/streaming/api/functions/sink/filesystem/OutputFileConfig;)V > from class > org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder > at > org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder.createBuckets(HadoopPathBasedBulkFormatBuilder.java:127) > > ~[flink-connector-hive_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.table.filesystem.stream.StreamingFileWriter.initializeState(StreamingFileWriter.java:81) > ~[flink-table-blink_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) > [flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) > [flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > {code} > The problem is the constructor of {{Buckets}} with default visibility which > is called from {{HadoopPathBasedBulkFormatBuilder}} . This works as long as > both classes are loaded with the same classloader but when they are loaded in > different classloaders, the access fails. > {{Buckets}} is loaded with the system CL because it is part of > flink-streaming-java. > > To solve this issue, we should change the visibility of the {{Buckets}} > constructor to {{public}}. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19398) Hive connector fails with IllegalAccessError if submitted as usercode
[ https://issues.apache.org/jira/browse/FLINK-19398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237774#comment-17237774 ] Dian Fu commented on FLINK-19398: - [~lzljs3620320] Could you help to review the PR submitted by [~gaoyunhaii]? > Hive connector fails with IllegalAccessError if submitted as usercode > - > > Key: FLINK-19398 > URL: https://issues.apache.org/jira/browse/FLINK-19398 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.12.0, 1.11.2 >Reporter: Fabian Hueske >Assignee: Yun Gao >Priority: Blocker > Labels: pull-request-available > Fix For: 1.12.0, 1.11.4 > > > Using Flink's Hive connector fails if the dependency is loaded with the user > code classloader with the following exception. > {code:java} > java.lang.IllegalAccessError: tried to access method > org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.(Lorg/apache/flink/core/fs/Path;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketAssigner;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketFactory;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketWriter;Lorg/apache/flink/streaming/api/functions/sink/filesystem/RollingPolicy;ILorg/apache/flink/streaming/api/functions/sink/filesystem/OutputFileConfig;)V > from class > org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder > at > org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder.createBuckets(HadoopPathBasedBulkFormatBuilder.java:127) > > ~[flink-connector-hive_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.table.filesystem.stream.StreamingFileWriter.initializeState(StreamingFileWriter.java:81) > ~[flink-table-blink_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) > [flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) > [flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > {code} > The problem is the constructor of {{Buckets}} with default visibility which > is called from {{HadoopPathBasedBulkFormatBuilder}} . This works as long as > both classes are loaded with the same classloader but when they are loaded in > different classloaders, the access fails. > {{Buckets}} is loaded with the system CL because it is part of > flink-streaming-java. > > To solve this issue, we should change the visibility of the {{Buckets}} > constructor to {{public}}. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19398) Hive connector fails with IllegalAccessError if submitted as usercode
[ https://issues.apache.org/jira/browse/FLINK-19398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237530#comment-17237530 ] Dawid Wysakowicz commented on FLINK-19398: -- This not the only case when Hive does not work if submitted from the user code. I had problems when trying to submit a query from sql-client with the flink-sql-connector-hive jar passed with the {{-j}} option. {code} org.apache.flink.table.client.gateway.SqlExecutionException: Invalid SQL query. at org.apache.flink.table.client.gateway.local.LocalExecutor.executeQueryInternal(LocalExecutor.java:527) ~[flink-sql-client_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at org.apache.flink.table.client.gateway.local.LocalExecutor.executeQuery(LocalExecutor.java:365) ~[flink-sql-client_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at org.apache.flink.table.client.cli.CliClient.callSelect(CliClient.java:634) ~[flink-sql-client_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:324) ~[flink-sql-client_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at java.util.Optional.ifPresent(Optional.java:159) [?:1.8.0_252] at org.apache.flink.table.client.cli.CliClient.open(CliClient.java:216) [flink-sql-client_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:141) [flink-sql-client_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at org.apache.flink.table.client.SqlClient.start(SqlClient.java:114) [flink-sql-client_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at org.apache.flink.table.client.SqlClient.main(SqlClient.java:196) [flink-sql-client_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] Caused by: org.apache.flink.table.catalog.exceptions.CatalogException: Failed to create Hive Metastore client at org.apache.flink.table.catalog.hive.client.HiveShimV230.getHiveMetastoreClient(HiveShimV230.java:52) ~[?:?] at org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:245) ~[?:?] at org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:76) ~[?:?] at org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35) ~[?:?] at org.apache.flink.connectors.hive.util.HivePartitionUtils.getAllPartitions(HivePartitionUtils.java:110) ~[?:?] at org.apache.flink.connectors.hive.HiveTableSource.getDataStream(HiveTableSource.java:134) ~[?:?] at org.apache.flink.connectors.hive.HiveTableSource$1.produceDataStream(HiveTableSource.java:120) ~[?:?] at org.apache.flink.table.planner.plan.nodes.common.CommonPhysicalTableSourceScan.createSourceTransformation(CommonPhysicalTableSourceScan.scala:88) ~[flink-table-blink_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at org.apache.flink.table.planner.plan.nodes.physical.batch.BatchExecTableSourceScan.translateToPlanInternal(BatchExecTableSourceScan.scala:94) ~[flink-table-blink_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at org.apache.flink.table.planner.plan.nodes.physical.batch.BatchExecTableSourceScan.translateToPlanInternal(BatchExecTableSourceScan.scala:44) ~[flink-table-blink_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:59) ~[flink-table-blink_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at org.apache.flink.table.planner.plan.nodes.physical.batch.BatchExecTableSourceScan.translateToPlan(BatchExecTableSourceScan.scala:44) ~[flink-table-blink_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at org.apache.flink.table.planner.plan.nodes.physical.batch.BatchExecLegacySink.translateToTransformation(BatchExecLegacySink.scala:129) ~[flink-table-blink_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at org.apache.flink.table.planner.plan.nodes.physical.batch.BatchExecLegacySink.translateToPlanInternal(BatchExecLegacySink.scala:95) ~[flink-table-blink_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at org.apache.flink.table.planner.plan.nodes.physical.batch.BatchExecLegacySink.translateToPlanInternal(BatchExecLegacySink.scala:48) ~[flink-table-blink_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:59) ~[flink-table-blink_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at org.apache.flink.table.planner.plan.nodes.physical.batch.BatchExecLegacySink.translateToPlan(BatchExecLegacySink.scala:48) ~[flink-table-blink_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at org.apache.flink.table.planner.delegation.BatchPlanner$$anonfun$translateToPlan$1.apply(BatchPlanner.scala:86) ~[flink-table-blink_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
[jira] [Commented] (FLINK-19398) Hive connector fails with IllegalAccessError if submitted as usercode
[ https://issues.apache.org/jira/browse/FLINK-19398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234197#comment-17234197 ] Xintong Song commented on FLINK-19398: -- I'm deferring the fix version to 1.11.4 for now. Will change it back if it makes the 1.11.3 release in the end. > Hive connector fails with IllegalAccessError if submitted as usercode > - > > Key: FLINK-19398 > URL: https://issues.apache.org/jira/browse/FLINK-19398 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.12.0, 1.11.2 >Reporter: Fabian Hueske >Assignee: Yun Gao >Priority: Critical > Labels: pull-request-available > Fix For: 1.11.4 > > > Using Flink's Hive connector fails if the dependency is loaded with the user > code classloader with the following exception. > {code:java} > java.lang.IllegalAccessError: tried to access method > org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.(Lorg/apache/flink/core/fs/Path;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketAssigner;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketFactory;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketWriter;Lorg/apache/flink/streaming/api/functions/sink/filesystem/RollingPolicy;ILorg/apache/flink/streaming/api/functions/sink/filesystem/OutputFileConfig;)V > from class > org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder > at > org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder.createBuckets(HadoopPathBasedBulkFormatBuilder.java:127) > > ~[flink-connector-hive_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.table.filesystem.stream.StreamingFileWriter.initializeState(StreamingFileWriter.java:81) > ~[flink-table-blink_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) > [flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) > [flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > {code} > The problem is the constructor of {{Buckets}} with default visibility which > is called from {{HadoopPathBasedBulkFormatBuilder}} . This works as long as > both classes are loaded with the same classloader but when they are loaded in > different classloaders, the access fails. > {{Buckets}} is loaded with the system CL because it is part of > flink-streaming-java. > > To solve this issue, we should change the visibility of the {{Buckets}} > constructor to {{public}}. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19398) Hive connector fails with IllegalAccessError if submitted as usercode
[ https://issues.apache.org/jira/browse/FLINK-19398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233430#comment-17233430 ] Yun Gao commented on FLINK-19398: - Hi I opened a PR for this issue. Currently I leave _HadoopPathBasedBulkFormatBuilder_ in the original package and add a specialized test to verify that it could be used normally in the user classloader. I did so since logically _HadoopPathBasedBulkFormatBuilder_ could be logically used by other users and it might be better to keep compatible, although theoretically it should not be frequent. > Hive connector fails with IllegalAccessError if submitted as usercode > - > > Key: FLINK-19398 > URL: https://issues.apache.org/jira/browse/FLINK-19398 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.12.0, 1.11.2 >Reporter: Fabian Hueske >Assignee: Yun Gao >Priority: Critical > Labels: pull-request-available > Fix For: 1.11.3 > > > Using Flink's Hive connector fails if the dependency is loaded with the user > code classloader with the following exception. > {code:java} > java.lang.IllegalAccessError: tried to access method > org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.(Lorg/apache/flink/core/fs/Path;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketAssigner;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketFactory;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketWriter;Lorg/apache/flink/streaming/api/functions/sink/filesystem/RollingPolicy;ILorg/apache/flink/streaming/api/functions/sink/filesystem/OutputFileConfig;)V > from class > org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder > at > org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder.createBuckets(HadoopPathBasedBulkFormatBuilder.java:127) > > ~[flink-connector-hive_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.table.filesystem.stream.StreamingFileWriter.initializeState(StreamingFileWriter.java:81) > ~[flink-table-blink_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) > [flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) > [flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > {code} > The problem is the constructor of {{Buckets}} with default visibility which > is called from {{HadoopPathBasedBulkFormatBuilder}} . This works as long as > both classes are loaded with the same classloader but when they are loaded in > different classloaders, the access fails. > {{Buckets}} is loaded with the system CL because it is part of > flink-streaming-java. > > To solve this issue, we should change the visibility of the {{Buckets}} > constructor to {{public}}. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19398) Hive connector fails with IllegalAccessError if submitted as usercode
[ https://issues.apache.org/jira/browse/FLINK-19398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216436#comment-17216436 ] Yun Gao commented on FLINK-19398: - I'll have a look at this issue. Very sorry for the inconvenience brought. > Hive connector fails with IllegalAccessError if submitted as usercode > - > > Key: FLINK-19398 > URL: https://issues.apache.org/jira/browse/FLINK-19398 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.12.0, 1.11.2 >Reporter: Fabian Hueske >Assignee: Yun Gao >Priority: Critical > Fix For: 1.11.3 > > > Using Flink's Hive connector fails if the dependency is loaded with the user > code classloader with the following exception. > {code:java} > java.lang.IllegalAccessError: tried to access method > org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.(Lorg/apache/flink/core/fs/Path;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketAssigner;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketFactory;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketWriter;Lorg/apache/flink/streaming/api/functions/sink/filesystem/RollingPolicy;ILorg/apache/flink/streaming/api/functions/sink/filesystem/OutputFileConfig;)V > from class > org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder > at > org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder.createBuckets(HadoopPathBasedBulkFormatBuilder.java:127) > > ~[flink-connector-hive_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.table.filesystem.stream.StreamingFileWriter.initializeState(StreamingFileWriter.java:81) > ~[flink-table-blink_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) > [flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) > [flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > {code} > The problem is the constructor of {{Buckets}} with default visibility which > is called from {{HadoopPathBasedBulkFormatBuilder}} . This works as long as > both classes are loaded with the same classloader but when they are loaded in > different classloaders, the access fails. > {{Buckets}} is loaded with the system CL because it is part of > flink-streaming-java. > > To solve this issue, we should change the visibility of the {{Buckets}} > constructor to {{public}}. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19398) Hive connector fails with IllegalAccessError if submitted as usercode
[ https://issues.apache.org/jira/browse/FLINK-19398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216434#comment-17216434 ] Jingsong Lee commented on FLINK-19398: -- CC: [~gaoyunhaii] > Hive connector fails with IllegalAccessError if submitted as usercode > - > > Key: FLINK-19398 > URL: https://issues.apache.org/jira/browse/FLINK-19398 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.12.0, 1.11.2 >Reporter: Fabian Hueske >Priority: Critical > Fix For: 1.11.3 > > > Using Flink's Hive connector fails if the dependency is loaded with the user > code classloader with the following exception. > {code:java} > java.lang.IllegalAccessError: tried to access method > org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.(Lorg/apache/flink/core/fs/Path;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketAssigner;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketFactory;Lorg/apache/flink/streaming/api/functions/sink/filesystem/BucketWriter;Lorg/apache/flink/streaming/api/functions/sink/filesystem/RollingPolicy;ILorg/apache/flink/streaming/api/functions/sink/filesystem/OutputFileConfig;)V > from class > org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder > at > org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder.createBuckets(HadoopPathBasedBulkFormatBuilder.java:127) > > ~[flink-connector-hive_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.table.filesystem.stream.StreamingFileWriter.initializeState(StreamingFileWriter.java:81) > ~[flink-table-blink_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528) > ~[flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) > [flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) > [flink-dist_2.12-1.11.2-stream2-SNAPSHOT.jar:1.11.2-stream2-SNAPSHOT] > {code} > The problem is the constructor of {{Buckets}} with default visibility which > is called from {{HadoopPathBasedBulkFormatBuilder}} . This works as long as > both classes are loaded with the same classloader but when they are loaded in > different classloaders, the access fails. > {{Buckets}} is loaded with the system CL because it is part of > flink-streaming-java. > > To solve this issue, we should change the visibility of the {{Buckets}} > constructor to {{public}}. > -- This message was sent by Atlassian Jira (v8.3.4#803005)