[jira] [Updated] (FLINK-20714) Hive delegation token is not obtained when using `kinit` to submit Yarn per-job
[ https://issues.apache.org/jira/browse/FLINK-20714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-20714: --- Labels: auto-deprioritized-major auto-deprioritized-minor keberos (was: auto-deprioritized-major keberos stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job > > > Key: FLINK-20714 > URL: https://issues.apache.org/jira/browse/FLINK-20714 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.11.2, 1.11.3, 1.12.0 > Environment: Flink 1.11.2 on Yarn >Reporter: jackwangcs >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor, > keberos > > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job. > In YarnClusterDescriptor, it calls org.apache.flink.yarn.Utils#setTokensFor > to obtain tokens for the job. But setTokensFor only obtains HDFS and HBase > tokens currently, since the Hive integration is supported, the Hive > delegation should be obtained also. > Otherwise, it will throw the following error when it tries to connect to > Hive metastore: > {code:java} > Caused by: MetaException(message:Could not connect to meta store using any of > the URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failedCaused > by: MetaException(message:Could not connect to meta store using any of the > URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failed at > org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) > at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:187) > at > org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:71) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:74) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:68) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:65) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:43) > at > org.apache.flink.table.filesystem.PartitionLoader.(PartitionLoader.java:61) > at > org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:97) > at > org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95) > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-20714) Hive delegation token is not obtained when using `kinit` to submit Yarn per-job
[ https://issues.apache.org/jira/browse/FLINK-20714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-20714: --- Labels: auto-deprioritized-major keberos stale-minor (was: auto-deprioritized-major keberos) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job > > > Key: FLINK-20714 > URL: https://issues.apache.org/jira/browse/FLINK-20714 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.11.2, 1.11.3, 1.12.0 > Environment: Flink 1.11.2 on Yarn >Reporter: jackwangcs >Priority: Minor > Labels: auto-deprioritized-major, keberos, stale-minor > > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job. > In YarnClusterDescriptor, it calls org.apache.flink.yarn.Utils#setTokensFor > to obtain tokens for the job. But setTokensFor only obtains HDFS and HBase > tokens currently, since the Hive integration is supported, the Hive > delegation should be obtained also. > Otherwise, it will throw the following error when it tries to connect to > Hive metastore: > {code:java} > Caused by: MetaException(message:Could not connect to meta store using any of > the URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failedCaused > by: MetaException(message:Could not connect to meta store using any of the > URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failed at > org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) > at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:187) > at > org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:71) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:74) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:68) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:65) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:43) > at > org.apache.flink.table.filesystem.PartitionLoader.(PartitionLoader.java:61) > at > org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:97) > at > org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95) > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-20714) Hive delegation token is not obtained when using `kinit` to submit Yarn per-job
[ https://issues.apache.org/jira/browse/FLINK-20714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-20714: --- Labels: auto-deprioritized-major keberos (was: keberos stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job > > > Key: FLINK-20714 > URL: https://issues.apache.org/jira/browse/FLINK-20714 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.11.2, 1.11.3, 1.12.0 > Environment: Flink 1.11.2 on Yarn >Reporter: jackwangcs >Priority: Minor > Labels: auto-deprioritized-major, keberos > > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job. > In YarnClusterDescriptor, it calls org.apache.flink.yarn.Utils#setTokensFor > to obtain tokens for the job. But setTokensFor only obtains HDFS and HBase > tokens currently, since the Hive integration is supported, the Hive > delegation should be obtained also. > Otherwise, it will throw the following error when it tries to connect to > Hive metastore: > {code:java} > Caused by: MetaException(message:Could not connect to meta store using any of > the URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failedCaused > by: MetaException(message:Could not connect to meta store using any of the > URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failed at > org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) > at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:187) > at > org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:71) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:74) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:68) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:65) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:43) > at > org.apache.flink.table.filesystem.PartitionLoader.(PartitionLoader.java:61) > at > org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:97) > at > org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20714) Hive delegation token is not obtained when using `kinit` to submit Yarn per-job
[ https://issues.apache.org/jira/browse/FLINK-20714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-20714: --- Labels: keberos stale-major (was: keberos) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job > > > Key: FLINK-20714 > URL: https://issues.apache.org/jira/browse/FLINK-20714 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.11.2, 1.11.3, 1.12.0 > Environment: Flink 1.11.2 on Yarn >Reporter: jackwangcs >Priority: Major > Labels: keberos, stale-major > > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job. > In YarnClusterDescriptor, it calls org.apache.flink.yarn.Utils#setTokensFor > to obtain tokens for the job. But setTokensFor only obtains HDFS and HBase > tokens currently, since the Hive integration is supported, the Hive > delegation should be obtained also. > Otherwise, it will throw the following error when it tries to connect to > Hive metastore: > {code:java} > Caused by: MetaException(message:Could not connect to meta store using any of > the URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failedCaused > by: MetaException(message:Could not connect to meta store using any of the > URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failed at > org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) > at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:187) > at > org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:71) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:74) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:68) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:65) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:43) > at > org.apache.flink.table.filesystem.PartitionLoader.(PartitionLoader.java:61) > at > org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:97) > at > org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20714) Hive delegation token is not obtained when using `kinit` to submit Yarn per-job
[ https://issues.apache.org/jira/browse/FLINK-20714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Knauf updated FLINK-20714: - Fix Version/s: (was: 1.12.4) (was: 1.14.0) (was: 1.11.4) > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job > > > Key: FLINK-20714 > URL: https://issues.apache.org/jira/browse/FLINK-20714 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.11.2, 1.11.3, 1.12.0 > Environment: Flink 1.11.2 on Yarn >Reporter: jackwangcs >Priority: Major > Labels: keberos > > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job. > In YarnClusterDescriptor, it calls org.apache.flink.yarn.Utils#setTokensFor > to obtain tokens for the job. But setTokensFor only obtains HDFS and HBase > tokens currently, since the Hive integration is supported, the Hive > delegation should be obtained also. > Otherwise, it will throw the following error when it tries to connect to > Hive metastore: > {code:java} > Caused by: MetaException(message:Could not connect to meta store using any of > the URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failedCaused > by: MetaException(message:Could not connect to meta store using any of the > URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failed at > org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) > at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:187) > at > org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:71) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:74) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:68) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:65) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:43) > at > org.apache.flink.table.filesystem.PartitionLoader.(PartitionLoader.java:61) > at > org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:97) > at > org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20714) Hive delegation token is not obtained when using `kinit` to submit Yarn per-job
[ https://issues.apache.org/jira/browse/FLINK-20714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz updated FLINK-20714: - Fix Version/s: (was: 1.13.0) 1.14.0 > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job > > > Key: FLINK-20714 > URL: https://issues.apache.org/jira/browse/FLINK-20714 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.11.2, 1.11.3, 1.12.0 > Environment: Flink 1.11.2 on Yarn >Reporter: jackwangcs >Priority: Major > Labels: keberos > Fix For: 1.11.4, 1.14.0, 1.12.4 > > > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job. > In YarnClusterDescriptor, it calls org.apache.flink.yarn.Utils#setTokensFor > to obtain tokens for the job. But setTokensFor only obtains HDFS and HBase > tokens currently, since the Hive integration is supported, the Hive > delegation should be obtained also. > Otherwise, it will throw the following error when it tries to connect to > Hive metastore: > {code:java} > Caused by: MetaException(message:Could not connect to meta store using any of > the URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failedCaused > by: MetaException(message:Could not connect to meta store using any of the > URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failed at > org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) > at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:187) > at > org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:71) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:74) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:68) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:65) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:43) > at > org.apache.flink.table.filesystem.PartitionLoader.(PartitionLoader.java:61) > at > org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:97) > at > org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20714) Hive delegation token is not obtained when using `kinit` to submit Yarn per-job
[ https://issues.apache.org/jira/browse/FLINK-20714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise updated FLINK-20714: Fix Version/s: (was: 1.12.3) 1.12.4 > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job > > > Key: FLINK-20714 > URL: https://issues.apache.org/jira/browse/FLINK-20714 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.11.2, 1.11.3, 1.12.0 > Environment: Flink 1.11.2 on Yarn >Reporter: jackwangcs >Priority: Major > Labels: keberos > Fix For: 1.11.4, 1.13.0, 1.12.4 > > > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job. > In YarnClusterDescriptor, it calls org.apache.flink.yarn.Utils#setTokensFor > to obtain tokens for the job. But setTokensFor only obtains HDFS and HBase > tokens currently, since the Hive integration is supported, the Hive > delegation should be obtained also. > Otherwise, it will throw the following error when it tries to connect to > Hive metastore: > {code:java} > Caused by: MetaException(message:Could not connect to meta store using any of > the URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failedCaused > by: MetaException(message:Could not connect to meta store using any of the > URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failed at > org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) > at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:187) > at > org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:71) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:74) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:68) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:65) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:43) > at > org.apache.flink.table.filesystem.PartitionLoader.(PartitionLoader.java:61) > at > org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:97) > at > org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20714) Hive delegation token is not obtained when using `kinit` to submit Yarn per-job
[ https://issues.apache.org/jira/browse/FLINK-20714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-20714: -- Priority: Major (was: Critical) > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job > > > Key: FLINK-20714 > URL: https://issues.apache.org/jira/browse/FLINK-20714 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.11.2, 1.11.3, 1.12.0 > Environment: Flink 1.11.2 on Yarn >Reporter: jackwangcs >Priority: Major > Labels: keberos > Fix For: 1.11.4, 1.13.0, 1.12.3 > > > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job. > In YarnClusterDescriptor, it calls org.apache.flink.yarn.Utils#setTokensFor > to obtain tokens for the job. But setTokensFor only obtains HDFS and HBase > tokens currently, since the Hive integration is supported, the Hive > delegation should be obtained also. > Otherwise, it will throw the following error when it tries to connect to > Hive metastore: > {code:java} > Caused by: MetaException(message:Could not connect to meta store using any of > the URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failedCaused > by: MetaException(message:Could not connect to meta store using any of the > URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failed at > org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) > at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:187) > at > org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:71) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:74) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:68) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:65) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:43) > at > org.apache.flink.table.filesystem.PartitionLoader.(PartitionLoader.java:61) > at > org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:97) > at > org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20714) Hive delegation token is not obtained when using `kinit` to submit Yarn per-job
[ https://issues.apache.org/jira/browse/FLINK-20714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Khachatryan updated FLINK-20714: -- Fix Version/s: 1.12.3 > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job > > > Key: FLINK-20714 > URL: https://issues.apache.org/jira/browse/FLINK-20714 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.11.2, 1.11.3, 1.12.0 > Environment: Flink 1.11.2 on Yarn >Reporter: jackwangcs >Priority: Critical > Labels: keberos > Fix For: 1.11.4, 1.13.0, 1.12.3 > > > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job. > In YarnClusterDescriptor, it calls org.apache.flink.yarn.Utils#setTokensFor > to obtain tokens for the job. But setTokensFor only obtains HDFS and HBase > tokens currently, since the Hive integration is supported, the Hive > delegation should be obtained also. > Otherwise, it will throw the following error when it tries to connect to > Hive metastore: > {code:java} > Caused by: MetaException(message:Could not connect to meta store using any of > the URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failedCaused > by: MetaException(message:Could not connect to meta store using any of the > URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failed at > org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) > at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:187) > at > org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:71) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:74) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:68) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:65) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:43) > at > org.apache.flink.table.filesystem.PartitionLoader.(PartitionLoader.java:61) > at > org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:97) > at > org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20714) Hive delegation token is not obtained when using `kinit` to submit Yarn per-job
[ https://issues.apache.org/jira/browse/FLINK-20714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Khachatryan updated FLINK-20714: -- Fix Version/s: (was: 1.12.2) > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job > > > Key: FLINK-20714 > URL: https://issues.apache.org/jira/browse/FLINK-20714 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.11.2, 1.11.3, 1.12.0 > Environment: Flink 1.11.2 on Yarn >Reporter: jackwangcs >Priority: Critical > Labels: keberos > Fix For: 1.11.4, 1.13.0 > > > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job. > In YarnClusterDescriptor, it calls org.apache.flink.yarn.Utils#setTokensFor > to obtain tokens for the job. But setTokensFor only obtains HDFS and HBase > tokens currently, since the Hive integration is supported, the Hive > delegation should be obtained also. > Otherwise, it will throw the following error when it tries to connect to > Hive metastore: > {code:java} > Caused by: MetaException(message:Could not connect to meta store using any of > the URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failedCaused > by: MetaException(message:Could not connect to meta store using any of the > URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failed at > org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) > at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:187) > at > org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:71) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:74) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:68) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:65) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:43) > at > org.apache.flink.table.filesystem.PartitionLoader.(PartitionLoader.java:61) > at > org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:97) > at > org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20714) Hive delegation token is not obtained when using `kinit` to submit Yarn per-job
[ https://issues.apache.org/jira/browse/FLINK-20714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-20714: - Fix Version/s: (was: 1.12.1) 1.12.2 > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job > > > Key: FLINK-20714 > URL: https://issues.apache.org/jira/browse/FLINK-20714 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.12.0, 1.11.2, 1.11.3 > Environment: Flink 1.11.2 on Yarn >Reporter: jackwangcs >Priority: Critical > Labels: keberos > Fix For: 1.13.0, 1.11.4, 1.12.2 > > > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job. > In YarnClusterDescriptor, it calls org.apache.flink.yarn.Utils#setTokensFor > to obtain tokens for the job. But setTokensFor only obtains HDFS and HBase > tokens currently, since the Hive integration is supported, the Hive > delegation should be obtained also. > Otherwise, it will throw the following error when it tries to connect to > Hive metastore: > {code:java} > Caused by: MetaException(message:Could not connect to meta store using any of > the URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failedCaused > by: MetaException(message:Could not connect to meta store using any of the > URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failed at > org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) > at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:187) > at > org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:71) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:74) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:68) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:65) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:43) > at > org.apache.flink.table.filesystem.PartitionLoader.(PartitionLoader.java:61) > at > org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:97) > at > org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20714) Hive delegation token is not obtained when using `kinit` to submit Yarn per-job
[ https://issues.apache.org/jira/browse/FLINK-20714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-20714: -- Fix Version/s: 1.12.1 1.11.4 1.13.0 > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job > > > Key: FLINK-20714 > URL: https://issues.apache.org/jira/browse/FLINK-20714 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.12.0, 1.11.2, 1.11.3 > Environment: Flink 1.11.2 on Yarn >Reporter: jackwangcs >Priority: Critical > Labels: keberos > Fix For: 1.13.0, 1.11.4, 1.12.1 > > > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job. > In YarnClusterDescriptor, it calls org.apache.flink.yarn.Utils#setTokensFor > to obtain tokens for the job. But setTokensFor only obtains HDFS and HBase > tokens currently, since the Hive integration is supported, the Hive > delegation should be obtained also. > Otherwise, it will throw the following error when it tries to connect to > Hive metastore: > {code:java} > Caused by: MetaException(message:Could not connect to meta store using any of > the URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failedCaused > by: MetaException(message:Could not connect to meta store using any of the > URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failed at > org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) > at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:187) > at > org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:71) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:74) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:68) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:65) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:43) > at > org.apache.flink.table.filesystem.PartitionLoader.(PartitionLoader.java:61) > at > org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:97) > at > org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20714) Hive delegation token is not obtained when using `kinit` to submit Yarn per-job
[ https://issues.apache.org/jira/browse/FLINK-20714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jackwangcs updated FLINK-20714: --- Issue Type: Improvement (was: Bug) > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job > > > Key: FLINK-20714 > URL: https://issues.apache.org/jira/browse/FLINK-20714 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.12.0, 1.11.2, 1.11.3 > Environment: Flink 1.11.2 on Yarn >Reporter: jackwangcs >Priority: Critical > Labels: keberos > > Hive delegation token is not obtained when using `kinit` to submit Yarn > per-job. > In YarnClusterDescriptor, it calls org.apache.flink.yarn.Utils#setTokensFor > to obtain tokens for the job. But setTokensFor only obtains HDFS and HBase > tokens currently, since the Hive integration is supported, the Hive > delegation should be obtained also. > Otherwise, it will throw the following error when it tries to connect to > Hive metastore: > {code:java} > Caused by: MetaException(message:Could not connect to meta store using any of > the URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failedCaused > by: MetaException(message:Could not connect to meta store using any of the > URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failed at > org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) > at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:187) > at > org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:71) > at > org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:74) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:68) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:65) > at > org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:43) > at > org.apache.flink.table.filesystem.PartitionLoader.(PartitionLoader.java:61) > at > org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:97) > at > org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20714) Hive delegation token is not obtained when using `kinit` to submit Yarn per-job
[ https://issues.apache.org/jira/browse/FLINK-20714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jackwangcs updated FLINK-20714: --- Description: Hive delegation token is not obtained when using `kinit` to submit Yarn per-job. In YarnClusterDescriptor, it calls org.apache.flink.yarn.Utils#setTokensFor to obtain tokens for the job. But setTokensFor only obtains HDFS and HBase tokens currently, since the Hive integration is supported, the Hive delegation should be obtained also. Otherwise, it will throw the following error when it tries to connect to Hive metastore: {code:java} Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: GSS initiate failedCaused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: GSS initiate failed at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:244) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:187) at org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97) at org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240) at org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:71) at org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35) at org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:74) at org.apache.flink.connectors.hive.HiveTableMetaStoreFactory$HiveTableMetaStore.(HiveTableMetaStoreFactory.java:68) at org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:65) at org.apache.flink.connectors.hive.HiveTableMetaStoreFactory.createTableMetaStore(HiveTableMetaStoreFactory.java:43) at org.apache.flink.table.filesystem.PartitionLoader.(PartitionLoader.java:61) at org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:97) at org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95) {code} was: Hive delegation token is not obtained when using `kinit` to submit Yarn per-job. In YarnClusterDescriptor, it calls org.apache.flink.yarn.Utils#setTokensFor to obtain tokens for the job. But setTokensFor only obtains HDFS and HBase tokens currently, since the Hive integration is supported, the Hive delegation should be obtained. Otherwise, it will throw following error: {code:java} Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: GSS initiate failedCaused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: GSS initiate failed at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) at