ligou525 opened a new issue, #10762:
URL: https://github.com/apache/hudi/issues/10762
For hudi table using file system with kerberos authentication, the timeline
server handler doesn't work. Handlers of timeline server doesn't consider the
authenticated filesystem, authentication exception is thrown when creating
HoodieTableMetaClient as creating the filesystem:
public SyncableFileSystemView getFileSystemView(String basePath) {
return globalViewMap.computeIfAbsent(basePath, (path) -> {
HoodieTableMetaClient metaClient =
HoodieTableMetaClient.builder().setConf(conf.newCopy()).setBasePath(path).build();
return viewCreator.apply(metaClient, viewStorageConfig);
});
}
**Expected behavior**
timeline server works well for kerberos-authenticated filesystem
**Environment Description**
* Hudi version : 0.11.0
* Spark version :
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : hdfs
* Running on Docker? (yes/no) : no
**Additional context**
**Stacktrace**
24/02/19 21:57:12 INFO view.RemoteHoodieTableFileSystemView: Sending request
:
(http://172.16.48.4:27329/v1/hoodie/view/compactions/pending/?basepath=hdfs%3A%2F%2Fhdfs_host%3A8020%2Fuser%2Fhdfs%2Ftest_location%2Ftest_decimal_hdfs&lastinstantts=20240219215711252&timeline
hash=4a679c704ec969a1ad4b2f434ac012ff10005ba03948c375356913f52e6d8ed4)
24/02/19 21:57:12 INFO table.HoodieTableMetaClient: Loading
HoodieTableMetaClient from
hdfs://172.16.48.6:8020/user/hdfs/test_location/test_decimal_hdfs
24/02/19 21:57:12 ERROR service.RequestHandler: Got runtime exception
servicing request
basepath=hdfs%3A%2F%2F2Fhdfs_host%3A8020%2Fuser%2Fhdfs%2Ftest_location%2Ftest_decimal_hdfs&lastinstantts=20240219215711252&timelinehash=4a679c704ec969a1ad4b2f434ac012ff10005ba03948c375
356913f52e6d8ed4
org.apache.hudi.exception.HoodieIOException: Could not check if
hdfs://2Fhdfs_host:8020/user/hdfs/test_location/test_decimal_hdfs is a valid
table
at
org.apache.hudi.exception.TableNotFoundException.checkTableValidity(TableNotFoundException.java:59)
at
org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:124)
at
org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:78)
at
org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:670)
at
org.apache.hudi.common.table.view.FileSystemViewManager.lambda$getFileSystemView$0(FileSystemViewManager.java:98)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at
org.apache.hudi.common.table.view.FileSystemViewManager.getFileSystemView(FileSystemViewManager.java:97)
at
org.apache.hudi.timeline.service.RequestHandler.isLocalViewBehind(RequestHandler.java:125)
at
org.apache.hudi.timeline.service.RequestHandler.syncIfLocalViewBehind(RequestHandler.java:153)
at
org.apache.hudi.timeline.service.RequestHandler.access$100(RequestHandler.java:66)
at
org.apache.hudi.timeline.service.RequestHandler$ViewHandler.handle(RequestHandler.java:495)
at
io.javalin.security.SecurityUtil.noopAccessManager(SecurityUtil.kt:22)
at io.javalin.Javalin.lambda$addHandler$0(Javalin.java:606)
at
io.javalin.core.JavalinServlet$service$2$1.invoke(JavalinServlet.kt:46)
at
io.javalin.core.JavalinServlet$service$2$1.invoke(JavalinServlet.kt:17)
at
io.javalin.core.JavalinServlet$service$1.invoke(JavalinServlet.kt:143)
at
io.javalin.core.JavalinServlet$service$2.invoke(JavalinServlet.kt:41)
at io.javalin.core.JavalinServlet.service(JavalinServlet.kt:107)
at
io.javalin.core.util.JettyServerUtil$initialize$httpHandler$1.doHandle(JettyServerUtil.kt:72)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at
org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61)
at
org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:502)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: DestHost:destPort 2Fhdfs_host:8020 ,
LocalHost:localPort core-25c178a-2/2Fclient_host:0. Failed on local exception:
java.io.IOException: org.apache.hadoop.security.AccessControlException: Client
cannot authenticate via:[TOKEN, KERBEROS]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:842)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:817)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1583)
at org.apache.hadoop.ipc.Client.call(Client.java:1525)
at org.apache.hadoop.ipc.Client.call(Client.java:1422)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:231)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
