[
https://issues.apache.org/jira/browse/HDDS-8385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17709054#comment-17709054
]
Prashant Pogde commented on HDDS-8385:
--------------------------------------
Adding some more context here
I'm using FreeIPA for my IM solution. I've configured the 'service' accounts
via IPA, not local accounts. So the 'hdfs' user running the Ozone services
has a UID of 1301600003. When OM is HA Ozone tries to 'tar' a snapshot that it
can distribute to (i think), bootstrap/sync another OM the process fails.
Speaking with [~weichiu] over slack, he mentioned that we use Apache
Commons-Compression Lib for 'tar' actions. The Ozone code doesn't provide a
means to setBigNumberMode(int bigNumberMode) in TarArchiveOutputStream when we
create the class in ReconUtils:
[[https://github.com/apache/ozone/blob/cb90fda2139f8cd2855a9941d4a723000edbbaf5/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconUtils.java#L111]]
Stack Trace:
{\{2023-03-31 13:57:17,018 WARN org.apache.ratis.grpc.server.GrpcLogAppender:
om2@group-84ACC46C8150->om1-InstallSnapshotResponseHandler: Failed
InstallSnapshot}}
{\{...skipping...}}
{\{java.lang.IllegalArgumentException: group id '1301600003' is too big ( >
2097151 ). Use STAR or POSIX extensions to overcome this limit}}
{\{ at
org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.failForBigNumber(TarArchiveOutputStream.java:651)}}
{\{ at
org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.failForBigNumberWithPosixMessage(TarArchiveOutputStream.java:644)}}
{\{ at
org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.failForBigNumbers(TarArchiveOutputStream.java:626)}}
{\{ at
org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.putArchiveEntry(TarArchiveOutputStream.java:377)}}
{\{ at
org.apache.hadoop.hdds.utils.DBCheckpointServlet.includeFile(DBCheckpointServlet.java:247)}}
{\{ at
org.apache.hadoop.hdds.utils.DBCheckpointServlet.writeDBCheckpointToStream(DBCheckpointServlet.java:228)}}
{\{ at
org.apache.hadoop.hdds.utils.DBCheckpointServlet.doGet(DBCheckpointServlet.java:178)}}
{\{ at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)}}
{\{ at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)}}
{\{ at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)}}
{\{ at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)}}
{\{ at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:110)}}
{\{ at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)}}
{\{ at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)}}
{\{ at
org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1678)}}
{\{ at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)}}
{\{ at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)}}
{\{ at
org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1678)}}
{\{ at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)}}
{\{ at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)}}
{\{ at
org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)}}
{\{ at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)}}
{\{ at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)}}
{\{ at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)}}
{\{ at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)}}
{\{ at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)}}
{\{ at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)}}
{\{ at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)}}
{\{ at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)}}
{\{ at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)}}
{\{ at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)}}
{\{ at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)}}
{\{ at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)}}
{\{ at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)}}
{\{ at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)}}
{\{ at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)}}
{\{ at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)}}
{\{ at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)}}
{\{ at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)}}
{\{ at org.eclipse.jetty.server.Server.handle(Server.java:516)}}
{\{ at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)}}
{\{ at
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)}}
{\{ at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)}}
{\{ at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)}}
{\{ at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)}}
{\{ at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)}}
{\{ at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)}}
{\{ at
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)}}
{\{ at
org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)}}
{\{ at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)}}
{\{ at
org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)}}
{\{ at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)}}
{\{ at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)}}
{\{ at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)}}
{\{ at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)}}
{\{ at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)}}
{\{ at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)}}
{\{ at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)}}
{\{ at java.lang.Thread.run(Thread.java:748)}}
{\{2023-03-31 14:00:27,647 INFO
org.apache.hadoop.hdds.utils.db.RocksDBCheckpoint: Cleaning up RocksDB
checkpoint at /hadoop/ozone/om/data/db.checkpoints/om.db_checkpoint_16}}
{\{80285627566}}
> Ozone can't process snapshot when service UID > 2097151
> -------------------------------------------------------
>
> Key: HDDS-8385
> URL: https://issues.apache.org/jira/browse/HDDS-8385
> Project: Apache Ozone
> Issue Type: Bug
> Components: Snapshot
> Reporter: Prashant Pogde
> Assignee: George Jahad
> Priority: Major
> Fix For: 1.4.0
>
>
> With Ozone OM HA configured, the process that tar's up a snapshot fails.
> {{{}2023-03-31 12:51:15,272 ERROR
> org.apache.hadoop.hdds.utils.DBCheckpointServlet: Unable to process metadata
> snapshot request.{}}}{{{}java.lang.IllegalArgumentException: group id
> '1301600003' is too big ( > 2097151 ). Use STAR or POSIX extensions to
> overcome this limit{}}}
>
> {{The only solution at this time was to rework my service account UID's to a
> value below 2097151.}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
