Re: [External] RE: Problem creating tables in Accumulo
Thank you every one for the response. I think the problem is related to slow / intermittent response from Amazon S3 due the network situation we are in. I am trying to use an internal S3 bucket to see if the problem persists. thanks again. Ranga From: dlmar...@comcast.net Sent: Wednesday, February 22, 2023 2:59 PM To: user@accumulo.apache.org ; Samudrala, Ranganath [USA] Subject: [External] RE: Problem creating tables in Accumulo It looks like the issue is that `maxTime` is not being set at [0], so `time` is not being set at [1]. If I had to guess, Accumulo is having an issue talking with ZooKeeper. Were there errors in the log when initializing Accumulo? Are you able to `exec` into one of the Accumulo Kubernetes pods and confirm connectivity with Accumulo? [0] https://github.com/apache/accumulo/blob/1.10/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/TabletData.java#L159<https://urldefense.com/v3/__https://github.com/apache/accumulo/blob/1.10/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/TabletData.java*L159__;Iw!!May37g!LFm9Bahei8b-iJypacQhWPQOTL5v3dvrCk-sRw87xEjf0D4tmTTJ81ySH0JaG_Md_oqrAVfSfFbDa0hTSd435920$> [1] https://github.com/apache/accumulo/blob/1.10/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/TabletData.java#L166<https://urldefense.com/v3/__https://github.com/apache/accumulo/blob/1.10/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/TabletData.java*L166__;Iw!!May37g!LFm9Bahei8b-iJypacQhWPQOTL5v3dvrCk-sRw87xEjf0D4tmTTJ81ySH0JaG_Md_oqrAVfSfFbDa0hTSdE9N0sW$> From: Samudrala, Ranganath [USA] via user Sent: Tuesday, February 21, 2023 1:43 PM To: Samudrala, Ranganath [USA] via user Subject: Problem creating tables in Accumulo Hello, Accumulo is running in Kubernetes cluster. Metdata data is stored in HDFS and table data is stored in Amazon S3. No attempt has been made yet to store/insert table data. But, when I attempt to create a table in Accumulo via Java API, I see the following error. WARN : org.apache.accumulo.core.clientImpl.ThriftTransportPool - Thread "main" stuck on IO to accumulo-masters-0.accumulo-masters.acc umulo-k8s.svc.cluster.local: (0) for at least 120033 ms I see the below error in T-SERVER logs: 2023-02-21 18:04:09,373 WARN [Root tablet assignment retry] [org.apache.accumulo.tserver.TabletServer]: exception trying to assign tablet +r<< hdfs://accumulo-hdfs-namenode-0.accumulo-hdfs-namenodes:8020/accumulo/tables/+r/root_tablet java.lang.NullPointerException at org.apache.accumulo.server.tablets.TabletTime.getInstance(TabletTime.java:60) at org.apache.accumulo.tserver.tablet.Tablet.(Tablet.java:341) at org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer.java:2504) at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35) at java.lang.Thread.run(Thread.java:750) 2023-02-21 18:04:09,374 WARN [Root tablet assignment retry] [org.apache.accumulo.tserver.TabletServer]: failed to open tablet +r<< reporting failure to master 2023-02-21 18:04:09,374 WARN [Root tablet assignment retry] [org.apache.accumulo.tserver.TabletServer]: rescheduling tablet load in 600.00 seconds 2023-02-21 18:07:03,210 WARN [SimpleTimer-0] [org.apache.accumulo.server.ServerUtil]: System swappiness setting is greater than ten (30) which can cause time-sensitive operations to be delayed. Accumulo is time sensitive because it needs to maintain distributed lock agreement. The installed configuration is made up of 2 instances of HDFS data nodes, 2 instances of T-SERVERs, 1 instance each of HDFS name node and Accumulo master. > config -t accumulo.root . default| table.file.replication | 0 site |@override .. | 2 table |@override .. | 5 . . > config -t accumulo.metadata . default| table.file.replication | 0 site |@override .. | 2 table |@override .. | 5 . How do I fix this problem? Thanks in advance. Ranga
RE: Problem creating tables in Accumulo
It looks like the issue is that `maxTime` is not being set at [0], so `time` is not being set at [1]. If I had to guess, Accumulo is having an issue talking with ZooKeeper. Were there errors in the log when initializing Accumulo? Are you able to `exec` into one of the Accumulo Kubernetes pods and confirm connectivity with Accumulo? [0] https://github.com/apache/accumulo/blob/1.10/server/tserver/src/main/java/or g/apache/accumulo/tserver/tablet/TabletData.java#L159 [1] https://github.com/apache/accumulo/blob/1.10/server/tserver/src/main/java/or g/apache/accumulo/tserver/tablet/TabletData.java#L166 From: Samudrala, Ranganath [USA] via user Sent: Tuesday, February 21, 2023 1:43 PM To: Samudrala, Ranganath [USA] via user Subject: Problem creating tables in Accumulo Hello, Accumulo is running in Kubernetes cluster. Metdata data is stored in HDFS and table data is stored in Amazon S3. No attempt has been made yet to store/insert table data. But, when I attempt to create a table in Accumulo via Java API, I see the following error. WARN : org.apache.accumulo.core.clientImpl.ThriftTransportPool - Thread "main" stuck on IO to accumulo-masters-0.accumulo-masters.acc umulo-k8s.svc.cluster.local: (0) for at least 120033 ms I see the below error in T-SERVER logs: 2023-02-21 18:04:09,373 WARN [Root tablet assignment retry] [org.apache.accumulo.tserver.TabletServer]: exception trying to assign tablet +r<< hdfs://accumulo-hdfs-namenode-0.accumulo-hdfs-namenodes:8020/accumulo/tables /+r/root_tablet java.lang.NullPointerException at org.apache.accumulo.server.tablets.TabletTime.getInstance(TabletTime.java:60 ) at org.apache.accumulo.tserver.tablet.Tablet.(Tablet.java:341) at org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer. java:2504) at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35) at java.lang.Thread.run(Thread.java:750) 2023-02-21 18:04:09,374 WARN [Root tablet assignment retry] [org.apache.accumulo.tserver.TabletServer]: failed to open tablet +r<< reporting failure to master 2023-02-21 18:04:09,374 WARN [Root tablet assignment retry] [org.apache.accumulo.tserver.TabletServer]: rescheduling tablet load in 600.00 seconds 2023-02-21 18:07:03,210 WARN [SimpleTimer-0] [org.apache.accumulo.server.ServerUtil]: System swappiness setting is greater than ten (30) which can cause time-sensitive operations to be delayed. Accumulo is time sensitive because it needs to maintain distributed lock agreement. The installed configuration is made up of 2 instances of HDFS data nodes, 2 instances of T-SERVERs, 1 instance each of HDFS name node and Accumulo master. > config -t accumulo.root . default| table.file.replication | 0 site |@override .. | 2 table |@override .. | 5 . . > config -t accumulo.metadata . default| table.file.replication | 0 site |@override .. | 2 table |@override .. | 5 . How do I fix this problem? Thanks in advance. Ranga
Re: Problem creating tables in Accumulo
Ranga, Your current error message seems to point to your root tablet not being able to be created (or assigned). I would start by trying to verify that HDFS is functional and that your `instance.volumes` entries include a valid HDFS volume entry. Does your monitor page report additional errors? - Daniel On Tue, Feb 21, 2023 at 1:43 PM Samudrala, Ranganath [USA] via user < user@accumulo.apache.org> wrote: > Hello, > > Accumulo is running in Kubernetes cluster. Metdata data is stored in HDFS > and table data is stored in Amazon S3. No attempt has been made yet to > store/insert table data. But, when I attempt to create a table in > Accumulo via Java API, I see the following error. > > > WARN : org.apache.accumulo.core.clientImpl.ThriftTransportPool - Thread > "main" stuck on IO to accumulo-masters-0.accumulo-masters.acc > umulo-k8s.svc.cluster.local: (0) for at least 120033 ms > > > > I see the below error in T-SERVER logs: > > 2023-02-21 18:04:09,373 WARN [Root tablet assignment retry] > [org.apache.accumulo.tserver.TabletServer]: exception trying to assign tablet > +r<< > hdfs://accumulo-hdfs-namenode-0.accumulo-hdfs-namenodes:8020/accumulo/tables/+r/root_tablet > java.lang.NullPointerException > at > org.apache.accumulo.server.tablets.TabletTime.getInstance(TabletTime.java:60) > at org.apache.accumulo.tserver.tablet.Tablet.(Tablet.java:341) > at > org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer.java:2504) > at > org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35) > at java.lang.Thread.run(Thread.java:750) > 2023-02-21 18:04:09,374 WARN [Root tablet assignment retry] > [org.apache.accumulo.tserver.TabletServer]: failed to open tablet +r<< > reporting failure to master > 2023-02-21 18:04:09,374 WARN [Root tablet assignment retry] > [org.apache.accumulo.tserver.TabletServer]: rescheduling tablet load in > 600.00 seconds > 2023-02-21 18:07:03,210 WARN [SimpleTimer-0] > [org.apache.accumulo.server.ServerUtil]: System swappiness setting is greater > than ten (30) which can cause time-sensitive operations to be delayed. > Accumulo is time sensitive because it needs to maintain distributed lock > agreement. > > > > The installed configuration is made up of 2 instances of HDFS data nodes, > 2 instances of T-SERVERs, 1 instance each of HDFS name node and Accumulo > master. > > > config -t accumulo.root > . > default| table.file.replication | 0 > site |@override .. | 2 > table |@override .. | 5 > . > . > > config -t accumulo.metadata > . > default| table.file.replication | 0 > site |@override .. | 2 > table |@override .. | 5 > . > > How do I fix this problem? > > Thanks in advance. > Ranga > >