[jira] [Commented] (HBASE-26007) java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI
[ https://issues.apache.org/jira/browse/HBASE-26007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774813#comment-17774813 ] Zilong Zhu commented on HBASE-26007: I think this issue is very similar to HDFS-16644. > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > --- > > Key: HBASE-26007 > URL: https://issues.apache.org/jira/browse/HBASE-26007 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.3.5 >Reporter: Venkat A >Priority: Major > > Hi All, > We have Hadoop 3.2.2 and HBase 2.3.5 Versions installed. (java version > "1.8.0_291") > > While bringing up HBase master, I'm seeing following error messages in HBase > master log. > > Other HDFS. clients like Spark,MapReduce,Solr etc are able to write HDFS but > HBase is unable to write its meta files in HDFS with following exceptions. > > > Summary of Error logs from hbase master > 2021-06-15 03:57:45,968 INFO [Thread-7] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:45,939 WARN [Thread-7] hdfs.DataStreamer: Abandoning > BP-1583998547-10.10.10.3-1622148262434:blk_1073743393_2570 > 2021-06-15 03:57:45,946 WARN [Thread-7] hdfs.DataStreamer: Excluding > datanode > DatanodeInfoWithStorage[10.10.10.3:50010,DS-281c3377-2bc1-47ea-8302-43108ee69430,DISK] > 2021-06-15 03:57:45,994 WARN [Thread-7] hdfs.DataStreamer: DataStreamer > Exception > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /hbase/data/data/hbase/meta/.tmp/.tableinfo.01 could only be written > to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3 > node(s) are excluded in this operation. > 2021-06-15 03:57:46,023 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > 2021-06-15 03:57:46,035 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:46,508 ERROR [main] regionserver.HRegionServer: Failed > construction RegionServer > java.io.IOException: Failed update hbase:meta table descriptor > 2021-06-15 03:57:46,509 ERROR [main] master.HMasterCommandLine: Master > exiting > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster. > Caused by: java.io.IOException: Failed update hbase:meta table descriptor > > Not sure what is the root cause behind this. Any comments/suggestions on this > is much appreciated. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26007) java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI
[ https://issues.apache.org/jira/browse/HBASE-26007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17479949#comment-17479949 ] HERINIAINA DESLANDES commented on HBASE-26007: -- After compiling, I still had the issue. The use of jdk 11 solved it. > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > --- > > Key: HBASE-26007 > URL: https://issues.apache.org/jira/browse/HBASE-26007 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.3.5 >Reporter: Venkat A >Priority: Major > > Hi All, > We have Hadoop 3.2.2 and HBase 2.3.5 Versions installed. (java version > "1.8.0_291") > > While bringing up HBase master, I'm seeing following error messages in HBase > master log. > > Other HDFS. clients like Spark,MapReduce,Solr etc are able to write HDFS but > HBase is unable to write its meta files in HDFS with following exceptions. > > > Summary of Error logs from hbase master > 2021-06-15 03:57:45,968 INFO [Thread-7] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:45,939 WARN [Thread-7] hdfs.DataStreamer: Abandoning > BP-1583998547-10.10.10.3-1622148262434:blk_1073743393_2570 > 2021-06-15 03:57:45,946 WARN [Thread-7] hdfs.DataStreamer: Excluding > datanode > DatanodeInfoWithStorage[10.10.10.3:50010,DS-281c3377-2bc1-47ea-8302-43108ee69430,DISK] > 2021-06-15 03:57:45,994 WARN [Thread-7] hdfs.DataStreamer: DataStreamer > Exception > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /hbase/data/data/hbase/meta/.tmp/.tableinfo.01 could only be written > to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3 > node(s) are excluded in this operation. > 2021-06-15 03:57:46,023 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > 2021-06-15 03:57:46,035 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:46,508 ERROR [main] regionserver.HRegionServer: Failed > construction RegionServer > java.io.IOException: Failed update hbase:meta table descriptor > 2021-06-15 03:57:46,509 ERROR [main] master.HMasterCommandLine: Master > exiting > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster. > Caused by: java.io.IOException: Failed update hbase:meta table descriptor > > Not sure what is the root cause behind this. Any comments/suggestions on this > is much appreciated. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26007) java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI
[ https://issues.apache.org/jira/browse/HBASE-26007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17479746#comment-17479746 ] Josh Elser commented on HBASE-26007: Maybe Venkat can report if recompiling HBase against his version of Hadoop will fix the issue for him too. IIRC, we have documented in the HBase book that it's a best practice to compile HBase against the specific version of Hadoop you're using. > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > --- > > Key: HBASE-26007 > URL: https://issues.apache.org/jira/browse/HBASE-26007 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.3.5 >Reporter: Venkat A >Priority: Major > > Hi All, > We have Hadoop 3.2.2 and HBase 2.3.5 Versions installed. (java version > "1.8.0_291") > > While bringing up HBase master, I'm seeing following error messages in HBase > master log. > > Other HDFS. clients like Spark,MapReduce,Solr etc are able to write HDFS but > HBase is unable to write its meta files in HDFS with following exceptions. > > > Summary of Error logs from hbase master > 2021-06-15 03:57:45,968 INFO [Thread-7] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:45,939 WARN [Thread-7] hdfs.DataStreamer: Abandoning > BP-1583998547-10.10.10.3-1622148262434:blk_1073743393_2570 > 2021-06-15 03:57:45,946 WARN [Thread-7] hdfs.DataStreamer: Excluding > datanode > DatanodeInfoWithStorage[10.10.10.3:50010,DS-281c3377-2bc1-47ea-8302-43108ee69430,DISK] > 2021-06-15 03:57:45,994 WARN [Thread-7] hdfs.DataStreamer: DataStreamer > Exception > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /hbase/data/data/hbase/meta/.tmp/.tableinfo.01 could only be written > to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3 > node(s) are excluded in this operation. > 2021-06-15 03:57:46,023 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > 2021-06-15 03:57:46,035 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:46,508 ERROR [main] regionserver.HRegionServer: Failed > construction RegionServer > java.io.IOException: Failed update hbase:meta table descriptor > 2021-06-15 03:57:46,509 ERROR [main] master.HMasterCommandLine: Master > exiting > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster. > Caused by: java.io.IOException: Failed update hbase:meta table descriptor > > Not sure what is the root cause behind this. Any comments/suggestions on this > is much appreciated. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26007) java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI
[ https://issues.apache.org/jira/browse/HBASE-26007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17479665#comment-17479665 ] HERINIAINA DESLANDES commented on HBASE-26007: -- Hello I do not know if it is still relevant. I had the same issue.with habse 2.4.9 and hadoop 3.3.1 To resolve it: * I compile hbase with hadoop 3.3.1 * I move to jdk 11 regards > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > --- > > Key: HBASE-26007 > URL: https://issues.apache.org/jira/browse/HBASE-26007 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.3.5 >Reporter: Venkat A >Priority: Major > > Hi All, > We have Hadoop 3.2.2 and HBase 2.3.5 Versions installed. (java version > "1.8.0_291") > > While bringing up HBase master, I'm seeing following error messages in HBase > master log. > > Other HDFS. clients like Spark,MapReduce,Solr etc are able to write HDFS but > HBase is unable to write its meta files in HDFS with following exceptions. > > > Summary of Error logs from hbase master > 2021-06-15 03:57:45,968 INFO [Thread-7] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:45,939 WARN [Thread-7] hdfs.DataStreamer: Abandoning > BP-1583998547-10.10.10.3-1622148262434:blk_1073743393_2570 > 2021-06-15 03:57:45,946 WARN [Thread-7] hdfs.DataStreamer: Excluding > datanode > DatanodeInfoWithStorage[10.10.10.3:50010,DS-281c3377-2bc1-47ea-8302-43108ee69430,DISK] > 2021-06-15 03:57:45,994 WARN [Thread-7] hdfs.DataStreamer: DataStreamer > Exception > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /hbase/data/data/hbase/meta/.tmp/.tableinfo.01 could only be written > to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3 > node(s) are excluded in this operation. > 2021-06-15 03:57:46,023 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > 2021-06-15 03:57:46,035 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:46,508 ERROR [main] regionserver.HRegionServer: Failed > construction RegionServer > java.io.IOException: Failed update hbase:meta table descriptor > 2021-06-15 03:57:46,509 ERROR [main] master.HMasterCommandLine: Master > exiting > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster. > Caused by: java.io.IOException: Failed update hbase:meta table descriptor > > Not sure what is the root cause behind this. Any comments/suggestions on this > is much appreciated. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26007) java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI
[ https://issues.apache.org/jira/browse/HBASE-26007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369697#comment-17369697 ] Venkat A commented on HBASE-26007: -- Hi [~elserj] and [~weichiu], Did further testing, turning off/on below property triggering reported issue. dfs.encrypt.data.transfer true If above value is set to 'fasel', hbase is able to communicate with hdfs fine but if i turn on as 'true', i see error . ERROR: 2021-06-25 20:23:44,177 INFO [Thread-13] hdfs.DataStreamer: Exception in createBlockOutputStream java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI Please let me know on how to fix this bug/behaviour. > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > --- > > Key: HBASE-26007 > URL: https://issues.apache.org/jira/browse/HBASE-26007 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.3.5 >Reporter: Venkat A >Priority: Major > > Hi All, > We have Hadoop 3.2.2 and HBase 2.3.5 Versions installed. (java version > "1.8.0_291") > > While bringing up HBase master, I'm seeing following error messages in HBase > master log. > > Other HDFS. clients like Spark,MapReduce,Solr etc are able to write HDFS but > HBase is unable to write its meta files in HDFS with following exceptions. > > > Summary of Error logs from hbase master > 2021-06-15 03:57:45,968 INFO [Thread-7] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:45,939 WARN [Thread-7] hdfs.DataStreamer: Abandoning > BP-1583998547-10.10.10.3-1622148262434:blk_1073743393_2570 > 2021-06-15 03:57:45,946 WARN [Thread-7] hdfs.DataStreamer: Excluding > datanode > DatanodeInfoWithStorage[10.10.10.3:50010,DS-281c3377-2bc1-47ea-8302-43108ee69430,DISK] > 2021-06-15 03:57:45,994 WARN [Thread-7] hdfs.DataStreamer: DataStreamer > Exception > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /hbase/data/data/hbase/meta/.tmp/.tableinfo.01 could only be written > to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3 > node(s) are excluded in this operation. > 2021-06-15 03:57:46,023 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > 2021-06-15 03:57:46,035 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:46,508 ERROR [main] regionserver.HRegionServer: Failed > construction RegionServer > java.io.IOException: Failed update hbase:meta table descriptor > 2021-06-15 03:57:46,509 ERROR [main] master.HMasterCommandLine: Master > exiting > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster. > Caused by: java.io.IOException: Failed update hbase:meta table descriptor > > Not sure what is the root cause behind this. Any comments/suggestions on this > is much appreciated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26007) java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI
[ https://issues.apache.org/jira/browse/HBASE-26007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369644#comment-17369644 ] Venkat A commented on HBASE-26007: -- Hi [~elserj] and [~weichiu] Got some clue now on why Hbase is not happy, In hdfs-site.xml, if i remove below properties, HBase works fine but not with these properties on. dfs.encrypt.data.transfer true dfs.encrypt.data.transfer.cipher.suites AES/CTR/NoPadding dfs.encrypt.data.transfer.cipher.key.bitlength 256 Do you all know if there any know bug/issue that prevents HBase (HDFS client) to work property on HDFS (Hadoop 3.2.2) ? > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > --- > > Key: HBASE-26007 > URL: https://issues.apache.org/jira/browse/HBASE-26007 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.3.5 >Reporter: Venkat A >Priority: Major > > Hi All, > We have Hadoop 3.2.2 and HBase 2.3.5 Versions installed. (java version > "1.8.0_291") > > While bringing up HBase master, I'm seeing following error messages in HBase > master log. > > Other HDFS. clients like Spark,MapReduce,Solr etc are able to write HDFS but > HBase is unable to write its meta files in HDFS with following exceptions. > > > Summary of Error logs from hbase master > 2021-06-15 03:57:45,968 INFO [Thread-7] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:45,939 WARN [Thread-7] hdfs.DataStreamer: Abandoning > BP-1583998547-10.10.10.3-1622148262434:blk_1073743393_2570 > 2021-06-15 03:57:45,946 WARN [Thread-7] hdfs.DataStreamer: Excluding > datanode > DatanodeInfoWithStorage[10.10.10.3:50010,DS-281c3377-2bc1-47ea-8302-43108ee69430,DISK] > 2021-06-15 03:57:45,994 WARN [Thread-7] hdfs.DataStreamer: DataStreamer > Exception > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /hbase/data/data/hbase/meta/.tmp/.tableinfo.01 could only be written > to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3 > node(s) are excluded in this operation. > 2021-06-15 03:57:46,023 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > 2021-06-15 03:57:46,035 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:46,508 ERROR [main] regionserver.HRegionServer: Failed > construction RegionServer > java.io.IOException: Failed update hbase:meta table descriptor > 2021-06-15 03:57:46,509 ERROR [main] master.HMasterCommandLine: Master > exiting > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster. > Caused by: java.io.IOException: Failed update hbase:meta table descriptor > > Not sure what is the root cause behind this. Any comments/suggestions on this > is much appreciated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26007) java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI
[ https://issues.apache.org/jira/browse/HBASE-26007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368401#comment-17368401 ] Josh Elser commented on HBASE-26007: Decreased priority as there's no ability to currently debug this farther. If you have more information you can provide, Venkat, perhaps someone can help you look into it. > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > --- > > Key: HBASE-26007 > URL: https://issues.apache.org/jira/browse/HBASE-26007 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.3.5 >Reporter: Venkat A >Priority: Major > > Hi All, > We have Hadoop 3.2.2 and HBase 2.3.5 Versions installed. (java version > "1.8.0_291") > > While bringing up HBase master, I'm seeing following error messages in HBase > master log. > > Other HDFS. clients like Spark,MapReduce,Solr etc are able to write HDFS but > HBase is unable to write its meta files in HDFS with following exceptions. > > > Summary of Error logs from hbase master > 2021-06-15 03:57:45,968 INFO [Thread-7] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:45,939 WARN [Thread-7] hdfs.DataStreamer: Abandoning > BP-1583998547-10.10.10.3-1622148262434:blk_1073743393_2570 > 2021-06-15 03:57:45,946 WARN [Thread-7] hdfs.DataStreamer: Excluding > datanode > DatanodeInfoWithStorage[10.10.10.3:50010,DS-281c3377-2bc1-47ea-8302-43108ee69430,DISK] > 2021-06-15 03:57:45,994 WARN [Thread-7] hdfs.DataStreamer: DataStreamer > Exception > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /hbase/data/data/hbase/meta/.tmp/.tableinfo.01 could only be written > to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3 > node(s) are excluded in this operation. > 2021-06-15 03:57:46,023 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > 2021-06-15 03:57:46,035 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:46,508 ERROR [main] regionserver.HRegionServer: Failed > construction RegionServer > java.io.IOException: Failed update hbase:meta table descriptor > 2021-06-15 03:57:46,509 ERROR [main] master.HMasterCommandLine: Master > exiting > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster. > Caused by: java.io.IOException: Failed update hbase:meta table descriptor > > Not sure what is the root cause behind this. Any comments/suggestions on this > is much appreciated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26007) java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI
[ https://issues.apache.org/jira/browse/HBASE-26007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17363628#comment-17363628 ] Venkat A commented on HBASE-26007: -- Hi [~weichiu] Thank You for your comment. In my hdfs-site.xml, I'm not specifying any of the above two properties. What i see is, It is only issue for HBase to write data onto HDFS, Other DFS Clients like Spark, MapReduce etc able to write data to HDFS fine as normal. Below is the attached log file for full trace info. [^hbase-hbase-master-log.log] > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > --- > > Key: HBASE-26007 > URL: https://issues.apache.org/jira/browse/HBASE-26007 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.3.5 >Reporter: Venkat A >Priority: Blocker > > Hi All, > We have Hadoop 3.2.2 and HBase 2.3.5 Versions installed. (java version > "1.8.0_291") > > While bringing up HBase master, I'm seeing following error messages in HBase > master log. > > Other HDFS. clients like Spark,MapReduce,Solr etc are able to write HDFS but > HBase is unable to write its meta files in HDFS with following exceptions. > > > Summary of Error logs from hbase master > 2021-06-15 03:57:45,968 INFO [Thread-7] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:45,939 WARN [Thread-7] hdfs.DataStreamer: Abandoning > BP-1583998547-10.10.10.3-1622148262434:blk_1073743393_2570 > 2021-06-15 03:57:45,946 WARN [Thread-7] hdfs.DataStreamer: Excluding > datanode > DatanodeInfoWithStorage[10.10.10.3:50010,DS-281c3377-2bc1-47ea-8302-43108ee69430,DISK] > 2021-06-15 03:57:45,994 WARN [Thread-7] hdfs.DataStreamer: DataStreamer > Exception > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /hbase/data/data/hbase/meta/.tmp/.tableinfo.01 could only be written > to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3 > node(s) are excluded in this operation. > 2021-06-15 03:57:46,023 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > 2021-06-15 03:57:46,035 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:46,508 ERROR [main] regionserver.HRegionServer: Failed > construction RegionServer > java.io.IOException: Failed update hbase:meta table descriptor > 2021-06-15 03:57:46,509 ERROR [main] master.HMasterCommandLine: Master > exiting > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster. > Caused by: java.io.IOException: Failed update hbase:meta table descriptor > > Not sure what is the root cause behind this. Any comments/suggestions on this > is much appreciated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26007) java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI
[ https://issues.apache.org/jira/browse/HBASE-26007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17363369#comment-17363369 ] Wei-Chiu Chuang commented on HBASE-26007: - You probably had an invalid configuration value in hdfs-site.xml. Because there's no full stack trace, it's hard to know exactly which config is wrong, but try search "^DDI" in your config file. dfs.encrypt.data.overwrite.downstream.new.qop > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > --- > > Key: HBASE-26007 > URL: https://issues.apache.org/jira/browse/HBASE-26007 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.3.5 >Reporter: Venkat A >Priority: Blocker > > Hi All, > We have Hadoop 3.2.2 and HBase 2.3.5 Versions installed. (java version > "1.8.0_291") > > While bringing up HBase master, I'm seeing following error messages in HBase > master log. > > Other HDFS. clients like Spark,MapReduce,Solr etc are able to write HDFS but > HBase is unable to write its meta files in HDFS with following exceptions. > > > Summary of Error logs from hbase master > 2021-06-15 03:57:45,968 INFO [Thread-7] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:45,939 WARN [Thread-7] hdfs.DataStreamer: Abandoning > BP-1583998547-10.10.10.3-1622148262434:blk_1073743393_2570 > 2021-06-15 03:57:45,946 WARN [Thread-7] hdfs.DataStreamer: Excluding > datanode > DatanodeInfoWithStorage[10.10.10.3:50010,DS-281c3377-2bc1-47ea-8302-43108ee69430,DISK] > 2021-06-15 03:57:45,994 WARN [Thread-7] hdfs.DataStreamer: DataStreamer > Exception > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /hbase/data/data/hbase/meta/.tmp/.tableinfo.01 could only be written > to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3 > node(s) are excluded in this operation. > 2021-06-15 03:57:46,023 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI > 2021-06-15 03:57:46,035 INFO [Thread-9] hdfs.DataStreamer: Exception in > createBlockOutputStream > java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD > 2021-06-15 03:57:46,508 ERROR [main] regionserver.HRegionServer: Failed > construction RegionServer > java.io.IOException: Failed update hbase:meta table descriptor > 2021-06-15 03:57:46,509 ERROR [main] master.HMasterCommandLine: Master > exiting > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster. > Caused by: java.io.IOException: Failed update hbase:meta table descriptor > > Not sure what is the root cause behind this. Any comments/suggestions on this > is much appreciated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
