Re: May I run hbase on top of Alluxio/tacyon

[kevin]

I have worked out this question :
https://alluxio.atlassian.net/browse/ALLUXIO-2025

2016-06-20 21:02 GMT+08:00 Jean-Marc Spaggiari :

> I think you might want to clean everything and retry. Clean the ZK /hbase
> content as well as your fs /hbase folder and restart...
>
> 2016-06-20 3:22 GMT-04:00 kevin :
>
> > *I got some error:*
> >
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:java.library.path=/home/dcos/hadoop-2.7.1/lib/native
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:java.io.tmpdir=/tmp
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:java.compiler=
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:os.name=Linux
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:os.arch=amd64
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:os.version=3.10.0-327.el7.x86_64
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:user.name=root
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:user.home=/root
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:user.dir=/home/dcos/hbase-0.98.16.1-hadoop2
> > 2016-06-20 14:50:45,454 INFO  [main] zookeeper.ZooKeeper: Initiating
> client
> > connection, connectString=slave1:2181,master:2181,slave2:2181
> > sessionTimeout=9
> > watcher=master:60x0, quorum=slave1:2181,master:2181,slave2:2181,
> > baseZNode=/hbase
> > 2016-06-20 14:50:45,490 INFO  [main-SendThread(slave2:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server slave2/
> > 10.1.3.177:2181. Will not attempt
> >  to authenticate using SASL (unknown error)
> > 2016-06-20 14:50:45,498 INFO  [main-SendThread(slave2:2181)]
> > zookeeper.ClientCnxn: Socket connection established to slave2/
> > 10.1.3.177:2181, initiating session
> > 2016-06-20 14:50:45,537 INFO  [main-SendThread(slave2:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server slave2/
> > 10.1.3.177:2181, sessionid =
> >  0x3556c8a93960004, negotiated timeout = 4
> > 2016-06-20 14:50:46,040 INFO  [RpcServer.responder] ipc.RpcServer:
> > RpcServer.responder: starting
> > 2016-06-20 14:50:46,043 INFO  [RpcServer.listener,port=6]
> > ipc.RpcServer: RpcServer.listener,port=6: starting
> > 2016-06-20 14:50:46,137 INFO  [master:master:6] mortbay.log: Logging
> to
> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> > org.mortbay.log.Slf4jLog
> > 2016-06-20 14:50:46,177 INFO  [master:master:6] http.HttpServer:
> Added
> > global filter 'safety'
> > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> > 2016-06-20 14:50:46,180 INFO  [master:master:6] http.HttpServer:
> Added
> > filter static_user_filter
> > (class=org.apache.hadoop.http.lib.StaticUserWebFilter$Stat
> > icUserFilter) to context master
> > 2016-06-20 14:50:46,180 INFO  [master:master:6] http.HttpServer:
> Added
> > filter static_user_filter
> > (class=org.apache.hadoop.http.lib.StaticUserWebFilter$Stat
> > icUserFilter) to context static
> > 2016-06-20 14:50:46,189 INFO  [master:master:6] http.HttpServer:
> Jetty
> > bound to port 60010
> > 2016-06-20 14:50:46,189 INFO  [master:master:6] mortbay.log:
> > jetty-6.1.26
> > 2016-06-20 14:50:46,652 INFO  [master:master:6] mortbay.log: Started
> > HttpServer$SelectChannelConnectorWithSafeStartup@0.0.0.0:60010
> > 2016-06-20 14:50:47,122 INFO  [master:master:6]
> > master.ActiveMasterManager: Registered Active
> > Master=master,6,1466405444533
> > 2016-06-20 14:50:47,123 DEBUG [main-EventThread]
> > master.ActiveMasterManager: A master is now available
> > 2016-06-20 14:50:47,127 INFO  [master:master:6] logger.type:
> > getWorkingDirectory: /
> > 2016-06-20 14:50:47,128 INFO  [master:master:6]
> > Configuration.deprecation: fs.default.name is deprecated. Instead, use
> > fs.defaultFS
> > 2016-06-20 14:50:47,131 INFO  [master:master:6] logger.type:
> > getFileStatus(alluxio://master:19998/hbase)
> > 2016-06-20 14:50:47,153 INFO  [master:master:6] logger.type: Alluxio
> > client (version 1.1.1-SNAPSHOT) is trying to connect with
> > FileSystemMasterClient maste
> > r @ master/10.1.3.181:19998
> > 2016-06-20 14:50:47,159 INFO  [master:master:6] logger.type: Client
> > registered with FileSystemMasterClient master @ master/10.1.3.181:19998
> > 2016-06-20 14:50:47,209 INFO  [master:master:6] logger.type:
> > mkdirs(alluxio://master:19998/hbase, rwxrwxrwx)
> > 2016-06-20 14:50:47,227 INFO  [master:master:6] logger.type:
> > create(alluxio://master:19998/hbase/.tmp/hbase.version, rw-r--r--, true,
> > 131072, 1, 536870912,
> >  null)
> > 2016-06-20 14:50:47,262 INFO  [master:master:6] logger.type: Alluxio
> > client (version 1.1.1-SNAPSHOT) is trying to connect 

Re: maybe waste on blockCache

[Enis Söztutar]

LRUBlock cache does not reserve the space for the in_memory tier. The space
is used for other tiers as well as long as there is space.

You can read the code at LRUBlockCache.evict() to learn more.

Enis

On Thu, Jun 16, 2016 at 1:21 AM, WangYQ  wrote:

> in hbase 0.98.10, if we use LruBlockCache, and set regionServer's max heap
> to 10G
> in default:
> the size of in_memory priority of LruBlockCache is :
> 10G * 0.4 * 0.25 = 1G
>
>
> 0.4: hfile.block.cache.size
> 0.25: hbase.lru.blockcache.memory.percentage
>
>
> if we do not set any user tables IN_MEMORY to true, then the whole hbase
> just need to cache hbase:meta data to in_memory LruBlockCache.
> hbase:meta does not split , so just need one regionServer to cache, so
> there is some waste in blockCache
>
>
> i think the regionServer open hbase:meta need to set  in_memory
> LruBlockCache to a certain size
> other regionServer set hbase.lru.blockcache.memory.percentage to 0, do not
> need to allocate  in_memory LruBlockCache.

Re: May I run hbase on top of Alluxio/tacyon

[Jean-Marc Spaggiari]

I think you might want to clean everything and retry. Clean the ZK /hbase
content as well as your fs /hbase folder and restart...

2016-06-20 3:22 GMT-04:00 kevin :

> *I got some error:*
>
> 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> environment:java.library.path=/home/dcos/hadoop-2.7.1/lib/native
> 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/tmp
> 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> environment:java.compiler=
> 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> environment:os.name=Linux
> 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> environment:os.version=3.10.0-327.el7.x86_64
> 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> environment:user.name=root
> 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> environment:user.home=/root
> 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> environment:user.dir=/home/dcos/hbase-0.98.16.1-hadoop2
> 2016-06-20 14:50:45,454 INFO  [main] zookeeper.ZooKeeper: Initiating client
> connection, connectString=slave1:2181,master:2181,slave2:2181
> sessionTimeout=9
> watcher=master:60x0, quorum=slave1:2181,master:2181,slave2:2181,
> baseZNode=/hbase
> 2016-06-20 14:50:45,490 INFO  [main-SendThread(slave2:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server slave2/
> 10.1.3.177:2181. Will not attempt
>  to authenticate using SASL (unknown error)
> 2016-06-20 14:50:45,498 INFO  [main-SendThread(slave2:2181)]
> zookeeper.ClientCnxn: Socket connection established to slave2/
> 10.1.3.177:2181, initiating session
> 2016-06-20 14:50:45,537 INFO  [main-SendThread(slave2:2181)]
> zookeeper.ClientCnxn: Session establishment complete on server slave2/
> 10.1.3.177:2181, sessionid =
>  0x3556c8a93960004, negotiated timeout = 4
> 2016-06-20 14:50:46,040 INFO  [RpcServer.responder] ipc.RpcServer:
> RpcServer.responder: starting
> 2016-06-20 14:50:46,043 INFO  [RpcServer.listener,port=6]
> ipc.RpcServer: RpcServer.listener,port=6: starting
> 2016-06-20 14:50:46,137 INFO  [master:master:6] mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2016-06-20 14:50:46,177 INFO  [master:master:6] http.HttpServer: Added
> global filter 'safety'
> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2016-06-20 14:50:46,180 INFO  [master:master:6] http.HttpServer: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$Stat
> icUserFilter) to context master
> 2016-06-20 14:50:46,180 INFO  [master:master:6] http.HttpServer: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$Stat
> icUserFilter) to context static
> 2016-06-20 14:50:46,189 INFO  [master:master:6] http.HttpServer: Jetty
> bound to port 60010
> 2016-06-20 14:50:46,189 INFO  [master:master:6] mortbay.log:
> jetty-6.1.26
> 2016-06-20 14:50:46,652 INFO  [master:master:6] mortbay.log: Started
> HttpServer$SelectChannelConnectorWithSafeStartup@0.0.0.0:60010
> 2016-06-20 14:50:47,122 INFO  [master:master:6]
> master.ActiveMasterManager: Registered Active
> Master=master,6,1466405444533
> 2016-06-20 14:50:47,123 DEBUG [main-EventThread]
> master.ActiveMasterManager: A master is now available
> 2016-06-20 14:50:47,127 INFO  [master:master:6] logger.type:
> getWorkingDirectory: /
> 2016-06-20 14:50:47,128 INFO  [master:master:6]
> Configuration.deprecation: fs.default.name is deprecated. Instead, use
> fs.defaultFS
> 2016-06-20 14:50:47,131 INFO  [master:master:6] logger.type:
> getFileStatus(alluxio://master:19998/hbase)
> 2016-06-20 14:50:47,153 INFO  [master:master:6] logger.type: Alluxio
> client (version 1.1.1-SNAPSHOT) is trying to connect with
> FileSystemMasterClient maste
> r @ master/10.1.3.181:19998
> 2016-06-20 14:50:47,159 INFO  [master:master:6] logger.type: Client
> registered with FileSystemMasterClient master @ master/10.1.3.181:19998
> 2016-06-20 14:50:47,209 INFO  [master:master:6] logger.type:
> mkdirs(alluxio://master:19998/hbase, rwxrwxrwx)
> 2016-06-20 14:50:47,227 INFO  [master:master:6] logger.type:
> create(alluxio://master:19998/hbase/.tmp/hbase.version, rw-r--r--, true,
> 131072, 1, 536870912,
>  null)
> 2016-06-20 14:50:47,262 INFO  [master:master:6] logger.type: Alluxio
> client (version 1.1.1-SNAPSHOT) is trying to connect with BlockMasterClient
> master @ m
> aster/10.1.3.181:19998
> 2016-06-20 14:50:47,263 INFO  [master:master:6] logger.type: Client
> registered with BlockMasterClient master @ master/10.1.3.181:19998
> 2016-06-20 14:50:47,369 INFO  [master:master:6] logger.type: Connecting
> to remote worker @ slave1/10.1.3.176:29998
> 2016-06-20 14:50:47,411 

Re: May I run hbase on top of Alluxio/tacyon

[kevin]

*I got some error:*

2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
environment:java.library.path=/home/dcos/hadoop-2.7.1/lib/native
2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
environment:java.compiler=
2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
environment:os.name=Linux
2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
environment:os.version=3.10.0-327.el7.x86_64
2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
environment:user.name=root
2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
environment:user.home=/root
2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
environment:user.dir=/home/dcos/hbase-0.98.16.1-hadoop2
2016-06-20 14:50:45,454 INFO  [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=slave1:2181,master:2181,slave2:2181
sessionTimeout=9
watcher=master:60x0, quorum=slave1:2181,master:2181,slave2:2181,
baseZNode=/hbase
2016-06-20 14:50:45,490 INFO  [main-SendThread(slave2:2181)]
zookeeper.ClientCnxn: Opening socket connection to server slave2/
10.1.3.177:2181. Will not attempt
 to authenticate using SASL (unknown error)
2016-06-20 14:50:45,498 INFO  [main-SendThread(slave2:2181)]
zookeeper.ClientCnxn: Socket connection established to slave2/
10.1.3.177:2181, initiating session
2016-06-20 14:50:45,537 INFO  [main-SendThread(slave2:2181)]
zookeeper.ClientCnxn: Session establishment complete on server slave2/
10.1.3.177:2181, sessionid =
 0x3556c8a93960004, negotiated timeout = 4
2016-06-20 14:50:46,040 INFO  [RpcServer.responder] ipc.RpcServer:
RpcServer.responder: starting
2016-06-20 14:50:46,043 INFO  [RpcServer.listener,port=6]
ipc.RpcServer: RpcServer.listener,port=6: starting
2016-06-20 14:50:46,137 INFO  [master:master:6] mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2016-06-20 14:50:46,177 INFO  [master:master:6] http.HttpServer: Added
global filter 'safety'
(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2016-06-20 14:50:46,180 INFO  [master:master:6] http.HttpServer: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$Stat
icUserFilter) to context master
2016-06-20 14:50:46,180 INFO  [master:master:6] http.HttpServer: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$Stat
icUserFilter) to context static
2016-06-20 14:50:46,189 INFO  [master:master:6] http.HttpServer: Jetty
bound to port 60010
2016-06-20 14:50:46,189 INFO  [master:master:6] mortbay.log:
jetty-6.1.26
2016-06-20 14:50:46,652 INFO  [master:master:6] mortbay.log: Started
HttpServer$SelectChannelConnectorWithSafeStartup@0.0.0.0:60010
2016-06-20 14:50:47,122 INFO  [master:master:6]
master.ActiveMasterManager: Registered Active
Master=master,6,1466405444533
2016-06-20 14:50:47,123 DEBUG [main-EventThread]
master.ActiveMasterManager: A master is now available
2016-06-20 14:50:47,127 INFO  [master:master:6] logger.type:
getWorkingDirectory: /
2016-06-20 14:50:47,128 INFO  [master:master:6]
Configuration.deprecation: fs.default.name is deprecated. Instead, use
fs.defaultFS
2016-06-20 14:50:47,131 INFO  [master:master:6] logger.type:
getFileStatus(alluxio://master:19998/hbase)
2016-06-20 14:50:47,153 INFO  [master:master:6] logger.type: Alluxio
client (version 1.1.1-SNAPSHOT) is trying to connect with
FileSystemMasterClient maste
r @ master/10.1.3.181:19998
2016-06-20 14:50:47,159 INFO  [master:master:6] logger.type: Client
registered with FileSystemMasterClient master @ master/10.1.3.181:19998
2016-06-20 14:50:47,209 INFO  [master:master:6] logger.type:
mkdirs(alluxio://master:19998/hbase, rwxrwxrwx)
2016-06-20 14:50:47,227 INFO  [master:master:6] logger.type:
create(alluxio://master:19998/hbase/.tmp/hbase.version, rw-r--r--, true,
131072, 1, 536870912,
 null)
2016-06-20 14:50:47,262 INFO  [master:master:6] logger.type: Alluxio
client (version 1.1.1-SNAPSHOT) is trying to connect with BlockMasterClient
master @ m
aster/10.1.3.181:19998
2016-06-20 14:50:47,263 INFO  [master:master:6] logger.type: Client
registered with BlockMasterClient master @ master/10.1.3.181:19998
2016-06-20 14:50:47,369 INFO  [master:master:6] logger.type: Connecting
to remote worker @ slave1/10.1.3.176:29998
2016-06-20 14:50:47,411 INFO  [master:master:6] logger.type: Connected
to remote machine slave1/10.1.3.176:2
2016-06-20 14:50:47,589 INFO  [master:master:6] logger.type: status:
SUCCESS from remote machine slave1/10.1.3.176:2 received
2016-06-20 14:50:47,612 INFO  [master:master:6] logger.type:
rename(alluxio://master:19998/hbase/.tmp/hbase.version,

Hbase release notes and class/package changes

[ps40]

Hi Guys,

I'm trying to upgrade the Apache Lily Project from Hbase 0.94 to 0.98.
Apache Lily uses some Hbase classes which have changed or disappeared
between these hbase versions. For example, there used to be an enum called:
org.apache.hadoop.hbase.regionserver.StoreFile.BloomType which became a
standalone enum in 0.95:org.apache.hadoop.hbase.regionserver.BloomType

In some cases the classes have disappeared entirely. For example:
org.apache.hadoop.hbase.ipc.CoprocessorProtocol

Is this stuff documented somewhere. Is there an upgrade guide from 0.94 to
0.95?

The closest analogy I have is the spring 3 to spring 4 upgrade guide which
went thru such changes in detail.

I have already looked at CHANGES file which contains a list of bug fixes in
the release but not these changes related to classes/packages.

Please let me know eitherways.

Thanks
Prashant





--
View this message in context: 
http://apache-hbase.679495.n3.nabble.com/Hbase-release-notes-and-class-package-changes-tp4080743.html
Sent from the HBase User mailing list archive at Nabble.com.