[
https://issues.apache.org/jira/browse/FLINK-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14804512#comment-14804512
]
Lydia Ickler commented on FLINK-2506:
-------------------------------------
I did setup the following:
Hadoop: 2.5.2
Flink: 0.9.0-milestone-1
HBase: hbase-0.98.11-hadoop2
And still this error:
22:59:57,184 INFO org.apache.zookeeper.ZooKeeper
- Client environment:java.home=/home/java/jdk1.8.0_05/jre
22:59:57,184 INFO org.apache.zookeeper.ZooKeeper
- Client
environment:java.class.path=/home/SHARES/FLINK/flink-0.9.0-milestone-1/build-target/bin/..$
22:59:57,184 INFO org.apache.zookeeper.ZooKeeper
- Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
22:59:57,184 INFO org.apache.zookeeper.ZooKeeper
- Client environment:java.io.tmpdir=/tmp
22:59:57,184 INFO org.apache.zookeeper.ZooKeeper
- Client environment:java.compiler=<NA>
22:59:57,184 INFO org.apache.zookeeper.ZooKeeper
- Client environment:os.name=Linux
22:59:57,184 INFO org.apache.zookeeper.ZooKeeper
- Client environment:os.arch=amd64
22:59:57,184 INFO org.apache.zookeeper.ZooKeeper
- Client environment:os.version=3.19.0-22-generic
22:59:57,184 INFO org.apache.zookeeper.ZooKeeper
- Client environment:user.name=hduser
22:59:57,184 INFO org.apache.zookeeper.ZooKeeper
- Client environment:user.home=/home/hduser
22:59:57,184 INFO org.apache.zookeeper.ZooKeeper
- Client environment:user.dir=/home/hduser
22:59:57,185 INFO org.apache.zookeeper.ZooKeeper
- Initiating client connection, connectString=grips2:2181 sessionTimeout=90000
watcher=hconnection-$
22:59:57,187 DEBUG org.apache.zookeeper.ClientCnxn
- zookeeper.disableAutoWatchReset is false
22:59:57,202 INFO org.apache.zookeeper.ClientCnxn
- Opening socket connection to server grips2/130.73.20.13:2181. Will not
attempt to authenticate us$
22:59:57,204 INFO org.apache.zookeeper.ClientCnxn
- Socket connection established to grips2/130.73.20.13:2181, initiating session
22:59:57,205 DEBUG org.apache.zookeeper.ClientCnxn
- Session establishment request sent on grips2/130.73.20.13:2181
22:59:57,223 INFO org.apache.zookeeper.ClientCnxn
- Session establishment complete on server grips2/130.73.20.13:2181, sessionid
= 0x14fdced9afa0013,$
22:59:57,226 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher
- hconnection-0x1283a7bd0x0, quorum=grips2:2181, baseZNode=/hbase Received
ZooKeeper Event, type=No$
22:59:57,229 DEBUG org.apache.zookeeper.ClientCnxn
- Reading reply sessionid:0x14fdced9afa0013, packet:: clientPath:null
serverPath:null finished:fals$
22:59:57,233 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher
- hconnection-0x1283a7bd-0x14fdced9afa0013 connected
22:59:57,233 DEBUG org.apache.zookeeper.ClientCnxn
- Reading reply sessionid:0x14fdced9afa0013, packet:: clientPath:null
serverPath:null finished:fals$
22:59:57,368 DEBUG org.apache.hadoop.ipc.RpcClient
- Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@718cf9a, compressor=null,
tcpKeepAlive=true, tc$
22:59:57,384 DEBUG org.apache.flink.runtime.operators.DataSourceTask
- Start registering input and output: DataSource (at
createInput(ExecutionEnvironment.java:487) (o$
22:59:57,392 DEBUG org.apache.flink.runtime.operators.DataSourceTask
- Finished registering input and output: DataSource (at
createInput(ExecutionEnvironment.java:487)$
22:59:57,392 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Register task DataSource (at createInput(ExecutionEnvironment.java:487)
(org.apache.flink.HBaseRe$
22:59:57,394 DEBUG
org.apache.flink.runtime.io.network.partition.ResultPartitionManager -
Registered ResultPartition 74af7622fa970bc1@c4c9370e8edfb1ff [PIPELINED, 1
subpartitions,$
22:59:57,396 INFO org.apache.flink.runtime.taskmanager.Task
- Starting execution of task DataSource (at
createInput(ExecutionEnvironment.java:487) (org.apache.$
22:59:57,399 DEBUG org.apache.flink.runtime.taskmanager.TaskManager
- Handled message SubmitTask(TaskDeploymentDescriptor [job id:
d9c0b0a11bf695767f7ae92e60eec2a4, jo$
22:59:57,405 DEBUG akka.serialization.Serialization(akka://flink)
- Using serializer[akka.serialization.JavaSerializer] for message
[org.apache.flink.runtime.message$
22:59:57,412 DEBUG org.apache.flink.runtime.operators.DataSourceTask
- Starting data source operator: DataSource (at
createInput(ExecutionEnvironment.java:487) (org.ap$
22:59:57,413 DEBUG org.apache.flink.runtime.operators.DataSourceTask
- DataSourceTask object reuse: DISABLED.
22:59:57,425 DEBUG akka.serialization.Serialization(akka://flink)
- Using serializer[akka.serialization.JavaSerializer] for message
[org.apache.flink.runtime.message$
22:59:57,447 DEBUG org.apache.flink.runtime.operators.DataSourceTask
- Opening input split Locatable Split (4) at [grips2:57774]: DataSource (at
createInput(ExecutionE$
22:59:57,447 INFO org.apache.flink.addons.hbase.TableInputFormat
- opening split [4|[grips2:57774]|92492490|b6db6db4]
22:59:57,468 DEBUG org.apache.zookeeper.ClientCnxn
- Reading reply sessionid:0x14fdced9afa0013, packet:: clientPath:null
serverPath:null finished:fals$
22:59:57,479 DEBUG org.apache.zookeeper.ClientCnxn
- Reading reply sessionid:0x14fdced9afa0013, packet:: clientPath:null
serverPath:null finished:fals$
22:59:57,503 DEBUG org.apache.hadoop.hbase.util.ByteStringer
- Failed to classload HBaseZeroCopyByteString: java.lang.IllegalAccessError:
class com.google.proto$
22:59:57,617 DEBUG org.apache.hadoop.ipc.RpcClient
- Use SIMPLE authentication for service ClientService, sasl=false
22:59:57,847 DEBUG org.apache.flink.runtime.taskmanager.TaskManager
- Sending heartbeat to JobManager
22:59:57,848 DEBUG org.apache.flink.runtime.taskmanager.TaskManager
- Received message SendHeartbeat at akka://flink/user/taskmanager from
Actor[akka://flink/deadLette$
22:59:57,849 DEBUG org.apache.flink.runtime.taskmanager.TaskManager
- Handled message SendHeartbeat in 0 ms from Actor[akka://flink/deadLetters].
22:59:57,859 DEBUG org.apache.hadoop.ipc.RpcClient
- Connecting to grips2/130.73.20.13:57774
22:59:57,873 DEBUG org.apache.hadoop.ipc.RpcClient
- IPC Client (886473564) connection to grips2/130.73.20.13:57774 from hduser:
starting, connections$
22:59:57,883 DEBUG org.apache.hadoop.ipc.RpcClient
- IPC Client (886473564) connection to grips2/130.73.20.13:57774 from hduser:
wrote request header $
22:59:57,883 DEBUG org.apache.hadoop.ipc.RpcClient
- IPC Client (886473564) connection to grips2/130.73.20.13:57774 from hduser:
got response header c$
22:59:57,902 DEBUG org.apache.zookeeper.ClientCnxn
- Reading reply sessionid:0x14fdced9afa0013, packet:: clientPath:null
serverPath:null finished:fals$
22:59:57,912 DEBUG org.apache.hadoop.ipc.RpcClient
- IPC Client (886473564) connection to grips2/130.73.20.13:57774 from hduser:
wrote request header $
22:59:57,914 DEBUG org.apache.hadoop.ipc.RpcClient
- IPC Client (886473564) connection to grips2/130.73.20.13:57774 from hduser:
got response header c$
22:59:57,919 DEBUG org.apache.hadoop.hbase.client.ClientSmallScanner
- Finished with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
STARTKEY => '', ENDK$
22:59:57,921 DEBUG org.apache.hadoop.ipc.RpcClient
- IPC Client (886473564) connection to grips2/130.73.20.13:57774 from hduser:
wrote request header $
22:59:57,922 DEBUG org.apache.hadoop.ipc.RpcClient
- IPC Client (886473564) connection to grips2/130.73.20.13:57774 from hduser:
got response header c$
22:59:57,923 DEBUG org.apache.flink.runtime.operators.DataSourceTask
- Starting to read input from split Locatable Split (4) at [grips2:57774]:
DataSource (at createIn$
22:59:57,931 DEBUG org.apache.hadoop.ipc.RpcClient
- IPC Client (886473564) connection to grips2/130.73.20.13:57774 from hduser:
wrote request header $
22:59:57,947 DEBUG org.apache.hadoop.ipc.RpcClient
- IPC Client (886473564) connection to grips2/130.73.20.13:57774 from hduser:
got response header c$
22:59:57,948 DEBUG org.apache.hadoop.ipc.RpcClient
- IPC Client (886473564) connection to grips2/130.73.20.13:57774 from hduser:
wrote request header $
22:59:57,948 DEBUG org.apache.hadoop.ipc.RpcClient
- IPC Client (886473564) connection to grips2/130.73.20.13:57774 from hduser:
got response header c$
22:59:57,949 DEBUG org.apache.flink.runtime.operators.DataSourceTask
- Closing input split Locatable Split (4) at [grips2:57774]: DataSource (at
createInput(ExecutionE$
22:59:57,949 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation -
Closing zookeeper sessionid=0x14fdced9afa0013
22:59:57,949 DEBUG org.apache.zookeeper.ZooKeeper
- Closing session: 0x14fdced9afa0013
22:59:57,949 DEBUG org.apache.zookeeper.ClientCnxn
- Closing client for session: 0x14fdced9afa0013
22:59:57,971 DEBUG org.apache.zookeeper.ClientCnxn
- Reading reply sessionid:0x14fdced9afa0013, packet:: clientPath:null
serverPath:null finished:fals$
22:59:57,971 DEBUG org.apache.zookeeper.ClientCnxn
- Disconnecting client for session: 0x14fdced9afa0013
22:59:57,971 INFO org.apache.zookeeper.ZooKeeper
- Session: 0x14fdced9afa0013 closed
22:59:57,971 DEBUG org.apache.hadoop.ipc.RpcClient
- Stopping rpc client
22:59:57,971 DEBUG org.apache.zookeeper.ClientCnxn
- An exception was thrown while closing send thread for session
0x14fdced9afa0013 : Unable to read $
22:59:57,971 INFO org.apache.zookeeper.ClientCnxn
- EventThread shut down
22:59:57,972 DEBUG org.apache.hadoop.ipc.RpcClient
- IPC Client (886473564) connection to grips2/130.73.20.13:57774 from hduser:
closed
22:59:57,972 DEBUG org.apache.hadoop.ipc.RpcClient
- IPC Client (886473564) connection to grips2/130.73.20.13:57774 from hduser:
stopped, connections 0
22:59:58,072 INFO org.apache.flink.addons.hbase.TableInputFormat
- Closing split (scanned 0 rows)
22:59:58,081 DEBUG org.apache.flink.runtime.operators.DataSourceTask
- Opening input split Locatable Split (0) at [grips2:57774]: DataSource (at
createInput(ExecutionE$
22:59:58,081 INFO org.apache.flink.addons.hbase.TableInputFormat
- opening split [0|[grips2:57774]|-|24924924]
23:00:02,619 DEBUG org.apache.flink.runtime.taskmanager.TaskManager
- Sending heartbeat to JobManager
23:00:02,619 DEBUG org.apache.flink.runtime.taskmanager.TaskManager
- Received message SendHeartbeat at akka://flink/user/taskmanager from
Actor[akka://flink/deadLette$
Right now I am using 4 machines for the HDFS, 1 machine for Flink and 1 for
HBase.
Do you have any suggestions?
> HBase connection closing down (table distributed over more than 1 region
> server - Flink Cluster-Mode)
> -----------------------------------------------------------------------------------------------------
>
> Key: FLINK-2506
> URL: https://issues.apache.org/jira/browse/FLINK-2506
> Project: Flink
> Issue Type: Bug
> Components: Examples
> Affects Versions: 0.10
> Reporter: Lydia Ickler
> Assignee: Aljoscha Krettek
> Attachments: hbase-example.tar.gz
>
>
> If I fill a default table (create 'test-table', 'someCf') with the
> HBaseWriteExample.java program from the HBase addon library then a table
> without start/end key is created.
> The data reading works great with the HBaseReadExample.java.
> Nevertheless, if I manually create a "test-table" that is distributed over
> more than one region server (create 'test-table2', 'someCf',{NUMREGIONS =>
> 3,SPLITALGO => 'HexStringSplit'}) the run is canceled with the following
> error message:
> {noformat}
> grips2
> Error: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=35, exceptions:
> Fri Aug 07 11:18:29 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:18:38 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:18:48 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:18:58 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:19:08 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:19:18 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:19:28 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:19:38 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:19:48 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:19:58 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:20:18 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:20:38 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:20:58 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:21:19 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:21:39 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:21:59 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:22:19 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:22:39 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:22:59 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:23:19 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:23:39 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:24:00 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:24:20 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:24:40 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:25:00 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:25:20 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:25:40 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:26:00 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:26:20 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:26:40 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:27:00 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:27:20 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:27:40 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:28:01 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> Fri Aug 07 11:28:21 CEST 2015,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68,
> java.io.IOException: hconnection-0x47bf79d7 closed
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:131)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:91)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:284)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:189)
> at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:184)
> at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:110)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:775)
> at
> org.apache.flink.addons.hbase.TableInputFormat.open(TableInputFormat.java:153)
> at
> org.apache.flink.addons.hbase.TableInputFormat.open(TableInputFormat.java:48)
> at
> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:151)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: hconnection-0x47bf79d7 closed
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(HConnectionManager.java:1812)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.isTableOnlineState(ZooKeeperRegistry.java:100)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isTableDisabled(HConnectionManager.java:980)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:1133)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:958)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:72)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:125)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
> ... 11 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
