[
https://issues.apache.org/jira/browse/GORA-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258097#comment-14258097
]
Alparslan Avcı commented on GORA-375:
-------------------------------------
[~talat], I've tested your patch but when running hadoop jobs, I could not see
the job progress of map jobs. You can see the log below:
Normal run:
{code}
14/12/23 17:04:49 INFO mapred.JobClient: map 0% reduce 0%
14/12/23 17:04:53 INFO mapred.MapTask: Starting flush of map output
14/12/23 17:04:53 INFO mapred.MapTask: Finished spill 0
14/12/23 17:04:53 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done.
And is in the process of commiting
14/12/23 17:04:54 INFO mapred.LocalJobRunner:
14/12/23 17:04:54 INFO mapred.LocalJobRunner:
14/12/23 17:04:54 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
14/12/23 17:04:54 INFO mapred.Task: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@30fd4204
14/12/23 17:04:54 INFO mapred.LocalJobRunner:
14/12/23 17:04:54 INFO mapred.Merger: Merging 1 sorted segments
14/12/23 17:04:54 INFO mapred.Merger: Down to the last merge-pass, with 1
segments left of total size: 532529 bytes
14/12/23 17:04:54 INFO mapred.LocalJobRunner:
14/12/23 17:04:54 ERROR store.HBaseStore: KeyClass in gora-hbase-mapping is not
the same as the one in the databean.
14/12/23 17:04:54 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=localhost:2181 sessionTimeout=180000
watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@436daf20
14/12/23 17:04:54 INFO zookeeper.ClientCnxn: Opening socket connection to
server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
14/12/23 17:04:54 INFO zookeeper.RecoverableZooKeeper: The identifier of this
process is 15793@alparslan
14/12/23 17:04:54 INFO zookeeper.ClientCnxn: Socket connection established to
localhost/127.0.0.1:2181, initiating session
14/12/23 17:04:54 INFO zookeeper.ClientCnxn: Session establishment complete on
server localhost/127.0.0.1:2181, sessionid = 0x14a77a9d8b90017, negotiated
timeout = 40000
14/12/23 17:04:54 INFO zookeeper.ZooKeeper: Session: 0x14a77a9d8b90017 closed
14/12/23 17:04:54 INFO zookeeper.ClientCnxn: EventThread shut down
14/12/23 17:04:54 INFO mapreduce.GoraRecordWriter: gora.buffer.write.limit =
10000
14/12/23 17:04:54 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done.
And is in the process of commiting
14/12/23 17:04:55 INFO mapred.JobClient: map 100% reduce 0%
14/12/23 17:04:57 INFO mapred.LocalJobRunner: reduce > reduce
14/12/23 17:04:57 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
14/12/23 17:04:57 WARN mapred.FileOutputCommitter: Output path is null in
cleanup
14/12/23 17:04:58 INFO mapred.JobClient: map 100% reduce 100%
14/12/23 17:04:58 INFO mapred.JobClient: Job complete: job_local_0001
{code}
When your patch applied:
{code}
14/12/23 17:11:23 INFO mapred.JobClient: map 0% reduce 0%
14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Session: 0x14a77b10921001e closed
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: EventThread shut down
14/12/23 17:11:24 INFO zookeeper.RecoverableZooKeeper: Process
identifier=catalogtracker-on-hconnection-0x50344c81 connecting to ZooKeeper
ensemble=localhost:2181
14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=localhost:2181 sessionTimeout=90000
watcher=catalogtracker-on-hconnection-0x50344c81, quorum=localhost:2181,
baseZNode=/hbase
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Opening socket connection to
server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Socket connection established to
localhost/127.0.0.1:2181, initiating session
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Session establishment complete on
server localhost/127.0.0.1:2181, sessionid = 0x14a77b10921001f, negotiated
timeout = 40000
14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Session: 0x14a77b10921001f closed
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: EventThread shut down
14/12/23 17:11:24 INFO zookeeper.RecoverableZooKeeper: Process
identifier=catalogtracker-on-hconnection-0x50344c81 connecting to ZooKeeper
ensemble=localhost:2181
14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=localhost:2181 sessionTimeout=90000
watcher=catalogtracker-on-hconnection-0x50344c81, quorum=localhost:2181,
baseZNode=/hbase
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Opening socket connection to
server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Socket connection established to
localhost/127.0.0.1:2181, initiating session
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Session establishment complete on
server localhost/127.0.0.1:2181, sessionid = 0x14a77b109210020, negotiated
timeout = 40000
14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Session: 0x14a77b109210020 closed
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: EventThread shut down
14/12/23 17:11:24 INFO mapred.MapTask: Processing split:
org.apache.gora.mapreduce.GoraInputSplit@40b86a70
14/12/23 17:11:24 INFO mapreduce.GoraRecordReader: gora.buffer.read.limit =
10000
14/12/23 17:11:24 INFO mapred.MapTask: io.sort.mb = 100
14/12/23 17:11:24 INFO mapred.MapTask: data buffer = 79691776/99614720
14/12/23 17:11:24 INFO mapred.MapTask: record buffer = 262144/327680
14/12/23 17:11:25 INFO mapred.MapTask: Starting flush of map output
14/12/23 17:11:25 INFO mapred.MapTask: Finished spill 0
14/12/23 17:11:25 INFO mapred.Task: Task:attempt_local139853648_0001_m_000000_0
is done. And is in the process of commiting
14/12/23 17:11:25 INFO mapred.LocalJobRunner:
14/12/23 17:11:25 INFO mapred.Task: Task
'attempt_local139853648_0001_m_000000_0' done.
14/12/23 17:11:25 INFO mapred.LocalJobRunner: Finishing task:
attempt_local139853648_0001_m_000000_0
14/12/23 17:11:25 INFO mapred.LocalJobRunner: Map task executor complete.
14/12/23 17:11:25 INFO mapred.Task: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3589085
14/12/23 17:11:25 INFO mapred.LocalJobRunner:
14/12/23 17:11:25 INFO mapred.Merger: Merging 1 sorted segments
14/12/23 17:11:25 INFO mapred.Merger: Down to the last merge-pass, with 1
segments left of total size: 532529 bytes
14/12/23 17:11:25 INFO mapred.LocalJobRunner:
14/12/23 17:11:25 ERROR store.HBaseStore: KeyClass in gora-hbase-mapping is not
the same as the one in the databean.
14/12/23 17:11:25 INFO zookeeper.RecoverableZooKeeper: Process
identifier=catalogtracker-on-hconnection-0x50344c81 connecting to ZooKeeper
ensemble=localhost:2181
14/12/23 17:11:25 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=localhost:2181 sessionTimeout=90000
watcher=catalogtracker-on-hconnection-0x50344c81, quorum=localhost:2181,
baseZNode=/hbase
14/12/23 17:11:25 INFO zookeeper.ClientCnxn: Opening socket connection to
server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
14/12/23 17:11:25 INFO zookeeper.ClientCnxn: Socket connection established to
localhost/127.0.0.1:2181, initiating session
14/12/23 17:11:25 INFO zookeeper.ClientCnxn: Session establishment complete on
server localhost/127.0.0.1:2181, sessionid = 0x14a77b109210021, negotiated
timeout = 40000
14/12/23 17:11:25 INFO zookeeper.ZooKeeper: Session: 0x14a77b109210021 closed
14/12/23 17:11:25 INFO zookeeper.ClientCnxn: EventThread shut down
14/12/23 17:11:25 INFO mapreduce.GoraRecordWriter: gora.buffer.write.limit =
10000
14/12/23 17:11:26 INFO mapred.Task: Task:attempt_local139853648_0001_r_000000_0
is done. And is in the process of commiting
14/12/23 17:11:26 INFO mapred.LocalJobRunner: reduce > reduce
14/12/23 17:11:26 INFO mapred.Task: Task
'attempt_local139853648_0001_r_000000_0' done.
14/12/23 17:11:26 WARN mapred.FileOutputCommitter: Output path is null in
cleanup
14/12/23 17:11:26 INFO mapred.JobClient: map 0% reduce 100%
14/12/23 17:11:26 INFO mapred.JobClient: Job complete: job_local139853648_0001
{code}
Is this normal on hadoop-1.2.1? Or is there a problem caused by us?
> Upgrade HBase to 0.98
> ---------------------
>
> Key: GORA-375
> URL: https://issues.apache.org/jira/browse/GORA-375
> Project: Apache Gora
> Issue Type: Improvement
> Components: gora-hbase
> Affects Versions: 0.5
> Reporter: Ted Yu
> Assignee: Talat UYARER
> Labels: hbase
> Fix For: 0.6
>
> Attachments: GORA-375.patch,
> org.apache.gora.avro.mapreduce.TestDataFileAvroStoreMapReduce.txt
>
>
> HBase 0.98 release is the current stable release.
> Gora should be built based on HBase 0.98.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
