Re: Error loading SHA-1 keys with load bulk

2014-05-06 Thread Guillermo Ortiz 
The error was that when I was emitting the K,V, I was doing SHA about K,
not about the key in the Value.
The Value is a KeyValue and here's where I had to do the SHA1.


2014-05-02 0:42 GMT+02:00 Guillermo Ortiz konstt2...@gmail.com:

 Yes, I do,


 job.setMapperClass(EventMapper.class);
 job.setMapOutputKeyClass(ImmutableBytesWritable.class);
 job.setMapOutputValueClass(KeyValue.class);

 FileOutputFormat.setOutputPath(job, hbasePath);
 HTable table = new HTable(jConf, MEM_TABLE_HBASE);
 HFileOutputFormat.configureIncrementalLoad(job, table);


 The error is happeing in a MRUnit, I don't know if it changes something
 about the behavior, because I had some troubles in the past for the same
 reason about the serialization in Hbase 0.96 and MRUnit.
 . Besides, in the setup of the MRUnit test I load some data in hbase with
 keys in sha1 and it works.

 El jueves, 1 de mayo de 2014, Jean-Daniel Cryans jdcry...@apache.org
 escribió:

 Are you using HFileOutputFormat.configureIncrementalLoad() to set up the
 partitioner and the reducers? That will take care of ordering your keys.

 J-D


 On Thu, May 1, 2014 at 5:38 AM, Guillermo Ortiz konstt2...@gmail.com
 wrote:

  I have been looking at the code in HBase, but, I don't really understand
  what this error happens. Why can I put in HBase those keys?
 
 
  2014-04-30 17:57 GMT+02:00 Guillermo Ortiz
  konstt2...@gmail.comjavascript:_e(%7B%7D,'cvml','konstt2...@gmail.com
  ');
  :
 
   I'm using HBase with MapReduce to load a lot of data, so I have
 decide to
   do it with bulk load.
  
  
   I parse my keys with SHA1, but when I try to load them, I got this
   exception.
  
   java.io.IOException: Added a key not lexically larger than previous
 
 key=\x00(6e9e59f36a7ec2ac54635b2d353e53e677839046\x01l\x00\x00\x01E\xB3\xC9\xC7\x0E,
 
 lastkey=\x00(b313a9f1f57c8a07c81dc3221c6151cf3637506a\x01l\x00\x00\x01E\xAE\x18k\x87\x0E
 at
 
 org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:207)
 at
 
 org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:324)
 at
 
 org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:289)
 at
 
 org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1206)
 at
 
 org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:168)
 at
 
 org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:124)
 at
 
 org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:551)
 at
 
 org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
  
   I work with HBase 0.94.6. I have been loking for if I could define any
  reducer, since, I have defined no one. I have read something about
  KeyValueSortReducer but, I don'tknow if there's something that extends
  TableReducer or I'm lookging for a wrong way.

Error loading SHA-1 keys with load bulk

2014-05-01 Thread Guillermo Ortiz 
I have been looking at the code in HBase, but, I don't really understand
what this error happens. Why can I put in HBase those keys?


2014-04-30 17:57 GMT+02:00 Guillermo Ortiz
konstt2...@gmail.comjavascript:_e(%7B%7D,'cvml','konstt2...@gmail.com');
:

 I'm using HBase with MapReduce to load a lot of data, so I have decide to
 do it with bulk load.


 I parse my keys with SHA1, but when I try to load them, I got this
 exception.

 java.io.IOException: Added a key not lexically larger than previous 
 key=\x00(6e9e59f36a7ec2ac54635b2d353e53e677839046\x01l\x00\x00\x01E\xB3\xC9\xC7\x0E,
  
 lastkey=\x00(b313a9f1f57c8a07c81dc3221c6151cf3637506a\x01l\x00\x00\x01E\xAE\x18k\x87\x0E
   at 
 org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:207)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:324)
   at 
 org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:289)
   at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1206)
   at 
 org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:168)
   at 
 org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:124)
   at 
 org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:551)
   at 
 org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)

 I work with HBase 0.94.6. I have been loking for if I could define any 
 reducer, since, I have defined no one. I have read something about 
 KeyValueSortReducer but, I don'tknow if there's something that extends 
 TableReducer or I'm lookging for a wrong way.

Re: Error loading SHA-1 keys with load bulk

2014-05-01 Thread Jean-Daniel Cryans 
Are you using HFileOutputFormat.configureIncrementalLoad() to set up the
partitioner and the reducers? That will take care of ordering your keys.

J-D


On Thu, May 1, 2014 at 5:38 AM, Guillermo Ortiz konstt2...@gmail.comwrote:

 I have been looking at the code in HBase, but, I don't really understand
 what this error happens. Why can I put in HBase those keys?


 2014-04-30 17:57 GMT+02:00 Guillermo Ortiz
 konstt2...@gmail.comjavascript:_e(%7B%7D,'cvml','konstt2...@gmail.com
 ');
 :

  I'm using HBase with MapReduce to load a lot of data, so I have decide to
  do it with bulk load.
 
 
  I parse my keys with SHA1, but when I try to load them, I got this
  exception.
 
  java.io.IOException: Added a key not lexically larger than previous
 key=\x00(6e9e59f36a7ec2ac54635b2d353e53e677839046\x01l\x00\x00\x01E\xB3\xC9\xC7\x0E,
 lastkey=\x00(b313a9f1f57c8a07c81dc3221c6151cf3637506a\x01l\x00\x00\x01E\xAE\x18k\x87\x0E
at
 org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:207)
at
 org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:324)
at
 org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:289)
at
 org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1206)
at
 org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:168)
at
 org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:124)
at
 org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:551)
at
 org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
 
  I work with HBase 0.94.6. I have been loking for if I could define any
 reducer, since, I have defined no one. I have read something about
 KeyValueSortReducer but, I don'tknow if there's something that extends
 TableReducer or I'm lookging for a wrong way.

Re: Error loading SHA-1 keys with load bulk

2014-05-01 Thread Guillermo Ortiz 
Yes, I do,


job.setMapperClass(EventMapper.class);
job.setMapOutputKeyClass(ImmutableBytesWritable.class);
job.setMapOutputValueClass(KeyValue.class);

FileOutputFormat.setOutputPath(job, hbasePath);
HTable table = new HTable(jConf, MEM_TABLE_HBASE);
HFileOutputFormat.configureIncrementalLoad(job, table);


The error is happeing in a MRUnit, I don't know if it changes something
about the behavior, because I had some troubles in the past for the same
reason about the serialization in Hbase 0.96 and MRUnit.
. Besides, in the setup of the MRUnit test I load some data in hbase with
keys in sha1 and it works.

El jueves, 1 de mayo de 2014, Jean-Daniel Cryans jdcry...@apache.org
escribió:

 Are you using HFileOutputFormat.configureIncrementalLoad() to set up the
 partitioner and the reducers? That will take care of ordering your keys.

 J-D


 On Thu, May 1, 2014 at 5:38 AM, Guillermo Ortiz 
 konstt2...@gmail.comjavascript:;
 wrote:

  I have been looking at the code in HBase, but, I don't really understand
  what this error happens. Why can I put in HBase those keys?
 
 
  2014-04-30 17:57 GMT+02:00 Guillermo Ortiz
  konstt2...@gmail.com javascript:;javascript:_e(%7B%7D,'cvml','
 konstt2...@gmail.com javascript:;
  ');
  :
 
   I'm using HBase with MapReduce to load a lot of data, so I have decide
 to
   do it with bulk load.
  
  
   I parse my keys with SHA1, but when I try to load them, I got this
   exception.
  
   java.io.IOException: Added a key not lexically larger than previous
 
 key=\x00(6e9e59f36a7ec2ac54635b2d353e53e677839046\x01l\x00\x00\x01E\xB3\xC9\xC7\x0E,
 
 lastkey=\x00(b313a9f1f57c8a07c81dc3221c6151cf3637506a\x01l\x00\x00\x01E\xAE\x18k\x87\x0E
 at
 
 org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:207)
 at
 
 org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:324)
 at
 
 org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:289)
 at
 
 org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1206)
 at
 
 org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:168)
 at
 
 org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:124)
 at
 
 org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:551)
 at
 
 org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
  
   I work with HBase 0.94.6. I have been loking for if I could define any
  reducer, since, I have defined no one. I have read something about
  KeyValueSortReducer but, I don'tknow if there's something that extends
  TableReducer or I'm lookging for a wrong way.

Error loading SHA-1 keys with load bulk

2014-04-30 Thread Guillermo Ortiz 
I'm using HBase with MapReduce to load a lot of data, so I have decide to
do it with bulk load.


I parse my keys with SHA1, but when I try to load them, I got this
exception.

java.io.IOException: Added a key not lexically larger than previous
key=\x00(6e9e59f36a7ec2ac54635b2d353e53e677839046\x01l\x00\x00\x01E\xB3\xC9\xC7\x0E,
lastkey=\x00(b313a9f1f57c8a07c81dc3221c6151cf3637506a\x01l\x00\x00\x01E\xAE\x18k\x87\x0E
at 
org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:207)
at 
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:324)
at 
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:289)
at 
org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1206)
at 
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:168)
at 
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:124)
at 
org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:551)
at 
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)

I work with HBase 0.94.6. I have been loking for if I could define any
reducer, since, I have defined no one. I have read something about
KeyValueSortReducer but, I don'tknow if there's something that extends
TableReducer or I'm lookging for a wrong way.