The line 59 in your example is commented, so I guess it's the line a bit
after.
Really, you should try to investigate a bit on your own. Else you will
always face issues that will take you days to get a reply on. And at some
points, people will stop looking at your questions if they figure you are
not making any effort.
So. Let's take a look at this one.
On the stack trace, you can see this:
at org.apache.hadoop.io.Text.set(Text.java:178)
Which mean error is from this:
k1.set(k);
And the exception is: NullPointerException
Since we see on the stack trace that set is calling encode, that mean k1 is
not null. So that mean the only other option is that k is null.
so, if K is null, where is k coming from?
>From here:String k=Bytes.toString(value.getValue(colfamily3,qul3));
Which mean Bytes.toString returned null.
Why? Let's look at Bytes.toString code then...
if (b == null) {
return null;
}
Which mean that if the parameter given to toString is null, Bytes.toString
will return null. and that's the only case where it will return null. Fine.
So that mean that value.getValue(colfamily3,qul3) returned null.
Why?
I will let you work a bit on this one. But few hints.
1) private static byte[] colfamily3=Bytes.toBytes(col_family1);
familiy3 is created from family1. Is it familiy1? Or it's familiy3?
2) Does ALL you keys have this family populated?
3) Have you checked that this is on the resultset you received?
Give that a look, and keep us posted.
JM
2013/8/19 manish dunani <[email protected]>
> I could not find it out!!
>
> Can You just look at it !!
>
> *Pastebin:*http://pastebin.com/pFKUhTgJ#
>
> Tell me where i did mistake?
>
>
> On Mon, Aug 19, 2013 at 7:16 PM, Jean-Marc Spaggiari <
> [email protected]> wrote:
>
> > Hi Manish,
> >
> > I did not even looked at it ;)
> >
> > Wha investigation have you done on your side to try to figure what the
> > issue is?
> >
> > Have you add some logging to see the values before they are used to
> figure
> > what they exactly are?
> > Have you looked at some code to see what it's doing?
> >
> > What is the line 59 on your code?
> >
> > Also, as asked previously, " can you please paste you code on pastbin?
> > Same for the exception."? It's easier to look at it.
> >
> > JM
> >
> >
> > 2013/8/19 manish dunani <[email protected]>
> >
> > > hello,
> > > jean
> > >
> > > Did u find it??
> > >
> > >
> > > On Sun, Aug 18, 2013 at 8:28 AM, manish dunani <[email protected]>
> > > wrote:
> > >
> > > > But i want my output likewise::
> > > >
> > > >
> > > > ROW CELL+COLUMN
> > > >
> > > > QXM column=stocks_output:average,
> > > > timestamp=XXXXXXXXXX, val
> > > > ue=XXXX
> > > > QTM column=stocks_output:average,
> > > > timestamp=XXXXXXXXXX, val
> > > > ue=XXXX
> > > >
> > > >
> > > > *sample dataset in hbase::**(table name:nyse4)*
> > > >
> > > >
> > > > 2010-02-04 column=stocks:open, timestamp=1376567559424,
> > > > value=2.5
> > > > 2010-02-04 column=stocks:symbol, timestamp=1376567559424,
> > > > value=QXM
> > > > 2010-02-05 column=stocks:open, timestamp=1376567559429,
> > > > value=2.42
> > > > 2010-02-05 column=stocks:symbol, timestamp=1376567559429,
> > > > value=QXM
> > > >
> > > >
> > > > ===>>In my previous output i didn't get any symbol(qulifier)'s values
> > in
> > > > my table as Row key..
> > > >
> > > >
> > > >
> > > > hbase(main):004:0> scan 'nyse5'
> > > > ROW COLUMN+CELL
> > > > symbol column=stocks_output:average,
> > > > timestamp=1376749641978, val
> > > > ue=@\xC6o\x11
> > > >
> > > > *So,that i changed my programme like wise::*
> > > >
> > > >
> > > > package com.maddy;
> > > >
> > > > import java.io.IOException;
> > > >
> > > > import org.apache.hadoop.conf.Configuration;
> > > > import org.apache.hadoop.fs.Path;
> > > > import org.apache.hadoop.hbase.HBaseConfiguration;
> > > > import org.apache.hadoop.hbase.client.Result;
> > > > import org.apache.hadoop.hbase.client.Scan;
> > > > import org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter;
> > > > import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
> > > > import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
> > > > import org.apache.hadoop.hbase.mapreduce.TableMapper;
> > > > import org.apache.hadoop.hbase.util.Bytes;
> > > > //import org.apache.hadoop.io.DoubleWritable;
> > > > import org.apache.hadoop.io.FloatWritable;
> > > > import org.apache.hadoop.io.Text;
> > > > import org.apache.hadoop.mapreduce.Job;
> > > > import org.apache.hadoop.mapreduce.Reducer;
> > > >
> > > > import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
> > > >
> > > >
> > > > public class openaveragestock
> > > > {
> > > > public static class map extends TableMapper<Text,FloatWritable>
> > > > {
> > > > private static String col_family="stocks";
> > > > private static String qul="open";
> > > >
> > > > private static String col_family1="stocks";
> > > > private static String qul1="symbol";
> > > >
> > > > private static byte[] colfamily2=Bytes.toBytes(col_family);
> > > > private static byte[] qul2=Bytes.toBytes(qul);
> > > >
> > > > private static byte[] colfamily3=Bytes.toBytes(col_family1);
> > > > private static byte[] qul3=Bytes.toBytes(qul1);
> > > >
> > > > // public static float toFloat(int qul2)
> > > > // {
> > > > // return Float.intBitsToFloat((qul2));
> > > > //
> > > > // }
> > > > //
> > > > private static Text k1=new Text();
> > > >
> > > >
> > > >
> > > > public void map(ImmutableBytesWritable row,Result
> value,Context
> > > > context) throws IOException
> > > > {
> > > >
> > > >
> > > > //byte[]
> > > > val1=(value.getValue("stocks".getBytes(),"symbol".getBytes()));
> > > > //String
> > > >
> > >
> >
> k=Bytes.toString(value.getValue(Bytes.toBytes("stocks"),Bytes.toBytes("symbol")));
> > > > byte[] val=value.getValue(colfamily2,qul2);
> > > > String k=Bytes.toString(value.getValue(colfamily3,qul3));
> > > >
> > > > //ImmutableBytesWritable stock_symbol=new
> > > > ImmutableBytesWritable(qul3);
> > > >
> > > > k1.set(k);
> > > >
> > > > try
> > > > {
> > > >
> > > > context.write(k1,new
> > > > FloatWritable(Float.parseFloat(Bytes.toString(val))));
> > > >
> > > > }
> > > >
> > > > catch(InterruptedException e)
> > > >
> > > > {
> > > > throw new IOException(e);
> > > > }
> > > >
> > > >
> > > > }
> > > >
> > > >
> > > > }
> > > >
> > > >
> > > > public static class Reduce extends Reducer<Text,FloatWritable,
> > > > Text,FloatWritable>
> > > > {
> > > >
> > > > public void reduce(Text
> > key,Iterable<FloatWritable>values,Context
> > > > context) throws IOException, InterruptedException
> > > >
> > > > {
> > > > float sum=0;
> > > > int count=0;
> > > > float average=0;
> > > > for(FloatWritable val:values)
> > > > {
> > > > sum+=val.get();
> > > > count++;
> > > > }
> > > > average=(sum/count);
> > > > // Put put=new Put(key.getBytes());
> > > > //
> > > >
> > >
> >
> put.add(Bytes.toBytes("stocks_output"),Bytes.toBytes("average"),Bytes.toBytes(average));
> > > >
> > > > System.out.println("For\t"+count+"\t average
> is:"+average);
> > > > context.write(key,new FloatWritable(average));
> > > >
> > > >
> > > > }
> > > >
> > > > }
> > > >
> > > > public static void main(String args[]) throws IOException,
> > > > ClassNotFoundException, InterruptedException
> > > > {
> > > > Configuration config=HBaseConfiguration.create();
> > > > config.addResource("/home/manish/workspace/hbase
> > > > project/bin/hbase-site.xml");
> > > > Job job=new Job(config,"openstockaverage1");
> > > >
> > > >
> > > > Scan scan=new Scan();
> > > > scan.addFamily("stocks".getBytes());
> > > > scan.setFilter(new FirstKeyOnlyFilter());
> > > >
> > > > TableMapReduceUtil.initTableMapperJob("nyse4",
> > > > scan,
> > > > map.class,
> > > > ImmutableBytesWritable.class,
> > > > FloatWritable.class,
> > > > job);
> > > >
> > > > // TableMapReduceUtil.initTableReducerJob("nyse5",
> > > > // Reduce.class,
> > > > // job);
> > > > job.setReducerClass(Reduce.class);
> > > >
> > > > FileOutputFormat.setOutputPath(job, new Path(
> > > >
> > > "hdfs://localhost:54310/user/manish/Full_final_output_5"));
> > > > job.waitForCompletion(true);
> > > > }
> > > >
> > > >
> > > >
> > > > }
> > > >
> > > >
> > > > *It throws the error::*
> > > >
> > > > 13/08/17 19:37:59 INFO mapred.JobClient: Running job: job_local_0001
> > > > 13/08/17 19:37:59 INFO util.ProcessTree: setsid exited with exit
> code 0
> > > > 13/08/17 19:37:59 INFO mapred.Task: Using ResourceCalculatorPlugin :
> > > > org.apache.hadoop.util.LinuxResourceCalculatorPlugin@bd96dd
> > > > 13/08/17 19:37:59 INFO mapred.MapTask: io.sort.mb = 100
> > > > 13/08/17 19:38:00 INFO mapred.JobClient: map 0% reduce 0%
> > > > 13/08/17 19:38:00 INFO mapred.MapTask: data buffer =
> 79691776/99614720
> > > > 13/08/17 19:38:00 INFO mapred.MapTask: record buffer = 262144/327680
> > > > 13/08/17 19:38:00 WARN mapred.LocalJobRunner: job_local_0001
> > > > java.lang.NullPointerException
> > > > at org.apache.hadoop.io.Text.encode(Text.java:388)
> > > > at org.apache.hadoop.io.Text.set(Text.java:178)
> > > > at com.maddy.openaveragestock$map.map(openaveragestock.java:59)
> > > >
> > > > at com.maddy.openaveragestock$map.map(openaveragestock.java:1)
> > > > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > > > at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> > > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> > > > at
> > > >
> > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:214)
> > > > 13/08/17 19:38:01 INFO mapred.JobClient: Job complete: job_local_0001
> > > > 13/08/17 19:38:01 INFO mapred.JobClient: Counters: 0
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Sun, Aug 18, 2013 at 7:55 AM, manish dunani <[email protected]
> > > >wrote:
> > > >
> > > >> *Here is output table===>>>
> > > >>
> > > >> *
> > > >>>
> > > >>> hbase(main):004:0> scan 'nyse5'
> > > >>> ROW
> > > >>> COLUMN+CELL
> > > >>> symbol column=stocks_output:average,
> > > >>> timestamp=1376749641978, val
> > > >>> ue=@\xC6o\x11
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> *Sample output at my eclipse:::*
> > > >>
> > > >> 13/08/17 07:27:21 INFO mapred.Merger: Merging 1 sorted segments
> > > >> 13/08/17 07:27:21 INFO mapred.Merger: Down to the last merge-pass,
> > with
> > > 1
> > > >> segments left of total size: 42242 bytes
> > > >> 13/08/17 07:27:21 INFO mapred.LocalJobRunner:
> > > >> 13/08/17 07:27:21 INFO mapred.JobClient: map 100% reduce 0%
> > > >> *
> > > >> For 2640 average is:6.201058*
> > > >>
> > > >>
> > > >> On Sat, Aug 17, 2013 at 11:59 PM, Jean-Marc Spaggiari <
> > > >> [email protected]> wrote:
> > > >>
> > > >>> Are you outputting to a table? From your code, I don't see any
> output
> > > >>> configured.
> > > >>>
> > > >>> 2013/8/17 manish dunani <[email protected]>
> > > >>>
> > > >>> > Thanx a lot!!
> > > >>> > Jean.
> > > >>> >
> > > >>> > I am very thankful to you..And off course Ted also doing very
> good
> > > job.
> > > >>> > *
> > > >>> >
> > > >>> > Revised Code ::*
> > > >>> >
> > > >>> > Package com.maddy;
> > > >>> > >
> > > >>> > > import java.io.IOException;
> > > >>> > >
> > > >>> > > import org.apache.hadoop.conf.Configuration;
> > > >>> > > import org.apache.hadoop.fs.Path;
> > > >>> > > import org.apache.hadoop.hbase.HBaseConfiguration;
> > > >>> > > import org.apache.hadoop.hbase.client.Put;
> > > >>> > > import org.apache.hadoop.hbase.client.Result;
> > > >>> > > import org.apache.hadoop.hbase.client.Scan;
> > > >>> > > import org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter;
> > > >>> > > import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
> > > >>> > > import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
> > > >>> > > import org.apache.hadoop.hbase.mapreduce.TableMapper;
> > > >>> > > import org.apache.hadoop.hbase.mapreduce.TableReducer;
> > > >>> > > import org.apache.hadoop.hbase.util.Bytes;
> > > >>> > > //import org.apache.hadoop.io.DoubleWritable;
> > > >>> > > import org.apache.hadoop.io.FloatWritable;
> > > >>> > > import org.apache.hadoop.mapreduce.Job;
> > > >>> > > import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
> > > >>> > >
> > > >>> > >
> > > >>> > > public class openaveragestock
> > > >>> > > {
> > > >>> > > public static class map extends
> > > >>> > > TableMapper<ImmutableBytesWritable,FloatWritable>
> > > >>> > > {
> > > >>> > > private static String col_family="stocks";
> > > >>> > > private static String qul="open";
> > > >>> > >
> > > >>> > > private static String col_family1="stocks";
> > > >>> > > private static String qul1="symbol";
> > > >>> > >
> > > >>> > > private static byte[]
> > colfamily2=Bytes.toBytes(col_family);
> > > >>> > > private static byte[] qul2=Bytes.toBytes(qul);
> > > >>> > >
> > > >>> > > private static byte[]
> > > colfamily3=Bytes.toBytes(col_family1);
> > > >>> > > private static byte[] qul3=Bytes.toBytes(qul1);
> > > >>> > >
> > > >>> > > // public static float toFloat(int qul2)
> > > >>> > > // {
> > > >>> > > // return Float.intBitsToFloat((qul2));
> > > >>> > > //
> > > >>> > > // }
> > > >>> > > //
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>> > > public void map(ImmutableBytesWritable row,Result
> > > >>> value,Context
> > > >>> > > context) throws IOException
> > > >>> > > {
> > > >>> > >
> > > >>> > >
> > > >>> > > //byte[]
> > > >>> > > val1=(value.getValue("stocks".getBytes(),"symbol".getBytes()));
> > > >>> > > byte[] val=value.getValue(colfamily2,qul2);
> > > >>> > >
> > > >>> > >
> > > >>> > > ImmutableBytesWritable stock_symbol=new
> > > >>> > > ImmutableBytesWritable(qul3);
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>> > > try
> > > >>> > > {
> > > >>> > >
> > > >>> > > context.write(stock_symbol,new
> > > >>> > > FloatWritable(Float.parseFloat(Bytes.toString(val))));
> > > >>> > > }
> > > >>> > >
> > > >>> > > catch(InterruptedException e)
> > > >>> > >
> > > >>> > > {
> > > >>> > > throw new IOException(e);
> > > >>> > > }
> > > >>> > >
> > > >>> > >
> > > >>> > > }
> > > >>> > >
> > > >>> > >
> > > >>> > > }
> > > >>> > >
> > > >>> > >
> > > >>> > > public static class reduce extends
> > > >>> > >
> > > >>>
> > >
> TableReducer<ImmutableBytesWritable,FloatWritable,ImmutableBytesWritable>
> > > >>> > > {
> > > >>> > >
> > > >>> > > @Override
> > > >>> > > public void reduce(ImmutableBytesWritable
> > > >>> > > key,Iterable<FloatWritable>values,Context context) throws
> > > >>> IOException,
> > > >>> > > InterruptedException
> > > >>> > > {
> > > >>> > > float sum=0;
> > > >>> > > int count=0;
> > > >>> > > float average=0;
> > > >>> > > for(FloatWritable val:values)
> > > >>> > > {
> > > >>> > > sum+=val.get();
> > > >>> > > count++;
> > > >>> > > }
> > > >>> > > average=(sum/count);
> > > >>> > > Put put=new Put(key.get());
> > > >>> > >
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
> put.add(Bytes.toBytes("stocks_output"),Bytes.toBytes("average"),Bytes.toBytes(average));
> > > >>> > > System.out.println("For\t"+count+"\t average
> > > >>> is:"+average);
> > > >>> > > context.write(key,put);
> > > >>> > >
> > > >>> > > }
> > > >>> > >
> > > >>> > > }
> > > >>> > >
> > > >>> > > public static void main(String args[]) throws IOException,
> > > >>> > > ClassNotFoundException, InterruptedException
> > > >>> > > {
> > > >>> > > Configuration config=HBaseConfiguration.create();
> > > >>> > > config.addResource("/home/manish/workspace/hbase
> > > >>> > > project/bin/hbase-site.xml");
> > > >>> > > Job job=new Job(config,"openstockaverage1");
> > > >>> > >
> > > >>> > >
> > > >>> > > Scan scan=new Scan();
> > > >>> > > scan.addFamily("stocks".getBytes());
> > > >>> > > scan.setFilter(new FirstKeyOnlyFilter());
> > > >>> > >
> > > >>> > > TableMapReduceUtil.initTableMapperJob("nyse4",
> > > >>> > > scan,
> > > >>> > > map.class,
> > > >>> > > ImmutableBytesWritable.class,
> > > >>> > > FloatWritable.class,
> > > >>> > > job);
> > > >>> > >
> > > >>> > > TableMapReduceUtil.initTableReducerJob("nyse5",
> > > >>> > > reduce.class,
> > > >>> > > job);
> > > >>> > > //job.setReducerClass(reduce.class);
> > > >>> > >
> > > >>> > > //FileOutputFormat.setOutputPath(job, new Path(
> > > >>> > > //
> > > >>> > > "hdfs://localhost:54310/user/manish/final_hbase_hadoop"));
> > > >>> > > job.waitForCompletion(true);
> > > >>> > > }
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>> > > }
> > > >>> > >
> > > >>> > >
> > > >>> > *Sample output at my eclipse:::*
> > > >>> >
> > > >>> > 13/08/17 07:27:21 INFO mapred.Merger: Merging 1 sorted segments
> > > >>> > > 13/08/17 07:27:21 INFO mapred.Merger: Down to the last
> > merge-pass,
> > > >>> with 1
> > > >>> > > segments left of total size: 42242 bytes
> > > >>> > > 13/08/17 07:27:21 INFO mapred.LocalJobRunner:
> > > >>> > > 13/08/17 07:27:21 INFO mapred.JobClient: map 100% reduce 0%
> > > >>> > > *For 2640 average is:6.201058*
> > > >>> > > 13/08/17 07:27:21 INFO mapred.Task:
> > > >>> Task:attempt_local_0001_r_000000_0 is
> > > >>> > > done. And is in the process of commiting
> > > >>> > > 13/08/17 07:27:21 INFO mapred.LocalJobRunner: reduce > reduce
> > > >>> > > 13/08/17 07:27:21 INFO mapred.Task: Task
> > > >>> 'attempt_local_0001_r_000000_0'
> > > >>> > > done.
> > > >>> > > 13/08/17 07:27:22 WARN mapred.FileOutputCommitter: Output path
> is
> > > >>> null in
> > > >>> > > cleanup
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: map 100% reduce 100%
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Job complete:
> > > job_local_0001
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Counters: 30
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: HBase Counters
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: REMOTE_RPC_CALLS=0
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: RPC_CALLS=2643
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: RPC_RETRIES=0
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient:
> > > >>> > NOT_SERVING_REGION_EXCEPTION=0
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient:
> > NUM_SCANNER_RESTARTS=0
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient:
> > > >>> MILLIS_BETWEEN_NEXTS=5849
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient:
> > > BYTES_IN_RESULTS=126719
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient:
> > > >>> BYTES_IN_REMOTE_RESULTS=0
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: REGIONS_SCANNED=1
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient:
> REMOTE_RPC_RETRIES=0
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: File Output Format
> > > >>> Counters
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Bytes Written=0
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: FileSystemCounters
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient:
> > > FILE_BYTES_READ=24176810
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient:
> > > >>> FILE_BYTES_WRITTEN=24552208
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: File Input Format
> > > Counters
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Bytes Read=0
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Map-Reduce Framework
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Reduce input
> > groups=1
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Map output
> > > materialized
> > > >>> > > bytes=42246
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Combine output
> > > records=0
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Map input
> > records=2640
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Reduce shuffle
> > bytes=0
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Physical memory
> > > (bytes)
> > > >>> > > snapshot=0
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Reduce output
> > > records=1
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Spilled
> Records=5280
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Map output
> > bytes=36960
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: CPU time spent
> > (ms)=0
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Total committed
> heap
> > > >>> usage
> > > >>> > > (bytes)=321527808
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Virtual memory
> > (bytes)
> > > >>> > > snapshot=0
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Combine input
> > > records=0
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Map output
> > > records=2640
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: SPLIT_RAW_BYTES=65
> > > >>> > > 13/08/17 07:27:22 INFO mapred.JobClient: Reduce input
> > > >>> records=2640
> > > >>> > >
> > > >>> >
> > > >>> > *
> > > >>> > *
> > > >>> > *Question:::(please don't laugh at me if i will ask you a silly
> > > >>> question)
> > > >>> >
> > > >>> > *
> > > >>> > Here in code i set output directory.But,when i seen in hdfs
> > directory
> > > >>> it is
> > > >>> > not contain any part-0000 file.It contains only SUCCESS file.*
> > > >>> > *
> > > >>> > Can i ask why it is happen.??*
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> > *
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> > On Sat, Aug 17, 2013 at 5:24 PM, Jean-Marc Spaggiari <
> > > >>> > [email protected]> wrote:
> > > >>> >
> > > >>> > > Hi Manish.
> > > >>> > >
> > > >>> > > Looking a bit more at this, I think the issue is because you
> > > >>> "floats" are
> > > >>> > > written in you table as strings and not as floats....
> > > >>> > >
> > > >>> > > Can you try something link:
> > > >>> > > context.write(stock_symbol,new FloatWritable(
> > > >>> > > Float.parseFloat(Bytes.toString(val)))));
> > > >>> > >
> > > >>> > > Also, as asked previously, " can you please paste you code on
> > > >>> pastbin?
> > > >>> > > Same for the exception."?
> > > >>> > >
> > > >>> > > Thanks,
> > > >>> > >
> > > >>> > > JM
> > > >>> > >
> > > >>> > > 2013/8/17 manish dunani <[email protected]>
> > > >>> > >
> > > >>> > > > Hey jean,
> > > >>> > > > I did it according to you.
> > > >>> > > > I convert as u told..But still face the same error.
> > > >>> > > >
> > > >>> > > > And ted I am new to this don't get an idea how can i use this
> > > >>> > method..Can
> > > >>> > > > You please show me..?
> > > >>> > > >
> > > >>> > > > Your Help will be appreciated..
> > > >>> > > >
> > > >>> > > >
> > > >>> > > >
> > > >>> > > > On Fri, Aug 16, 2013 at 10:04 PM, Ted Yu <
> [email protected]>
> > > >>> wrote:
> > > >>> > > >
> > > >>> > > > > Here is javadoc for toFloat():
> > > >>> > > > >
> > > >>> > > > > * Presumes float encoded as IEEE 754 floating-point
> > "single
> > > >>> > format"
> > > >>> > > > >
> > > >>> > > > > * @param bytes byte array
> > > >>> > > > >
> > > >>> > > > > * @return Float made from passed byte array.
> > > >>> > > > >
> > > >>> > > > > */
> > > >>> > > > >
> > > >>> > > > > public static float toFloat(byte [] bytes) {
> > > >>> > > > >
> > > >>> > > > > So for values of '2.5', toFloat() is not the proper method.
> > > >>> > > > >
> > > >>> > > > > You can float conversion provided by Java.
> > > >>> > > > >
> > > >>> > > > >
> > > >>> > > > > Cheers
> > > >>> > > > >
> > > >>> > > > >
> > > >>> > > > > On Fri, Aug 16, 2013 at 6:57 AM, Ted Yu <
> [email protected]
> > >
> > > >>> wrote:
> > > >>> > > > >
> > > >>> > > > > > Here is code from Bytes:
> > > >>> > > > > >
> > > >>> > > > > > public static float toFloat(byte [] bytes, int offset)
> {
> > > >>> > > > > >
> > > >>> > > > > > return Float.intBitsToFloat(toInt(bytes, offset,
> > > >>> SIZEOF_INT));
> > > >>> > > > > >
> > > >>> > > > > > Looking at your sample data:
> > > >>> > > > > >
> > > >>> > > > > > 2010-02-04 column=stocks:open,
> > > >>> timestamp=1376567559424,
> > > >>> > > > > > value=*2.5*
> > > >>> > > > > >
> > > >>> > > > > > The length of value didn't match SIZEOF_INT.
> > > >>> > > > > >
> > > >>> > > > > > It seems you need to validate the values first.
> > > >>> > > > > >
> > > >>> > > > > >
> > > >>> > > > > > Cheers
> > > >>> > > > > >
> > > >>> > > > > >
> > > >>> > > > > > On Fri, Aug 16, 2013 at 3:42 AM, manish dunani <
> > > >>> > [email protected]
> > > >>> > > > > >wrote:
> > > >>> > > > > >
> > > >>> > > > > >> hello,
> > > >>> > > > > >>
> > > >>> > > > > >> I am using apache hadoop 1.1.2 and hbase 0.94.9 on
> pseudo
> > > >>> > distibuted
> > > >>> > > > > mode.
> > > >>> > > > > >>
> > > >>> > > > > >> I am trying to find Average open stocks values.
> > > >>> > > > > >>
> > > >>> > > > > >> *sample dataset in hbase::**(table name:nyse4)*
> > > >>> > > > > >>
> > > >>> > > > > >>
> > > >>> > > > > >> 2010-02-04 column=stocks:open,
> > > >>> timestamp=1376567559424,
> > > >>> > > > > >> value=2.5
> > > >>> > > > > >> 2010-02-04 column=stocks:symbol,
> > > >>> > timestamp=1376567559424,
> > > >>> > > > > >> value=QXM
> > > >>> > > > > >> 2010-02-05 column=stocks:open,
> > > >>> timestamp=1376567559429,
> > > >>> > > > > >> value=2.42
> > > >>> > > > > >> 2010-02-05 column=stocks:symbol,
> > > >>> > timestamp=1376567559429,
> > > >>> > > > > >> value=QXM
> > > >>> > > > > >> 2010-02-08 column=stocks:open,
> > > >>> timestamp=1376567559431,
> > > >>> > > > > >> value=2.33
> > > >>> > > > > >> 2010-02-08 column=stocks:symbol,
> > > >>> > timestamp=1376567559431,
> > > >>> > > > > >> value=QXM
> > > >>> > > > > >>
> > > >>> > > > > >> *code:*(please ignores the lines that are commenting)
> > > >>> > > > > >>
> > > >>> > > > > >>
> > > >>> > > > > >> > package com.maddy;
> > > >>> > > > > >> >
> > > >>> > > > > >> > import java.io.IOException;
> > > >>> > > > > >> >
> > > >>> > > > > >> > import org.apache.hadoop.conf.Configuration;
> > > >>> > > > > >> > import org.apache.hadoop.fs.Path;
> > > >>> > > > > >> > import org.apache.hadoop.hbase.HBaseConfiguration;
> > > >>> > > > > >> > import org.apache.hadoop.hbase.client.Put;
> > > >>> > > > > >> > import org.apache.hadoop.hbase.client.Result;
> > > >>> > > > > >> > import org.apache.hadoop.hbase.client.Scan;
> > > >>> > > > > >> > import
> > org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter;
> > > >>> > > > > >> > import
> > org.apache.hadoop.hbase.io.ImmutableBytesWritable;
> > > >>> > > > > >> > import
> > > org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
> > > >>> > > > > >> > import org.apache.hadoop.hbase.mapreduce.TableMapper;
> > > >>> > > > > >> > import org.apache.hadoop.hbase.mapreduce.TableReducer;
> > > >>> > > > > >> > import org.apache.hadoop.hbase.util.Bytes;
> > > >>> > > > > >> > //import org.apache.hadoop.io.DoubleWritable;
> > > >>> > > > > >> > import org.apache.hadoop.io.FloatWritable;
> > > >>> > > > > >> > import org.apache.hadoop.mapreduce.Job;
> > > >>> > > > > >> > import
> > > >>> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
> > > >>> > > > > >> >
> > > >>> > > > > >> >
> > > >>> > > > > >> > public class openaveragestock
> > > >>> > > > > >> > {
> > > >>> > > > > >> > public static class map extends
> > > >>> > > > > >> > TableMapper<ImmutableBytesWritable,FloatWritable>
> > > >>> > > > > >> > {
> > > >>> > > > > >> > @Override
> > > >>> > > > > >> >
> > > >>> > > > > >> > public void map(ImmutableBytesWritable
> > row,Result
> > > >>> > > > > value,Context
> > > >>> > > > > >> > context) throws IOException
> > > >>> > > > > >> > {
> > > >>> > > > > >> >
> > > >>> > > > > >> > byte[]
> > > >>> > > > > >> >
> > > val=(value.getValue("stocks".getBytes(),"open".getBytes()));
> > > >>> > > > > >> > //byte[]
> > > >>> > > > > >> >
> > > >>> val1=(value.getValue("stocks".getBytes(),"symbol".getBytes()));
> > > >>> > > > > >> >
> > > >>> > > > > >> >
> > > >>> > > > > >> > ImmutableBytesWritable stock_symbol=new
> > > >>> > > > > >> > ImmutableBytesWritable("symbol".getBytes());
> > > >>> > > > > >> >
> > > >>> > > > > >> >
> > > >>> > > > > >> > try
> > > >>> > > > > >> > {
> > > >>> > > > > >> > context.write(stock_symbol,new
> > > >>> > > > > >> > FloatWritable(Bytes.toFloat(val)));
> > > >>> > > > > >> > }
> > > >>> > > > > >> > catch(InterruptedException e)
> > > >>> > > > > >> >
> > > >>> > > > > >> > {
> > > >>> > > > > >> > throw new IOException(e);
> > > >>> > > > > >> > }
> > > >>> > > > > >> >
> > > >>> > > > > >> >
> > > >>> > > > > >> > }
> > > >>> > > > > >> >
> > > >>> > > > > >> >
> > > >>> > > > > >> > }
> > > >>> > > > > >> >
> > > >>> > > > > >> >
> > > >>> > > > > >> > public static class reduce extends
> > > >>> > > > > >> >
> > > >>> > > > > >>
> > > >>> > > > >
> > > >>> > >
> > > >>>
> > >
> TableReducer<ImmutableBytesWritable,FloatWritable,ImmutableBytesWritable>
> > > >>> > > > > >> > {
> > > >>> > > > > >> >
> > > >>> > > > > >> > @Override
> > > >>> > > > > >> > public void reduce(ImmutableBytesWritable
> > > >>> > > > > >> > key,Iterable<FloatWritable>values,Context context)
> > throws
> > > >>> > > > IOException,
> > > >>> > > > > >> > InterruptedException
> > > >>> > > > > >> > {
> > > >>> > > > > >> > float sum=0;
> > > >>> > > > > >> > int count=0;
> > > >>> > > > > >> > // float average=0;
> > > >>> > > > > >> > for(FloatWritable val:values)
> > > >>> > > > > >> > {
> > > >>> > > > > >> > sum+=val.get();
> > > >>> > > > > >> > count++;
> > > >>> > > > > >> > }
> > > >>> > > > > >> > //average=(sum/count);
> > > >>> > > > > >> > Put put=new Put(key.get());
> > > >>> > > > > >> >
> > > >>> > > > > >> >
> > > >>> > > > > >>
> > > >>> > > > >
> > > >>> > > >
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
> put.add(Bytes.toBytes("stocks_output"),Bytes.toBytes("average"),Bytes.toBytes(sum/count));
> > > >>> > > > > >> > System.out.println("For\t"+count+"\t
> average
> > > >>> > > > > >> is:"+(sum/count));
> > > >>> > > > > >> > context.write(key,put);
> > > >>> > > > > >> >
> > > >>> > > > > >> > }
> > > >>> > > > > >> >
> > > >>> > > > > >> > }
> > > >>> > > > > >> >
> > > >>> > > > > >> > public static void main(String args[]) throws
> > > >>> IOException,
> > > >>> > > > > >> > ClassNotFoundException, InterruptedException
> > > >>> > > > > >> > {
> > > >>> > > > > >> > Configuration
> > config=HBaseConfiguration.create();
> > > >>> > > > > >> >
> config.addResource("/home/manish/workspace/hbase
> > > >>> > > > > >> > project/bin/hbase-site.xml");
> > > >>> > > > > >> > Job job=new Job(config,"openstockaverage1");
> > > >>> > > > > >> >
> > > >>> > > > > >> >
> > > >>> > > > > >> > Scan scan=new Scan();
> > > >>> > > > > >> > scan.addFamily("stocks".getBytes());
> > > >>> > > > > >> > scan.setFilter(new FirstKeyOnlyFilter());
> > > >>> > > > > >> >
> > > >>> > > > > >> > TableMapReduceUtil.initTableMapperJob("nyse4",
> > > >>> > > > > >> > scan,
> > > >>> > > > > >> > map.class,
> > > >>> > > > > >> > ImmutableBytesWritable.class,
> > > >>> > > > > >> > FloatWritable.class,
> > > >>> > > > > >> > job);
> > > >>> > > > > >> >
> > > >>> > > > > >> >
> TableMapReduceUtil.initTableReducerJob("nyse5",
> > > >>> > > > > >> > reduce.class,
> > > >>> > > > > >> > job);
> > > >>> > > > > >> > //job.setReducerClass(reduce.class);
> > > >>> > > > > >> >
> > > >>> > > > > >> > FileOutputFormat.setOutputPath(job, new Path(
> > > >>> > > > > >> >
> > > >>> > > > > >>
> > "hdfs://localhost:54310/user/manish/edurekahbasehadoop1"));
> > > >>> > > > > >> > job.waitForCompletion(true);
> > > >>> > > > > >> > }
> > > >>> > > > > >> >
> > > >>> > > > > >> > }
> > > >>> > > > > >> >
> > > >>> > > > > >> >
> > > >>> > > > > >> >
> > > >>> > > > > >> > *===>Got stuck into error:*
> > > >>> > > > > >> >
> > > >>> > > > > >> >
> > > >>> > > > > >> > 13/08/16 03:21:45 INFO mapred.JobClient: Running job:
> > > >>> > > job_local_0001
> > > >>> > > > > >> > 13/08/16 03:21:46 INFO mapred.JobClient: map 0%
> reduce
> > 0%
> > > >>> > > > > >> > 13/08/16 03:21:46 INFO mapreduce.TableOutputFormat:
> > > Created
> > > >>> > table
> > > >>> > > > > >> instance
> > > >>> > > > > >> > for nyse5
> > > >>> > > > > >> > 13/08/16 03:21:46 INFO util.ProcessTree: setsid exited
> > > with
> > > >>> exit
> > > >>> > > > code
> > > >>> > > > > 0
> > > >>> > > > > >> > 13/08/16 03:21:47 INFO mapred.Task: Using
> > > >>> > > ResourceCalculatorPlugin
> > > >>> > > > :
> > > >>> > > > > >> >
> > > org.apache.hadoop.util.LinuxResourceCalculatorPlugin@50b77c
> > > >>> > > > > >> > 13/08/16 03:21:47 INFO mapred.MapTask: io.sort.mb =
> 100
> > > >>> > > > > >> > 13/08/16 03:21:53 INFO mapred.MapTask: data buffer =
> > > >>> > > > 79691776/99614720
> > > >>> > > > > >> > 13/08/16 03:21:53 INFO mapred.MapTask: record buffer =
> > > >>> > > 262144/327680
> > > >>> > > > > >> > 13/08/16 03:21:54 WARN mapred.LocalJobRunner:
> > > job_local_0001
> > > >>> > > > > >> > java.lang.IllegalArgumentException: offset (0) +
> length
> > > (4)
> > > >>> > exceed
> > > >>> > > > the
> > > >>> > > > > >> > capacity of the array: 3
> > > >>> > > > > >> > at
> > > >>> > > > > >> >
> > > >>> > > > > >>
> > > >>> > > > >
> > > >>> > > >
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
> org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:543)
> > > >>> > > > > >> > at
> > > >>> org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:690)
> > > >>> > > > > >> > at
> > > >>> > org.apache.hadoop.hbase.util.Bytes.toFloat(Bytes.java:584)
> > > >>> > > > > >> > at
> > > >>> > org.apache.hadoop.hbase.util.Bytes.toFloat(Bytes.java:574)
> > > >>> > > > > >> > at
> > > >>> > > com.maddy.openaveragestock$map.map(openaveragestock.java:41)
> > > >>> > > > > >> > at
> > > >>> > com.maddy.openaveragestock$map.map(openaveragestock.java:1)
> > > >>> > > > > >> > at
> > > >>> org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > > >>> > > > > >> > at
> > > >>> > > >
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> > > >>> > > > > >> > at
> > > >>> org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> > > >>> > > > > >> > at
> > > >>> > > > > >> >
> > > >>> > > > >
> > > >>> > >
> > > >>>
> > >
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:214)
> > > >>> > > > > >> >
> > > >>> > > > > >> >
> > > >>> > > > > >> I cannot find where is it fail??
> > > >>> > > > > >> Can you please tell me??
> > > >>> > > > > >> where i was wrong..?
> > > >>> > > > > >>
> > > >>> > > > > >>
> > > >>> > > > > >> Your help will be appreciated.
> > > >>> > > > > >>
> > > >>> > > > > >>
> > > >>> > > > > >>
> > > >>> > > > > >>
> > > >>> > > > > >>
> > > >>> > > > > >>
> > > >>> > > > > >>
> > > >>> > > > > >>
> > > >>> > > > > >>
> > > >>> > > > > >>
> > > >>> > > > > >> --
> > > >>> > > > > >> Regards
> > > >>> > > > > >>
> > > >>> > > > > >> *Manish Dunani*
> > > >>> > > > > >> *Contact No* : +91 9408329137
> > > >>> > > > > >> *skype id* : manish.dunani*
> > > >>> > > > > >> *
> > > >>> > > > > >>
> > > >>> > > > > >
> > > >>> > > > > >
> > > >>> > > > >
> > > >>> > > >
> > > >>> > > >
> > > >>> > > >
> > > >>> > > > --
> > > >>> > > > Regards
> > > >>> > > >
> > > >>> > > > *Manish Dunani*
> > > >>> > > > *Contact No* : +91 9408329137
> > > >>> > > > *skype id* : manish.dunani*
> > > >>> > > > *
> > > >>> > > >
> > > >>> > >
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> > --
> > > >>> > Regards
> > > >>> >
> > > >>> > *Manish Dunani*
> > > >>> > *Contact No* : +91 9408329137
> > > >>> > *skype id* : manish.dunani*
> > > >>> > *
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Regards
> > > >>
> > > >> *Manish Dunani*
> > > >> *Contact No* : +91 9408329137
> > > >> *skype id* : manish.dunani*
> > > >> *
> > > >>
> > > >>
> > > >>
> > > >
> > > >
> > > > --
> > > > Regards
> > > >
> > > > *Manish Dunani*
> > > > *Contact No* : +91 9408329137
> > > > *skype id* : manish.dunani*
> > > > *
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Regards
> > >
> > > *Manish Dunani*
> > > *Contact No* : +91 9408329137
> > > *skype id* : manish.dunani*
> > > *
> > >
> >
>
>
>
> --
> Regards
>
> *Manish Dunani*
> *Contact No* : +91 9408329137
> *skype id* : manish.dunani*
> *
>
