Jenkins build is back to normal : Hama-Nightly-for-Hadoop-1.x #1014

2013-08-24 Thread Apache Jenkins Server 
See

ML test case fails

2013-08-24 Thread Chia-Hung Lin 
When testing with the latest source obtained from svn (r1514736), ml
test seems to fail. Is any setting is required? Just to check in case
it's environment specific issue.

---
 T E S T S
---
Running org.apache.hama.ml.ann.TestSmallLayeredNeuralNetwork
13/08/24 17:38:20 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable

13/08/24 17:38:29 INFO mortbay.log: Training time: 9.488000s


13/08/24 17:38:29 INFO mortbay.log: Relative error: 20.00%


13/08/24 17:38:20 INFO mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog

13/08/24 17:38:20 INFO bsp.FileInputFormat: Total input paths to process : 1

13/08/24 17:38:20 INFO bsp.FileInputFormat: Total input paths to process : 1

13/08/24 17:38:20 WARN bsp.BSPJobClient: No job jar file set.  User
classes may not be found. See BSPJob#setJar(String) or check Your jar
file.

13/08/24 17:38:20 INFO bsp.BSPJobClient: Running job: job_localrunner_0001

13/08/24 17:38:20 INFO bsp.LocalBSPRunner: Setting up a new barrier for 1 tasks!

13/08/24 17:38:23 INFO bsp.BSPJobClient: Current supersteps number: 1

13/08/24 17:38:23 INFO bsp.BSPJobClient: The total number of supersteps: 1

13/08/24 17:38:23 INFO bsp.BSPJobClient: Counters: 6

13/08/24 17:38:23 INFO bsp.BSPJobClient:
org.apache.hama.bsp.JobInProgress$JobCounter

13/08/24 17:38:23 INFO bsp.BSPJobClient: SUPERSTEPS=1

13/08/24 17:38:23 INFO bsp.BSPJobClient: LAUNCHED_TASKS=1

13/08/24 17:38:23 INFO bsp.BSPJobClient:
org.apache.hama.bsp.BSPPeerImpl$PeerCounter

13/08/24 17:38:23 INFO bsp.BSPJobClient: SUPERSTEP_SUM=2

13/08/24 17:38:23 INFO bsp.BSPJobClient: IO_BYTES_READ=57416

13/08/24 17:38:23 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=0

13/08/24 17:38:23 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=618

13/08/24 17:38:23 INFO bsp.FileInputFormat: Total input paths to process : 5

13/08/24 17:38:23 WARN bsp.BSPJobClient: No job jar file set.  User
classes may not be found. See BSPJob#setJar(String) or check Your jar
file.

13/08/24 17:38:23 INFO bsp.BSPJobClient: Running job: job_localrunner_0001

13/08/24 17:38:23 INFO bsp.LocalBSPRunner: Setting up a new barrier for 5 tasks!

13/08/24 17:38:23 INFO mortbay.log: Begin to train

13/08/24 17:38:26 INFO bsp.BSPJobClient: Current supersteps number: 375

13/08/24 17:38:29 INFO bsp.BSPJobClient: Current supersteps number: 849

13/08/24 17:38:32 INFO bsp.BSPJobClient: Current supersteps number: 1377

13/08/24 17:38:35 INFO bsp.BSPJobClient: Current supersteps number: 1877

13/08/24 17:38:38 INFO bsp.BSPJobClient: Current supersteps number: 2435

13/08/24 17:38:41 INFO bsp.BSPJobClient: Current supersteps number: 3001

13/08/24 17:38:44 INFO bsp.BSPJobClient: Current supersteps number: 3573

13/08/24 17:38:46 INFO mortbay.log: End of training, number of iterations: 2001.


13/08/24 17:38:46 INFO mortbay.log: Write model back to /tmp/distributed-model


13/08/24 17:38:47 INFO bsp.BSPJobClient: Current supersteps number: 3999

13/08/24 17:38:47 INFO bsp.BSPJobClient: The total number of supersteps: 3999

13/08/24 17:38:47 INFO bsp.BSPJobClient: Counters: 8

13/08/24 17:38:47 INFO bsp.BSPJobClient:
org.apache.hama.bsp.JobInProgress$JobCounter

13/08/24 17:38:47 INFO bsp.BSPJobClient: SUPERSTEPS=3999

13/08/24 17:38:47 INFO bsp.BSPJobClient: LAUNCHED_TASKS=5

13/08/24 17:38:47 INFO bsp.BSPJobClient:
org.apache.hama.bsp.BSPPeerImpl$PeerCounter

13/08/24 17:38:47 INFO bsp.BSPJobClient: SUPERSTEP_SUM=2

13/08/24 17:38:47 INFO bsp.BSPJobClient: IO_BYTES_READ=278427240

13/08/24 17:38:47 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=39611

13/08/24 17:38:47 INFO bsp.BSPJobClient: TOTAL_MESSAGES_SENT=2

13/08/24 17:38:47 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=300

13/08/24 17:38:47 INFO bsp.BSPJobClient: TOTAL_MESSAGES_RECEIVED=2

13/08/24 17:38:47 INFO mortbay.log: Reload model from /tmp/distributed-model.

13/08/24 17:38:47 INFO mortbay.log: Training time: 27.49s


13/08/24 17:38:47 INFO mortbay.log: Relative error: 24.67%


Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.009
sec <<< FAILURE!

Re: HybridBSP (CPU and GPU) Task Integration

2013-08-24 Thread Chia-Hung Lin 
It seems to me that an additional process or thread will be launched
for running a GPU-based bsp task, which will then communicate with
PipesBSP process, as [1]. Please correct me if it is wrong.

If this is the case, BSPTask looks like the place to work on. When
BSPTask process is running, it can check (e.g. in runBSP) if
additional GPU process/ thread is needed to be created; then launch/
destroy such task accordingly.

By the way, it is mentioned that scheduling is needed. Can you please
give a bit more detail on what kind of scheduling is required?

[1]. http://i.imgur.com/RP3ETBW.png


On 24 August 2013 00:59, Martin Illecker  wrote:
>>
>> What's the difference between launching `bsp task' and `gpu bsp task'?
>> Will gpu bsp task fork and execute c/ c++ process?
>
>
> The GPU bsp task can also be executed within a Java process.
>
> In detail I want to run a Rootbeer Kernel (e.g., PiEstimationKernel [1])
> within the bspGpu method.
> A Rootbeer Kernel is written in Java and converted to CUDA. (the entry
> point is the gpuMethod)
> Finally there is a Java wrapper around the CUDA code, so it can be invoked
> within the JVM.
>
> So far there is no difference between a normal bsp task execution but I
> want to use Hama Pipes to communicate via sockets.
> The GPU bsp task should start like the default one but I will have to
> establish the Pipes Server for communication.
> And of course I need scheduling for theses GPU and CPU tasks.
>
> I hope the following source will illustrate my scenario better:
>
> public class MyHybridBSP extends
>  HybridBSP Text> {
>
>@Override
>public void bsp(BSPPeer NullWritable, Text> peer)
>throws IOException, SyncException, InterruptedException {
>
>  MyGPUKernel kernel = new MyGPUKernel();
>  Rootbeer rootbeer = new Rootbeer();
>  rootbeer.setThreadConfig(BLOCK_SIZE, GRID_SIZE, BLOCK_SIZE*GRID_SIZE);
>
>
>  // Run GPU Kernels
>  rootbeer.runAll(kernel);
>}
>
>@Override
>public void bspGpu(BSPPeer NullWritable, Text> peer)
>throws IOException, SyncException, InterruptedException {
>
>  // process algorithm on CPU
>}
>
>class MyGPUKernel implements Kernel {
>  public PiEstimatorKernel() { }
>
>  public void gpuMethod() {
>// process algorithm on GPU
>
>// the following commands will need Hama Pipes
>HamaPeer.getConfiguration();
>HamaPeer.readNext(...,...);
>// and others
>  }
> }
>
> Thanks!
>
> Martin
>
> [1]
> https://github.com/millecker/applications/blob/master/hama/rootbeer/piestimator/src/at/illecker/hama/rootbeer/examples/piestimator/gpu/PiEstimatorKernel.java
>
> 2013/8/23 Chia-Hung Lin 
>
>> What's the difference between launching `bsp task' and `gpu bsp task'?
>> Will gpu bsp task fork and execute c/ c++ process?
>>
>> It might be good to distinguish how gpu bsp task will be executed,
>> then deciding how to launch such task.
>>
>> Basically for launching a bsp task, an external process is created.
>> The logic to execute BSP.bsp() is at
>>
>> BSPTask.java
>>
>> where the method
>>
>> runBSP()
>>
>> is called with a BSP implementation class loaded at runtime
>>
>> Class workClass = job.getConfiguration().getClass("bsp.work.class",
>> BSP.class);
>>
>> and then the bsp method is executed
>>
>> bsp.bsp(bspPeer);
>>
>>
>>
>>
>>
>>
>> On 23 August 2013 21:45, Martin Illecker  wrote:
>> > Hi,
>> >
>> > I have created a HybridBSP [1] class which should combine the default BSP
>> > (CPU) class with GPU methods [2].
>> >
>> > The abstract HybridBSP class extends the BSP class and adds bspGpu,
>> > setupGpu and cleanupGpu method.
>> >
>> > public abstract class HybridBSP
>> extends
>> > BSP implements BSPGpuInterface
>> {
>> >
>> >   @Override
>> >   public abstract void bspGpu(BSPPeer peer)
>> >   throws IOException, SyncException, InterruptedException;
>> >
>> >   @Override
>> >   public void setupGpu(BSPPeer peer) throws
>> IOException,
>> >   SyncException, InterruptedException {
>> >   }
>> >
>> >   @Override
>> >   public void cleanupGpu(BSPPeer peer) throws
>> IOException {
>> >   }
>> > }
>> >
>> >
>> > Now I want to add a new scheduling technique which checks the conf
>> property
>> > (gpuBspTaskNum) and executes the bspGpu instead of default bsp method.
>> >
>> > e.g., bspTaskNum=3 and gpuBspTaskNum=1
>> > The scheduler should run four bsp tasks simultaneously and execute three
>> > times the bsp method and once the bspGpu. (both defined within one
>> derived
>> > HybridBSP class)
>> >
>> > Do I have to modify the taskrunner or create a new SimpleTaskScheduler?
>> >
>> > How can I integrate this into Hama?
>> >
>> > Thanks!
>> >
>> > Martin
>> >
>> > [1]
>> >
>> https://github.com/millecker/hama/blob/5d0e8b26abd6b63fa5afad09a2ba960bf9922868/core/src/main/java/org/apache/hama/bsp/gpu/HybridBSP.java
>> > [2]
>> >
>> https://github.com/millecker/hama/blob/5d0e8b26abd6b63fa5afad09a2ba960bf9922868/core/src/mai

ML test fails

2013-08-24 Thread Chia-Hung Lin 
When testing with the latest source obtained from svn (r1514736), ml
test seems to fail. Is any setting is required? Just to check in case
it's environment specific issue.

---
 T E S T S
---
Running org.apache.hama.ml.ann.TestSmallLayeredNeuralNetwork
13/08/24 17:38:20 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable

13/08/24 17:38:29 INFO mortbay.log: Training time: 9.488000s


13/08/24 17:38:29 INFO mortbay.log: Relative error: 20.00%


13/08/24 17:38:20 INFO mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog

13/08/24 17:38:20 INFO bsp.FileInputFormat: Total input paths to process : 1

13/08/24 17:38:20 INFO bsp.FileInputFormat: Total input paths to process : 1

13/08/24 17:38:20 WARN bsp.BSPJobClient: No job jar file set.  User
classes may not be found. See BSPJob#setJar(String) or check Your jar
file.

13/08/24 17:38:20 INFO bsp.BSPJobClient: Running job: job_localrunner_0001

13/08/24 17:38:20 INFO bsp.LocalBSPRunner: Setting up a new barrier for 1 tasks!

13/08/24 17:38:23 INFO bsp.BSPJobClient: Current supersteps number: 1

13/08/24 17:38:23 INFO bsp.BSPJobClient: The total number of supersteps: 1

13/08/24 17:38:23 INFO bsp.BSPJobClient: Counters: 6

13/08/24 17:38:23 INFO bsp.BSPJobClient:
org.apache.hama.bsp.JobInProgress$JobCounter

13/08/24 17:38:23 INFO bsp.BSPJobClient: SUPERSTEPS=1

13/08/24 17:38:23 INFO bsp.BSPJobClient: LAUNCHED_TASKS=1

13/08/24 17:38:23 INFO bsp.BSPJobClient:
org.apache.hama.bsp.BSPPeerImpl$PeerCounter

13/08/24 17:38:23 INFO bsp.BSPJobClient: SUPERSTEP_SUM=2

13/08/24 17:38:23 INFO bsp.BSPJobClient: IO_BYTES_READ=57416

13/08/24 17:38:23 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=0

13/08/24 17:38:23 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=618

13/08/24 17:38:23 INFO bsp.FileInputFormat: Total input paths to process : 5

13/08/24 17:38:23 WARN bsp.BSPJobClient: No job jar file set.  User
classes may not be found. See BSPJob#setJar(String) or check Your jar
file.

13/08/24 17:38:23 INFO bsp.BSPJobClient: Running job: job_localrunner_0001

13/08/24 17:38:23 INFO bsp.LocalBSPRunner: Setting up a new barrier for 5 tasks!

13/08/24 17:38:23 INFO mortbay.log: Begin to train

13/08/24 17:38:26 INFO bsp.BSPJobClient: Current supersteps number: 375

13/08/24 17:38:29 INFO bsp.BSPJobClient: Current supersteps number: 849

13/08/24 17:38:32 INFO bsp.BSPJobClient: Current supersteps number: 1377

13/08/24 17:38:35 INFO bsp.BSPJobClient: Current supersteps number: 1877

13/08/24 17:38:38 INFO bsp.BSPJobClient: Current supersteps number: 2435

13/08/24 17:38:41 INFO bsp.BSPJobClient: Current supersteps number: 3001

13/08/24 17:38:44 INFO bsp.BSPJobClient: Current supersteps number: 3573

13/08/24 17:38:46 INFO mortbay.log: End of training, number of iterations: 2001.


13/08/24 17:38:46 INFO mortbay.log: Write model back to /tmp/distributed-model


13/08/24 17:38:47 INFO bsp.BSPJobClient: Current supersteps number: 3999

13/08/24 17:38:47 INFO bsp.BSPJobClient: The total number of supersteps: 3999

13/08/24 17:38:47 INFO bsp.BSPJobClient: Counters: 8

13/08/24 17:38:47 INFO bsp.BSPJobClient:
org.apache.hama.bsp.JobInProgress$JobCounter

13/08/24 17:38:47 INFO bsp.BSPJobClient: SUPERSTEPS=3999

13/08/24 17:38:47 INFO bsp.BSPJobClient: LAUNCHED_TASKS=5

13/08/24 17:38:47 INFO bsp.BSPJobClient:
org.apache.hama.bsp.BSPPeerImpl$PeerCounter

13/08/24 17:38:47 INFO bsp.BSPJobClient: SUPERSTEP_SUM=2

13/08/24 17:38:47 INFO bsp.BSPJobClient: IO_BYTES_READ=278427240

13/08/24 17:38:47 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=39611

13/08/24 17:38:47 INFO bsp.BSPJobClient: TOTAL_MESSAGES_SENT=2

13/08/24 17:38:47 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=300

13/08/24 17:38:47 INFO bsp.BSPJobClient: TOTAL_MESSAGES_RECEIVED=2

13/08/24 17:38:47 INFO mortbay.log: Reload model from /tmp/distributed-model.

13/08/24 17:38:47 INFO mortbay.log: Training time: 27.49s


13/08/24 17:38:47 INFO mortbay.log: Relative error: 24.67%


Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.009
sec <<< FAILURE!