[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405426#comment-15405426 ] ASF GitHub Bot commented on HAMA-992: - Github user asfgit closed the pull request at: https://github.com/apache/hama/pull/16 > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > 16/06/01 14:48:52 ERROR bsp.BSPTask: Error running bsp setup and bsp function. > java.io.IOException: Stream closed > at > java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) > at java.io.OutputStream.write(OutputStream.java:116) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) > at java.io.BufferedOutp
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15320052#comment-15320052 ] Hudson commented on HAMA-992: - ABORTED: Integrated in Hama-Nightly-for-Hadoop-1.x #1731 (See [https://builds.apache.org/job/Hama-Nightly-for-Hadoop-1.x/1731/]) HAMA-992: Adding Hama streaming (edwardyoon: rev 556c7b386b1fb27a41763a61d1126e1dd1bd216a) * bin/hama-daemon.sh * python/BspJobConfiguration.py * python/BinaryProtocol.py * bin/hama * python/README.txt * bin/zookeepers.sh * bin/start-bspd.sh * conf/hama-default.xml * python/HelloWorldBSP.py * python/BSPPeer.py * conf/hama-env.sh * CHANGES.txt * python/BSPRunner.py * conf/hama-site.xml * bin/grooms.sh * bin/hama-config.sh * bin/hama-daemons.sh * python/KMeansBSP.py * conf/groomservers * python/BSP.py * bin/stop-bspd.sh > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code:
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319777#comment-15319777 ] Hudson commented on HAMA-992: - SUCCESS: Integrated in Hama-trunk #480 (See [https://builds.apache.org/job/Hama-trunk/480/]) HAMA-992: Adding Hama streaming (edwardyoon: rev 556c7b386b1fb27a41763a61d1126e1dd1bd216a) * python/BSPRunner.py * bin/hama-daemon.sh * bin/hama * bin/start-bspd.sh * python/BspJobConfiguration.py * python/BinaryProtocol.py * python/BSP.py * python/BSPPeer.py * bin/hama-daemons.sh * python/HelloWorldBSP.py * bin/stop-bspd.sh * python/KMeansBSP.py * bin/zookeepers.sh * conf/groomservers * python/README.txt * CHANGES.txt * bin/hama-config.sh * conf/hama-default.xml * conf/hama-site.xml * bin/grooms.sh * conf/hama-env.sh > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pi
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319710#comment-15319710 ] Hudson commented on HAMA-992: - SUCCESS: Integrated in Hama-Nightly-for-Hadoop-2.x #728 (See [https://builds.apache.org/job/Hama-Nightly-for-Hadoop-2.x/728/]) HAMA-992: Adding Hama streaming (edwardyoon: rev 556c7b386b1fb27a41763a61d1126e1dd1bd216a) * conf/hama-site.xml * bin/hama * conf/hama-default.xml * python/BinaryProtocol.py * python/KMeansBSP.py * python/BSP.py * python/BSPRunner.py * bin/hama-daemons.sh * python/HelloWorldBSP.py * python/BSPPeer.py * bin/zookeepers.sh * bin/grooms.sh * conf/groomservers * bin/hama-config.sh * conf/hama-env.sh * bin/start-bspd.sh * bin/stop-bspd.sh * bin/hama-daemon.sh * python/README.txt * python/BspJobConfiguration.py * CHANGES.txt > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code: -
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15318241#comment-15318241 ] Edward J. Yoon commented on HAMA-992: - Thanks. +1 I'll merge tomorrow. > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > 16/06/01 14:48:52 ERROR bsp.BSPTask: Error running bsp setup and bsp function. > java.io.IOException: Stream closed > at > java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) > at java.io.OutputStream.write(OutputStream.java:116) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141) > at jav
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15315566#comment-15315566 ] ASF GitHub Bot commented on HAMA-992: - GitHub user thomasjungblut opened a pull request: https://github.com/apache/hama/pull/16 [HAMA-992] Add Python Hama streaming - updated the protocol to be in sync again - added license headers You can merge this pull request into a Git repository by running: $ git pull https://github.com/thomasjungblut/hama master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hama/pull/16.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16 commit 4fcc3159f1009a1a7cce4511b7b5febdac84aa86 Author: Thomas Jungblut Date: 2016-06-04T16:34:25Z adding the python module for streaming > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 >
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311979#comment-15311979 ] Thomas Jungblut commented on HAMA-992: -- Thanks, but this is not proper BSP code and thus won't work. Have a look at the kmeans example: https://github.com/thomasjungblut/HamaStreaming/blob/master/KMeansBSP.py You need to inherit from BSP and use the framework, otherwise you code won't be able to run and communicate with the Hama runtime. > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > 16/06/01 14:48:52 ERROR bsp.BSPTask: Error running bsp setup and bsp function. > java.io.IOException: Stream closed > at > java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) > at java.io.OutputStream.write(Out
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311869#comment-15311869 ] Chaitanya commented on HAMA-992: I made a new jar file according to the changes suggested by you and copied it into the hdfs. Now its a new copy of hama on the HDFS with streaming folder inside hama/conf. But everything came back to square one, it is behaving in the same way as my post describes. > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > 16/06/01 14:48:52 ERROR bsp.BSPTask: Error running bsp setup and bsp function. > java.io.IOException: Stream closed > at > java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) > at java.io.OutputStream.write(OutputStream.java:116) > at > java.io.Buff
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311776#comment-15311776 ] Chaitanya commented on HAMA-992: I initially forgot to upload the file in HDFS. Now, I have but it's throwing the exact same error. Also, It shows the same error irrespective of any python code i run. > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > 16/06/01 14:48:52 ERROR bsp.BSPTask: Error running bsp setup and bsp function. > java.io.IOException: Stream closed > at > java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) > at java.io.OutputStream.write(OutputStream.java:116) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.BufferedOutputStream.
[jira] [Commented] (HAMA-992) Hama streaming
[
https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311760#comment-15311760
]
Chaitanya commented on HAMA-992:
Hello Thomas,
I am running the following python code meant for CFD simulations which runs
well outside the hama environment.
# In[1]:
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from matplotlib import pyplot
import numpy
#get_ipython().magic(u'matplotlib inline')
nx = 41
ny = 41
nt = 500
nit=50
c = 1
dx = 2./(nx-1)
dy = 2./(ny-1)
x = numpy.linspace(0,2,nx)
y = numpy.linspace(0,2,ny)
X,Y = numpy.meshgrid(x,y)
rho = 1
nu = .1
dt = .001
u = numpy.zeros((ny, nx))
v = numpy.zeros((ny, nx))
p = numpy.zeros((ny, nx))
b = numpy.zeros((ny, nx))
# The pressure Poisson equation that's written above can be hard to write out
without typos. The function `buildUpB` below represents the contents of the
square brackets, so that the entirety of the PPE is slightly more manageable.
# In[2]:
def buildUpB(b, rho, dt, u, v, dx, dy):
b[1:-1,1:-1]=rho*(1/dt*((u[1:-1,2:]-u[1:-1,0:-2])/(2*dx)+(v[2:,1:-1]-v[0:-2,1:-1])/(2*dy))-((u[1:-1,2:]-u[1:-1,0:-2])/(2*dx))**2-2*((u[2:,1:-1]-u[0:-2,1:-1])/(2*dy)*(v[1:-1,2:]-v[1:-1,0:-2])/(2*dx))-
((v[2:,1:-1]-v[0:-2,1:-1])/(2*dy))**2)
return b
# The function `presPoisson` is also defined to help segregate the different
rounds of calculations. Note the presence of the pseudo-time variable `nit`.
This sub-iteration in the Poisson calculation helps ensure a divergence-free
field.
# In[3]:
def presPoisson(p, dx, dy, b):
pn = numpy.empty_like(p)
pn = p.copy()
for q in range(nit):
pn = p.copy()
p[1:-1,1:-1] =
((pn[1:-1,2:]+pn[1:-1,0:-2])*dy**2+(pn[2:,1:-1]+pn[0:-2,1:-1])*dx**2)/
(2*(dx**2+dy**2)) -
dx**2*dy**2/(2*(dx**2+dy**2))*b[1:-1,1:-1]
p[:,-1] =p[:,-2] ##dp/dy = 0 at x = 2
p[0,:] = p[1,:] ##dp/dy = 0 at y = 0
p[:,0]=p[:,1]##dp/dx = 0 at x = 0
p[-1,:]=0##p = 0 at y = 2
return p
# Finally, the rest of the cavity flow equations are wrapped inside the
function `cavityFlow`, allowing us to easily plot the results of the cavity
flow solver for different lengths of time.
# In[4]:
def cavityFlow(nt, u, v, dt, dx, dy, p, rho, nu):
un = numpy.empty_like(u)
vn = numpy.empty_like(v)
b = numpy.zeros((ny, nx))
for n in range(nt):
un = u.copy()
vn = v.copy()
b = buildUpB(b, rho, dt, u, v, dx, dy)
p = presPoisson(p, dx, dy, b)
u[1:-1,1:-1] = un[1:-1,1:-1]-
un[1:-1,1:-1]*dt/dx*(un[1:-1,1:-1]-un[1:-1,0:-2])-
vn[1:-1,1:-1]*dt/dy*(un[1:-1,1:-1]-un[0:-2,1:-1])-
dt/(2*rho*dx)*(p[1:-1,2:]-p[1:-1,0:-2])+
nu*(dt/dx**2*(un[1:-1,2:]-2*un[1:-1,1:-1]+un[1:-1,0:-2])+
dt/dy**2*(un[2:,1:-1]-2*un[1:-1,1:-1]+un[0:-2,1:-1]))
v[1:-1,1:-1] = vn[1:-1,1:-1]-
un[1:-1,1:-1]*dt/dx*(vn[1:-1,1:-1]-vn[1:-1,0:-2])-
vn[1:-1,1:-1]*dt/dy*(vn[1:-1,1:-1]-vn[0:-2,1:-1])-
dt/(2*rho*dy)*(p[2:,1:-1]-p[0:-2,1:-1])+
nu*(dt/dx**2*(vn[1:-1,2:]-2*vn[1:-1,1:-1]+vn[1:-1,0:-2])+
(dt/dy**2*(vn[2:,1:-1]-2*vn[1:-1,1:-1]+vn[0:-2,1:-1])))
u[0,:] = 0
u[:,0] = 0
u[:,-1] = 0
u[-1,:] = 1#set velocity on cavity lid equal to 1
v[0,:] = 0
v[-1,:]=0
v[:,0] = 0
v[:,-1] = 0
return u, v, p
# Let's start with `nt = 100` and see what the solver gives us:
# In[5]:
u = numpy.zeros((ny, nx))
v = numpy.zeros((ny, nx))
p = numpy.zeros((ny, nx))
b = numpy.zeros((ny, nx))
nt = 100
u, v, p = cavityFlow(nt, u, v, dt, dx, dy, p, rho, nu)
fig = pyplot.figure(figsize=(11,7), dpi=100)
pyplot.contourf(X,Y,p,alpha=0.5)###plnttong the pressure field as a contour
pyplot.colorbar()
pyplot.contour(X,Y,p) ###plotting the pressure field outlines
pyplot.quiver(X[::2,::2],Y[::2,::2],u[::2,::2],v[::2,::2]) ##plotting velocity
pyplot.xlabel('X')
pyplot.ylabel('Y');
# You can see that two distinct pressure zones are forming and that the spiral
pattern expected from lid-driven cavity flow is beginning to form. Experiment
with different values of `nt` to see how long the system takes to stabilize.
# In[6]:
u = numpy.zeros((ny, nx))
v = numpy.zeros((ny, nx))
p = numpy.zeros((ny, nx))
b = numpy.zeros((ny, nx))
nt = 700
u, v, p = cavityFlow(nt, u, v, dt, dx, dy, p, rho, nu)
fig = pyplot.figure(figsize=(11,7), dpi=100)
pyplot.contourf(X,Y,p,alpha=0.5)
pyplot.colorbar()
pyplot.contour(X,Y,p)
pyplot.quiver(X[::2,::2],Y[::2,::2],u[::2,::2],v[::2,::2])
pyplot.xlabel('X')
pyplot.ylabel('Y');
pyplot.show
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311757#comment-15311757 ] Edward J. Yoon commented on HAMA-992: - Hi, if possible please attach your bsp python code here. And, I mean, you should update the BinaryProtocol.py and copy to hdfs again (instead of change the hama project). > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > 16/06/01 14:48:52 ERROR bsp.BSPTask: Error running bsp setup and bsp function. > java.io.IOException: Stream closed > at > java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) > at java.io.OutputStream.write(OutputStream.java:116) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.Buffere
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311744#comment-15311744 ] Chaitanya commented on HAMA-992: Hello Edward, I have made the class file and and updated the jar file. I tried running it once. It doesn't start the job and shows the following error:- Error: Could not find or load main class org.apache.hama.pipes.Submitter > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > 16/06/01 14:48:52 ERROR bsp.BSPTask: Error running bsp setup and bsp function. > java.io.IOException: Stream closed > at > java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) > at java.io.OutputStream.write(OutputStream.java:116) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOu
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310890#comment-15310890 ] Thomas Jungblut commented on HAMA-992: -- This actually looks like a duplicate of #HAMA-978. [~udanax] is the c++ code also outdated? [~chaitanya2692] can you please attach your bsp python code so I can debug? Thanks! > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > 16/06/01 14:48:52 ERROR bsp.BSPTask: Error running bsp setup and bsp function. > java.io.IOException: Stream closed > at > java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) > at java.io.OutputStream.write(OutputStream.java:116) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.B
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310141#comment-15310141 ] Edward J. Yoon commented on HAMA-992: - +1. Thanks for your opinion and action. > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > 16/06/01 14:48:52 ERROR bsp.BSPTask: Error running bsp setup and bsp function. > java.io.IOException: Stream closed > at > java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) > at java.io.OutputStream.write(OutputStream.java:116) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141) >
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310140#comment-15310140 ] Edward J. Yoon commented on HAMA-992: - +1. Thanks for your opinion and action. > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > 16/06/01 14:48:52 ERROR bsp.BSPTask: Error running bsp setup and bsp function. > java.io.IOException: Stream closed > at > java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) > at java.io.OutputStream.write(OutputStream.java:116) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141) >
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310142#comment-15310142 ] Edward J. Yoon commented on HAMA-992: - +1. Thanks for your opinion and action. > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > 16/06/01 14:48:52 ERROR bsp.BSPTask: Error running bsp setup and bsp function. > java.io.IOException: Stream closed > at > java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) > at java.io.OutputStream.write(OutputStream.java:116) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141) >
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310113#comment-15310113 ] Thomas Jungblut commented on HAMA-992: -- Let's move it to the Hama repo. I don't think it is a great experience for people to use a broken integration. WIthin Hama we can make sure that this is tested with an integration test. I'll try to work on this soon. > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > 16/06/01 14:48:52 ERROR bsp.BSPTask: Error running bsp setup and bsp function. > java.io.IOException: Stream closed > at > java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) > at java.io.OutputStream.write(OutputStream.java:116) > at > java.io.BufferedOutputStream.flushBuffer(Buffer
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310101#comment-15310101 ] Edward J. Yoon commented on HAMA-992: - Thomas, I'm fine either way but adding to hama repo would be good idea if we can put some efforts to increase python supports and attract python users. > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > 16/06/01 14:48:52 ERROR bsp.BSPTask: Error running bsp setup and bsp function. > java.io.IOException: Stream closed > at > java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) > at java.io.OutputStream.write(OutputStream.java:116) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.BufferedOutputStream.flush(B
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310089#comment-15310089 ] Thomas Jungblut commented on HAMA-992: -- Edward, should we update the code on github? Or should we transfer it to the Hama repo? > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > 16/06/01 14:48:52 ERROR bsp.BSPTask: Error running bsp setup and bsp function. > java.io.IOException: Stream closed > at > java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) > at java.io.OutputStream.write(OutputStream.java:116) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) > at java.io.BufferedOutpu
[jira] [Commented] (HAMA-992) Hama streaming
[ https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310080#comment-15310080 ] Edward J. Yoon commented on HAMA-992: - Thanks for the report. BTW, Thomas's code is quite old - https://github.com/thomasjungblut/HamaStreaming Could you please retry after updating binary protocol code within BinaryProtocol.py [1] according to 0.7 version MessageType [2]? 1. https://github.com/thomasjungblut/HamaStreaming/blob/master/BinaryProtocol.py 2. https://github.com/apache/hama/blob/0.7/core/src/main/java/org/apache/hama/pipes/protocol/MessageType.java > Hama streaming > -- > > Key: HAMA-992 > URL: https://issues.apache.org/jira/browse/HAMA-992 > Project: Hama > Issue Type: Question > Components: bsp core, pipes >Affects Versions: 0.7.1 > Environment: RASPBIAN JESSIE > Full desktop image based on Debian Jessie >Reporter: Chaitanya > Labels: features, github-import, newbie > > Hello all, > I am trying to implement apache hama on Raspberry pi model 3 to establish a > distributed computing platform for scientific computation. I am trying to run > hama streaming over hadoop on a single namenode but I am facing a bit of a > difficulty in streaming my python code. I have downloaded the hama streaming > repository from :- > https://github.com/thomasjungblut/HamaStreaming > I ran the examples and also HelloWorldBSP.py on Hama and they work well. But > as soon as I switch to running my python code, the job fails. > I am trying to run the code with the following command:- > hama pipes -streaming true -bspTasks 1 -interpreter python -output > /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs > python.py > Below is the log file for your reference. I hope you can find time to help me > in this minor project:- > 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001 > 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting > 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting > 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer > address:localhost port:61001 > 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is > deprecated. Instead, use mapreduce.job.cache.local.files > 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to > Zookeeper! At localhost/127.0.0.1:61001 > java.lang.NumberFormatException: For input string: "Traceback (most recent > call last):" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:580) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) > 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad > command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > java.util.concurrent.BrokenBarrierException > at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) > at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) > at > org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) > at > org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > at > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255) > Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: > java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182) > Caused by: java.lang.Exception: Bad command code: -2 > at > org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174) > 16/06/01 14:48:52 ERROR bsp.BSPTask: Error running bsp setup and bsp function. > java.io.IOException: Stream closed >
