[
https://issues.apache.org/jira/browse/HAMA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311760#comment-15311760
]
Chaitanya commented on HAMA-992:
--------------------------------
Hello Thomas,
I am running the following python code meant for CFD simulations which runs
well outside the hama environment.
# In[1]:
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from matplotlib import pyplot
import numpy
#get_ipython().magic(u'matplotlib inline')
nx = 41
ny = 41
nt = 500
nit=50
c = 1
dx = 2./(nx-1)
dy = 2./(ny-1)
x = numpy.linspace(0,2,nx)
y = numpy.linspace(0,2,ny)
X,Y = numpy.meshgrid(x,y)
rho = 1
nu = .1
dt = .001
u = numpy.zeros((ny, nx))
v = numpy.zeros((ny, nx))
p = numpy.zeros((ny, nx))
b = numpy.zeros((ny, nx))
# The pressure Poisson equation that's written above can be hard to write out
without typos. The function `buildUpB` below represents the contents of the
square brackets, so that the entirety of the PPE is slightly more manageable.
# In[2]:
def buildUpB(b, rho, dt, u, v, dx, dy):
b[1:-1,1:-1]=rho*(1/dt*((u[1:-1,2:]-u[1:-1,0:-2])/(2*dx)+(v[2:,1:-1]-v[0:-2,1:-1])/(2*dy))-((u[1:-1,2:]-u[1:-1,0:-2])/(2*dx))**2-2*((u[2:,1:-1]-u[0:-2,1:-1])/(2*dy)*(v[1:-1,2:]-v[1:-1,0:-2])/(2*dx))-
((v[2:,1:-1]-v[0:-2,1:-1])/(2*dy))**2)
return b
# The function `presPoisson` is also defined to help segregate the different
rounds of calculations. Note the presence of the pseudo-time variable `nit`.
This sub-iteration in the Poisson calculation helps ensure a divergence-free
field.
# In[3]:
def presPoisson(p, dx, dy, b):
pn = numpy.empty_like(p)
pn = p.copy()
for q in range(nit):
pn = p.copy()
p[1:-1,1:-1] =
((pn[1:-1,2:]+pn[1:-1,0:-2])*dy**2+(pn[2:,1:-1]+pn[0:-2,1:-1])*dx**2)/
(2*(dx**2+dy**2)) -
dx**2*dy**2/(2*(dx**2+dy**2))*b[1:-1,1:-1]
p[:,-1] =p[:,-2] ##dp/dy = 0 at x = 2
p[0,:] = p[1,:] ##dp/dy = 0 at y = 0
p[:,0]=p[:,1] ##dp/dx = 0 at x = 0
p[-1,:]=0 ##p = 0 at y = 2
return p
# Finally, the rest of the cavity flow equations are wrapped inside the
function `cavityFlow`, allowing us to easily plot the results of the cavity
flow solver for different lengths of time.
# In[4]:
def cavityFlow(nt, u, v, dt, dx, dy, p, rho, nu):
un = numpy.empty_like(u)
vn = numpy.empty_like(v)
b = numpy.zeros((ny, nx))
for n in range(nt):
un = u.copy()
vn = v.copy()
b = buildUpB(b, rho, dt, u, v, dx, dy)
p = presPoisson(p, dx, dy, b)
u[1:-1,1:-1] = un[1:-1,1:-1]-
un[1:-1,1:-1]*dt/dx*(un[1:-1,1:-1]-un[1:-1,0:-2])-
vn[1:-1,1:-1]*dt/dy*(un[1:-1,1:-1]-un[0:-2,1:-1])-
dt/(2*rho*dx)*(p[1:-1,2:]-p[1:-1,0:-2])+
nu*(dt/dx**2*(un[1:-1,2:]-2*un[1:-1,1:-1]+un[1:-1,0:-2])+
dt/dy**2*(un[2:,1:-1]-2*un[1:-1,1:-1]+un[0:-2,1:-1]))
v[1:-1,1:-1] = vn[1:-1,1:-1]-
un[1:-1,1:-1]*dt/dx*(vn[1:-1,1:-1]-vn[1:-1,0:-2])-
vn[1:-1,1:-1]*dt/dy*(vn[1:-1,1:-1]-vn[0:-2,1:-1])-
dt/(2*rho*dy)*(p[2:,1:-1]-p[0:-2,1:-1])+
nu*(dt/dx**2*(vn[1:-1,2:]-2*vn[1:-1,1:-1]+vn[1:-1,0:-2])+
(dt/dy**2*(vn[2:,1:-1]-2*vn[1:-1,1:-1]+vn[0:-2,1:-1])))
u[0,:] = 0
u[:,0] = 0
u[:,-1] = 0
u[-1,:] = 1 #set velocity on cavity lid equal to 1
v[0,:] = 0
v[-1,:]=0
v[:,0] = 0
v[:,-1] = 0
return u, v, p
# Let's start with `nt = 100` and see what the solver gives us:
# In[5]:
u = numpy.zeros((ny, nx))
v = numpy.zeros((ny, nx))
p = numpy.zeros((ny, nx))
b = numpy.zeros((ny, nx))
nt = 100
u, v, p = cavityFlow(nt, u, v, dt, dx, dy, p, rho, nu)
fig = pyplot.figure(figsize=(11,7), dpi=100)
pyplot.contourf(X,Y,p,alpha=0.5) ###plnttong the pressure field as a contour
pyplot.colorbar()
pyplot.contour(X,Y,p) ###plotting the pressure field outlines
pyplot.quiver(X[::2,::2],Y[::2,::2],u[::2,::2],v[::2,::2]) ##plotting velocity
pyplot.xlabel('X')
pyplot.ylabel('Y');
# You can see that two distinct pressure zones are forming and that the spiral
pattern expected from lid-driven cavity flow is beginning to form. Experiment
with different values of `nt` to see how long the system takes to stabilize.
# In[6]:
u = numpy.zeros((ny, nx))
v = numpy.zeros((ny, nx))
p = numpy.zeros((ny, nx))
b = numpy.zeros((ny, nx))
nt = 700
u, v, p = cavityFlow(nt, u, v, dt, dx, dy, p, rho, nu)
fig = pyplot.figure(figsize=(11,7), dpi=100)
pyplot.contourf(X,Y,p,alpha=0.5)
pyplot.colorbar()
pyplot.contour(X,Y,p)
pyplot.quiver(X[::2,::2],Y[::2,::2],u[::2,::2],v[::2,::2])
pyplot.xlabel('X')
pyplot.ylabel('Y');
pyplot.show()
> Hama streaming
> --------------
>
> Key: HAMA-992
> URL: https://issues.apache.org/jira/browse/HAMA-992
> Project: Hama
> Issue Type: Question
> Components: bsp core, pipes
> Affects Versions: 0.7.1
> Environment: RASPBIAN JESSIE
> Full desktop image based on Debian Jessie
> Reporter: Chaitanya
> Labels: features, github-import, newbie
>
> Hello all,
> I am trying to implement apache hama on Raspberry pi model 3 to establish a
> distributed computing platform for scientific computation. I am trying to run
> hama streaming over hadoop on a single namenode but I am facing a bit of a
> difficulty in streaming my python code. I have downloaded the hama streaming
> repository from :-
> https://github.com/thomasjungblut/HamaStreaming
> I ran the examples and also HelloWorldBSP.py on Hama and they work well. But
> as soon as I switch to running my python code, the job fails.
> I am trying to run the code with the following command:-
> hama pipes -streaming true -bspTasks 1 -interpreter python -output
> /tmp/pystream-out_2/ -program /tmp/PyStreaming/BSPRunner.py -programArgs
> python.py
> Below is the log file for your reference. I hope you can find time to help me
> in this minor project:-
> 16/06/01 14:48:46 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 16/06/01 14:48:49 INFO ipc.Server: Starting Socket Reader #1 for port 61001
> 16/06/01 14:48:49 INFO ipc.Server: IPC Server listener on 61001: starting
> 16/06/01 14:48:49 INFO ipc.Server: IPC Server Responder: starting
> 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 0 on 61001: starting
> 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 1 on 61001: starting
> 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 3 on 61001: starting
> 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 2 on 61001: starting
> 16/06/01 14:48:49 INFO ipc.Server: IPC Server handler 4 on 61001: starting
> 16/06/01 14:48:49 INFO message.HamaMessageManagerImpl: BSPPeer
> address:localhost port:61001
> 16/06/01 14:48:51 INFO Configuration.deprecation: mapred.cache.localFiles is
> deprecated. Instead, use mapreduce.job.cache.local.files
> 16/06/01 14:48:51 INFO sync.ZKSyncClient: Initializing ZK Sync Client
> 16/06/01 14:48:51 INFO sync.ZooKeeperSyncClientImpl: Start connecting to
> Zookeeper! At localhost/127.0.0.1:61001
> java.lang.NumberFormatException: For input string: "Traceback (most recent
> call last):"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:580)
> at java.lang.Integer.parseInt(Integer.java:615)
> at
> org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174)
> at
> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106)
> 16/06/01 14:48:52 ERROR protocol.UplinkReader: java.lang.Exception: Bad
> command code: -2
> at
> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174)
> java.util.concurrent.BrokenBarrierException
> at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
> at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
> at
> org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223)
> at
> org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293)
> at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43)
> at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170)
> at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144)
> at
> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255)
> Exception in thread "pipe-uplink-handler" java.lang.RuntimeException:
> java.lang.Exception: Bad command code: -2
> at
> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:182)
> Caused by: java.lang.Exception: Bad command code: -2
> at
> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:174)
> 16/06/01 14:48:52 ERROR bsp.BSPTask: Error running bsp setup and bsp function.
> java.io.IOException: Stream closed
> at
> java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433)
> at java.io.OutputStream.write(OutputStream.java:116)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141)
> at java.io.DataOutputStream.flush(DataOutputStream.java:123)
> at
> org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287)
> at
> org.apache.hama.pipes.protocol.StreamingProtocol.runSetup(StreamingProtocol.java:248)
> at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:45)
> at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170)
> at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144)
> at
> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255)
> 16/06/01 14:48:52 ERROR bsp.BSPTask: Error cleaning up after bsp executed.
> java.io.IOException: Stream closed
> at
> java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433)
> at java.io.OutputStream.write(OutputStream.java:116)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141)
> at java.io.DataOutputStream.flush(DataOutputStream.java:123)
> at
> org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287)
> at
> org.apache.hama.pipes.protocol.StreamingProtocol.runCleanup(StreamingProtocol.java:271)
> at org.apache.hama.pipes.PipesBSP.cleanup(PipesBSP.java:95)
> at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:177)
> at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144)
> at
> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255)
> 16/06/01 14:48:53 INFO ipc.Server: Stopping server on 61001
> 16/06/01 14:48:53 INFO ipc.Server: IPC Server handler 1 on 61001: exiting
> 16/06/01 14:48:53 INFO ipc.Server: Stopping IPC Server listener on 61001
> 16/06/01 14:48:53 INFO ipc.Server: IPC Server handler 0 on 61001: exiting
> 16/06/01 14:48:53 INFO ipc.Server: IPC Server handler 3 on 61001: exiting
> 16/06/01 14:48:53 INFO ipc.Server: IPC Server handler 4 on 61001: exiting
> 16/06/01 14:48:53 INFO ipc.Server: IPC Server handler 2 on 61001: exiting
> 16/06/01 14:48:53 INFO ipc.Server: Stopping IPC Server Responder
> 16/06/01 14:48:53 ERROR bsp.BSPTask: Shutting down ping service.
> 16/06/01 14:48:53 FATAL bsp.GroomServer: Error running child
> java.io.IOException: Stream closed
> at
> java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433)
> at java.io.OutputStream.write(OutputStream.java:116)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141)
> at java.io.DataOutputStream.flush(DataOutputStream.java:123)
> at
> org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287)
> at
> org.apache.hama.pipes.protocol.StreamingProtocol.runSetup(StreamingProtocol.java:248)
> at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:45)
> at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170)
> at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144)
> at
> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255)
> java.io.IOException: Stream closed
> at
> java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433)
> at java.io.OutputStream.write(OutputStream.java:116)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141)
> at java.io.DataOutputStream.flush(DataOutputStream.java:123)
> at
> org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287)
> at
> org.apache.hama.pipes.protocol.StreamingProtocol.runSetup(StreamingProtocol.java:248)
> at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:45)
> at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170)
> at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144)
> at
> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
