Found a solution:
export CL_LOG_ERRORS="stdout"
That turns on error logging. This produced the following:
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pyopencl-2011.2-py2.7-macosx-10.6-x86_64.egg/pyopencl/reduction.py:178:
UserWarning: Reduction might be unnecessarily slow: can't query SIMD group size
warn("Reduction might be unnecessarily slow: "
[CL_OUT_OF_RESOURCES] : OpenCL Error : Failed to wait for events! Event 0 in
waitlist failed. Out of resources
Break on OpenCLErrorBreak to debug.
[CL_INVALID_COMMAND_QUEUE] : OpenCL Fatal Error : Finish caused an error that
invalidated the queue (0x1048105a0). This may be due to a resource allocation
failure at execution time.
Break on OpenCLFatalBreak to debug.
Traceback (most recent call last):
File "visual_cortex.py", line 387, in <module>
do_facial_rec()
File "visual_cortex.py", line 352, in do_facial_rec
train_network(network,200,train_path,test_path,results_path,answer_func)
File "visual_cortex.py", line 252, in train_network
result,guess,cycles_uc,cycles_c,wt_change = network.train(folder_path +
path,answer,randomize=False) # TRANSLATION DISABLED
File "visual_cortex.py", line 100, in train
cycles_unclamp,wt_change_uc = self.settle()
File "/Users/lewisanderson/Dropbox/Cortex/code/cortex.py", line 131, in settle
change = self.update_activations()
File "/Users/lewisanderson/Dropbox/Cortex/code/cortex.py", line 169, in
update_activations
changes = [x.calc_activation() for x in self.layers]
File "/Users/lewisanderson/Dropbox/Cortex/code/cortex.py", line 421, in
calc_activation
inputs = [x.get_net_i() for x in self.connections]
File "/Users/lewisanderson/Dropbox/Cortex/code/cortex.py", line 988, in
get_net_i
queue.finish()
pyopencl.LogicError: clFinish failed: invalid command queue
So, now I can actually fix it!
Lewis
On Dec 23, 2011, at 11:41 PM, Lewis Anderson wrote:
> Hello,
>
> I am having trouble with some PyOpenCL code. When I run it, it will execute a
> few commands properly, then fail with "invalid command queue". I searched for
> this on Google, and found a recommendation that I add some error handlers
> (http://www.khronos.org/message_boards/viewtopic.php?f=28&t=2061) to the
> context.
>
> How do I do get this additional error information in PyOpenCL? Or is there
> some other way to figure out what is causing the command queue to become
> invalid?
>
>
> Stack Trace:
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "test.py", line 59, in test_visual_cortex
> net.update_weights()
> File "cortex.py", line 182, in update_weights
> return sum([x.do_learning() for x in self.layers])
> File "cortex.py", line 570, in do_learning
> total_change = sum(x.do_XCAL() for x in self.connections)
> File "cortex.py", line 570, in <genexpr>
> total_change = sum(x.do_XCAL() for x in self.connections)
> File "cortex.py", line 1015, in do_XCAL
>
> cl_do_xcal(self.source_activations(),self.source_idxs_cl,self.dest.act_cl,self.dest.act_l_cl,self.weights_cl,eta).wait()
> File "visual_cortex.py", line 215, in source_activations
> c.queue.finish()
> pyopencl.LogicError: clFinish failed: invalid command queue
>
>
> Relevant code:
> test_visual_cortex() is the main function called. It builds a neural network,
> then calls reset_act(), update_activations(), and update_weights(). Each of
> these three functions has PyOpenCL calls, and they all appear to function
> properly until the crash happens. This code previously used Numpy for all
> array operations, and worked well. I am moving to OpenCL in order to improve
> performance.
>
> def test_visual_cortex():
> net = v.build_visual_cortex_3('../data/objrec
> tests/opencl-test/',scale_gabor=True)
> input = net.get_layer('Gabor')
> img_path = '../facial/train/018_a2.pgm'
> input.load_image(img_path,randomize=False)
>
> [x.reset_act() for x in net.layers]
> net.update_activations()
> net.update_weights()
>
> def update_activations(self):
> ''' update_activations(): update activations of all layers in
> network
> inputs
> none
> outputs
> maximum change for any layer
> effects
> updates the activations for every layer in the network
> via the
> calc_activation() method
> '''
> changes = [x.calc_activation() for x in self.layers]
> return max(changes)
>
> def update_weights(self):
> ''' update_activations(): update activations of all layers in
> network
> inputs
> none
> outputs
> sum of weight change
> effects
> updates the weights for every layer in the network via
> the do_learning()
> method
> '''
> return sum([x.do_learning() for x in self.layers])
>
> def calc_activation(self):
> ''' calc_activation(): calculate activation of each neuron in
> this layer
> algorithm
> calculate strength of each connection
> sum for each neuron in this layer
> do sigma function
> set activations based on kWTA
> output
> sum of absolute value of change in activation this time
> effects
> updates self.activations
> '''
> if len(self.connections) == 0:
> return 0
> if not self.clamped:
> if USE_GPU:
> inputs = [x.get_net_i() for x in
> self.connections]
> net_i = cl_a.zeros_like(self.act_cl)
> queue.finish()
> for input in inputs:
> # print input
> net_i = net_i + input
> queue.finish()
>
> # debug
> print "calc_activation in",self.name
> print "queue:",queue
> device =
> queue.get_info(cl.command_queue_info.DEVICE)
> print "DEVICE:",device
> print
> "MEM_SIZE:",device.get_info(cl.device_info.GLOBAL_MEM_SIZE)
> print
> "AVAILABLE:",device.get_info(cl.device_info.AVAILABLE)
> # end debug
>
> net_i = net_i/np.float32(len(self.connections))
> queue.finish()
>
> thresh = self.kWTA(net_i)
> y = net_i - np.float32(thresh)
> queue.finish()
> y = cl_a.maximum(y,cl_a.zeros_like(y))
> queue.finish()
> new_act = self.amplify_activation(y)
> old_act = self.act_cl
> #self.act_cl = old_act + dt*(new_act - old_act)
> temp_act = cl_a.zeros_like(old_act)
> queue.finish()
>
> cl_calc_activation(old_act,dt,new_act,temp_act).wait()
> self.act_cl = temp_act
>
>
> def do_XCAL(self):
> ''' do_XCAL(): perform XCAL learning algorithm on all connections
> algorithm
> get xy averages
> calculate necessary change in weight using xcal function
> modify weights based on learning rate
> update xy_m averages
> output
> None **not done anymore: float describing total change in
> weights**
> effects
> weights and xy_m changed for each connection
> '''
> total_change = 0.0
> if not self.fixed_weights:
>
> if USE_GPU:
>
> cl_do_xcal(self.source_activations(),self.source_idxs_cl,self.dest.act_cl,self.dest.act_l_cl,self.weights_cl,eta).wait()
>
> def source_activations(self):
> ''' source_activations(): get the proper channel of gabor_data
> returns
> numpy array for source activations
> '''
> if c.USE_GPU:
> height = self.source.gabor_data.shape[0]
> orien_idx = self.orientation_idx
> num_oriens = 4
> idxs =
> c.cl_a.arange(c.queue,orien_idx,orien_idx+height*num_oriens,num_oriens,dtype=np.int32)
> print c.queue
> c.queue.finish()
>
>
>
>
> Thanks,
> Lewis Anderson
_______________________________________________
PyOpenCL mailing list
[email protected]
http://lists.tiker.net/listinfo/pyopencl
