date:20170329

Re: [theano-users] Straight-through gradient for stochastic node in theano

2017-03-29 Thread Pascal Lamblin

You can try OpFromGraph with inline=True, and specify
override_gradients=... with the right expression.
This is still experimental.

On Wed, Mar 29, 2017, nokunok...@gmail.com wrote:
> Hi guys!
> 
> I recently started using theano and am struggling to implement custom 
> gradient for stochastic node. Can anyone help me?
> 
> What I want is an op that produces one hot vector whose hot element is 
> sampled from softmax distribution.
> The op is not differentiable, but I want to "fake" as if its gradient is 
> softmax's one ("straight through estimator").
> Below is the minimum code that perform forward path, which raises 
> DisconnectedInputError due to missing gradient.
> 
> 
> import theano
> 
> import theano.tensor as T
> 
> import numpy as np
> 
> 
> logits_values = np.random.uniform(-1, 1, size=3)
> 
> logits = theano.shared(logits_values, 'logits')
> 
> probabilities = T.nnet.softmax(logits)
> 
> print('probabilities', probabilities.eval())
> 
> # result: probabilities [[ 0.55155489  0.2907730.15767211]]
> 
> 
> random_streams = T.shared_randomstreams.RandomStreams()
> 
> index = random_streams.choice(size=(1,), a=3, p=probabilities[0])
> 
> samples = T.extra_ops.to_one_hot(index, logits.shape[-1])
> 
> print('samples', samples.eval())
> 
> # result: samples [[ 1.  0.  0.]]
> 
> 
> # We want to use gradient of probabilities instead of samples!
> 
> samples_grad = T.grad(samples[0][0], logits)
> 
> # result: raise DisconnectedInputError
> 
> 
> The node is not the final layer, so I can't use categorical cross entropy 
> loss for training it.
> 
> I am trying to implement custom op (see attached stochastic_softmax.py), but 
> it is not working in practice.
> 
> Since I have working expression for forward path, can I simply override 
> gradient of existing expression?
> 
> -- 
> 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "theano-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to theano-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

> import numpy as np
> import theano
> import theano.tensor as T
> 
> 
> class StochasticSoftmax(theano.Op):
> def __init__(self, random_state=np.random.RandomState()):
> self.random_state = random_state
> 
> def make_node(self, x):
> x = T.as_tensor_variable(x)
> return theano.Apply(self, [x], [x.type()])
> 
> def perform(self, node, inputs, output_storage):
> # Gumbel-max trick
> x, = inputs
> z = self.random_state.gumbel(loc=0, scale=1, size=x.shape)
> indices = (x + z).argmax(axis=-1)
> y = np.eye(x.shape[-1], dtype=np.float32)[indices]
> output_storage[0][0] = y
> 
> def grad(self, inp, grads):
> x, = inp
> g_sm, = grads
> 
> sm = T.nnet.softmax(x)
> return [T.nnet.softmax_grad(g_sm, sm)]
> 
> def infer_shape(self, node, i0_shapes):
> return i0_shapes


-- 
Pascal

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [theano-users] Does the Theano 0.9(the lastest version) support python 2.7?

2017-03-29 Thread Pascal Lamblin

Python 2.7 is still supported.

On Wed, Mar 29, 2017, xqg mickle wrote:
> Does the new version only support python>=3.3?
> 
> -- 
> 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "theano-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to theano-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.


-- 
Pascal

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Straight-through gradient for stochastic node in theano

2017-03-29 Thread nokunokuno

Hi guys!

I recently started using theano and am struggling to implement custom 
gradient for stochastic node. Can anyone help me?

What I want is an op that produces one hot vector whose hot element is 
sampled from softmax distribution.
The op is not differentiable, but I want to "fake" as if its gradient is 
softmax's one ("straight through estimator").
Below is the minimum code that perform forward path, which raises 
DisconnectedInputError due to missing gradient.


import theano

import theano.tensor as T

import numpy as np


logits_values = np.random.uniform(-1, 1, size=3)

logits = theano.shared(logits_values, 'logits')

probabilities = T.nnet.softmax(logits)

print('probabilities', probabilities.eval())

# result: probabilities [[ 0.55155489  0.2907730.15767211]]


random_streams = T.shared_randomstreams.RandomStreams()

index = random_streams.choice(size=(1,), a=3, p=probabilities[0])

samples = T.extra_ops.to_one_hot(index, logits.shape[-1])

print('samples', samples.eval())

# result: samples [[ 1.  0.  0.]]


# We want to use gradient of probabilities instead of samples!

samples_grad = T.grad(samples[0][0], logits)

# result: raise DisconnectedInputError


The node is not the final layer, so I can't use categorical cross entropy loss 
for training it.

I am trying to implement custom op (see attached stochastic_softmax.py), but it 
is not working in practice.

Since I have working expression for forward path, can I simply override 
gradient of existing expression?

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
import numpy as np
import theano
import theano.tensor as T


class StochasticSoftmax(theano.Op):
def __init__(self, random_state=np.random.RandomState()):
self.random_state = random_state

def make_node(self, x):
x = T.as_tensor_variable(x)
return theano.Apply(self, [x], [x.type()])

def perform(self, node, inputs, output_storage):
# Gumbel-max trick
x, = inputs
z = self.random_state.gumbel(loc=0, scale=1, size=x.shape)
indices = (x + z).argmax(axis=-1)
y = np.eye(x.shape[-1], dtype=np.float32)[indices]
output_storage[0][0] = y

def grad(self, inp, grads):
x, = inp
g_sm, = grads

sm = T.nnet.softmax(x)
return [T.nnet.softmax_grad(g_sm, sm)]

def infer_shape(self, node, i0_shapes):
return i0_shapes

[theano-users] Does the Theano 0.9(the lastest version) support python 2.7?

2017-03-29 Thread xqg mickle

Does the new version only support python>=3.3?

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [theano-users] Re: AlexNet_theano_generate_toy_data.sh problems

2017-03-29 Thread Goffredo Giordano

I resolved this issue that was due to the ImageNet subset downloaded. The 
program doesn't recognize the wnid because it cannot find the folders (like 
n01440764, n01443537, ...) in the training folder defined in paths.yaml. 
However when I tryed the train.py it gives me these errors:

C:\deep_learning\alexnet>python train.py
Traceback (most recent call last):
  File "train.py", line 180, in 
config = dict(list(config.items()) + list(yaml.load(f).items()))
  File 
"C:\deep_learning\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\yaml\__init__.py",
 
line 72, in load
return loader.get_single_data()
  File 
"C:\deep_learning\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\yaml\constructor.py",
 
line 35, in get_single_data
node = self.get_single_node()
  File 
"C:\deep_learning\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\yaml\composer.py",
 
line 36, in get_single_node
document = self.compose_document()
  File 
"C:\deep_learning\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\yaml\composer.py",
 
line 55, in compose_document
node = self.compose_node(None, None)
  File 
"C:\deep_learning\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\yaml\composer.py",
 
line 84, in compose_node
node = self.compose_mapping_node(anchor)
  File 
"C:\deep_learning\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\yaml\composer.py",
 
line 127, in compose_mapping_node
while not self.check_event(MappingEndEvent):
  File 
"C:\deep_learning\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\yaml\parser.py",
 
line 98, in check_event
self.current_event = self.state()
  File 
"C:\deep_learning\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\yaml\parser.py",
 
line 428, in parse_block_mapping_key
if self.check_token(KeyToken):
  File 
"C:\deep_learning\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\yaml\scanner.py",
 
line 116, in check_token
self.fetch_more_tokens()
  File 
"C:\deep_learning\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\yaml\scanner.py",
 
line 220, in fetch_more_tokens
return self.fetch_value()
  File 
"C:\deep_learning\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\yaml\scanner.py",
 
line 580, in fetch_value
self.get_mark())
yaml.scanner.ScannerError: mapping values are not allowed here
  in "spec_1gpu.yaml", line 9, column 92

Please anyone could help me again?
Thanks, 
Goffredo



Il giorno lunedì 20 marzo 2017 16:01:40 UTC+1, Goffredo Giordano ha scritto:
>
> Hi guys, I would thank you because I resolved the previous errors: the 
> errors related to scipy probably were attributable to the meta_clsloc.mat 
> file. I have downloaded another file from 
> https://github.com/heuritech/convnets-keras/blob/a06ebbbca392a7eedc8a2e71ddcc8796e086c35a/convnetskeras/data/meta_clsloc.mat
>  
> and it fixed them. However I found another error, and I don't know if it is 
> related to the Python 3.4 version. Anyone could help me again?
>
> $ sh generate_toy_data.sh
> generating toy dataset ...
> Traceback (most recent call last):
>   File "make_train_val_txt.py", line 61, in 
> str(dict_orig_id_to_sorted_id[int(val_labels[ind])]) + '\n'
> KeyError: 490
>
> Goffredo
>
> Il giorno martedì 14 marzo 2017 15:42:05 UTC+1, Goffredo Giordano ha 
> scritto:
>>
>> Thanks Fred! If I will fix it, I would like to keep up you.
>>
>> Il giorno martedì 14 marzo 2017 15:17:16 UTC+1, nouiz ha scritto:
>>>
>>> I don't know scipy.io.matlab. Search for that error on the web. I can't 
>>> help with that one.
>>>
>>> Fred
>>>
>>> On Tue, Mar 14, 2017 at 9:45 AM Goffredo Giordano  
>>> wrote:
>>>
 Thank you Fred! I read that it was written for Python 2.7. According to 
 you is so complex to convert it to Python 3.4? I followed your advices and 
 the errors are these ones:


 $ sh generate_toy_data.sh
 ciao
 generating toy dataset ...
 make_hkl.py:72: VisibleDeprecationWarning: using a non-integer number 
 instead of an integer will result in an error in the future
   hkl.dump(img_batch[:, :, :, :half_size],
 make_hkl.py:76: VisibleDeprecationWarning: using a non-integer number 
 instead of an integer will result in an error in the future
   hkl.dump(img_batch[:, :, :, half_size:],
 Traceback (most recent call last):
   File "make_train_val_txt.py", line 26, in 
 synsets = scipy.io.loadmat(meta_clsloc_mat)['synsets'][0]
   File 
 "C:\deep_learning\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\scipy\io\matlab\mio.py",
  
 line 136, in loadmat
 matfile_dict = MR.get_variables(variable_names)
   File 
 "C:\deep_learning\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\scipy\io\matlab\mio5.py",
  
 line 272, in get_variables
 hdr, next_position =

Re: [theano-users] Re: about conv2d on CPU

2017-03-29 Thread C. Ng

I tried both 0.8 and 0.9 releases.

Thanks for the suggestion.


On Thursday, March 23, 2017 at 8:43:53 PM UTC+8, nouiz wrote:
>
> True as of Theano 0.9. 0.8 don't include a cpu corrmm from memory.
>
> If you are really intersted by CPU speed, Intel have a fork of Theano that 
> they have optimized on CPU:
>
> http://github.com/intel/Theano
>
> It will probably be merged into the master of Theano at some point, but no 
> timeline.
>
> Fred
>
> On Tue, Mar 21, 2017 at 9:18 PM Jesse Livezey  > wrote:
>
>> That is correct as of theano 0.8 (I think).
>>
>> If you use the bleeding edge version of theano, you can let CorrMM use 
>> openmp to parallelize across batches. If you have more than 2 cores, this 
>> should give additional speedup. GPUs are going to be much faster than CPUs 
>> generally, if you have large batches and lots of cores, CPUs can catch up a 
>> bit, but GPUs are still going to be faster.
>>
>>
>> On Monday, March 20, 2017 at 11:59:52 PM UTC-7, C. Ng wrote:
>>>
>>> Hi,
>>>
>>> Just want to confirm that theano.tensor.nnet.conv2d uses CorrMM (not the 
>>> legacy convolution) by default in CPU mode ?
>>>
>>> I was hoping that forward prop (doing inference only, no training) using 
>>> CPU for convolution might be as fast as GPU (using CorrMM), given my batch 
>>> size is only 10. But using GPU is still quite a bit faster.  
>>>
>>>
>>>
>>>
>>>  
>>>
>> -- 
>>
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "theano-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to theano-users...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [theano-users] Straight-through gradient for stochastic node in theano

Re: [theano-users] Does the Theano 0.9(the lastest version) support python 2.7?

[theano-users] Straight-through gradient for stochastic node in theano

[theano-users] Does the Theano 0.9(the lastest version) support python 2.7?

Re: [theano-users] Re: AlexNet_theano_generate_toy_data.sh problems

Re: [theano-users] Re: about conv2d on CPU

6 matches

Site Navigation

Mail list logo

Footer information