Re: [Numpy-discussion] how to name "contagious" keyword in np.ma.convolve
On Tue, Oct 18, 2016 at 4:18 PM, Allan Haldanewrote: > As for whether it should default to "True" or "False", the arguments I > see are: > > * False, because that is the way most functions like `np.ma.sum` >already work, as well as matlab and octave's similar "nanconv". > > * True, because its effects are more visible and might lead to less >surprises. The "False" case seems like it is often not what the user >intended. Eg, it affects the overall normalization of normalized >kernels, and the choice of 0 seems arbitrary. > > If no one says anything, I'd probably go with True > I also have serious concerns about if it ever actually makes sense to use `propagate_mask=False`. So, I think it's definitely appropriate to default to `propagate_mask=True`. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to name "contagious" keyword in np.ma.convolve
On 10/17/2016 01:01 PM, Pierre Haessig wrote: > Le 16/10/2016 à 11:52, Hanno Klemm a écrit : >> When I have similar situations, I usually interpolate between the valid >> values. I assume there are a lot of use cases for convolutions but I have >> difficulties imagining that ignoring a missing value and, for the purpose of >> the computation, treating it as zero is useful in many of them. > When estimating the autocorrelation of a signal, it make sense to drop > missing pairs of values. Only in this use case, it opens the question of > correcting or not correcting for the number of missing elements when > computing the mean. I don't remember what R function "acf" is doing. > > > Also, coming back to the initial question, I feel that it is necessary > that the name "mask" (or "na" or similar) appears in the parameter name. > Otherwise, people will wonder : "what on earth is contagious/being > propagated" Based on feedback so far, I think "propagate_mask" sounds like the best word to use. Let's go with that. As for whether it should default to "True" or "False", the arguments I see are: * False, because that is the way most functions like `np.ma.sum` already work, as well as matlab and octave's similar "nanconv". * True, because its effects are more visible and might lead to less surprises. The "False" case seems like it is often not what the user intended. Eg, it affects the overall normalization of normalized kernels, and the choice of 0 seems arbitrary. If no one says anything, I'd probably go with True. Allan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to name "contagious" keyword in np.ma.convolve
On 10/16/2016 05:52 AM, Hanno Klemm wrote: > > >> On 16 Oct 2016, at 03:21, Allan Haldanewrote: >> >>> On 10/14/2016 07:49 PM, Juan Nunez-Iglesias wrote: >>> +1 for propagate_mask. That is the only proposal that immediately makes >>> sense to me. "contagious" may be cute but I think approximately 0% of >>> users would guess its purpose on first use. >>> >>> Can you elaborate on what happens with the masks exactly? I didn't quite >>> get why propagate_mask=False was unintuitive. My expectation is that any >>> mask present in the input will not be set in the output, but the mask >>> will be "respected" by the function. >> >> Here's an illustration of how the PR currently works with convolve, using >> the name "propagate_mask": >> >>>>> m = np.ma.masked >>>>> a = np.ma.array([1,1,1,m,1,1,1,m,m,m,1,1,1]) >>>>> b = np.ma.array([1,1,1]) >>>>> >>>>> print np.ma.convolve(a, b, propagate_mask=True) >>[1 2 3 -- -- -- 3 -- -- -- -- -- 3 2 1] >>>>> print np.ma.convolve(a, b, propagate_mask=False) >>[1 2 3 2 2 2 3 2 1 -- 1 2 3 2 1] >> >> Allan >> > > Given this behaviour, I'm actually more concerned about the logic ma.convolve > uses in the propagate_mask=False case. It appears that the masked values are > essentially replaced by zero. Is my interpretation correct and if so does > this make sense? > I think that's right. Its usefulness wasn't obvious to me either, but googling shows that in matlab people like the file "nanconv.m" which works this way, using nans similarly to how the mask is used here. Just as convolution functions often add zero-padding around an image, here the mask behavior would allow you to have different borders, eg [m,m,m,1,1,1,1,m,m,m,m] using my notation from before. Octave's "nanconv" does this too. I still agree that in most cases people should be handling the missing values more carefully (manually) if they are doing convolutions, but this default behaviour maybe seems reasonable to me. Allan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to name "contagious" keyword in np.ma.convolve
On 10/17/2016 01:01 PM, Pierre Haessig wrote: > Hi, > > > Le 16/10/2016 à 11:52, Hanno Klemm a écrit : >> When I have similar situations, I usually interpolate between the valid >> values. I assume there are a lot of use cases for convolutions but I have >> difficulties imagining that ignoring a missing value and, for the purpose of >> the computation, treating it as zero is useful in many of them. > When estimating the autocorrelation of a signal, it make sense to drop > missing pairs of values. Only in this use case, it opens the question of > correcting or not correcting for the number of missing elements when > computing the mean. I don't remember what R function "acf" is doing. > > > Also, coming back to the initial question, I feel that it is necessary > that the name "mask" (or "na" or similar) appears in the parameter name. > Otherwise, people will wonder : "what on earth is contagious/being > propagated" > > just thinking of yet another keyword name : ignore_masked (or drop_masked) > > If I remember well, in R it is dropna. It would be nice if the boolean > switch followed the same logic. There is an old unimplemented NEP which uses similar language, like "ignorena", and np.NA. http://docs.scipy.org/doc/numpy/neps/missing-data.html But right now that isn't part of numpy, so I think it would be confusing to use that terminology. Allan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to name "contagious" keyword in np.ma.convolve
On Tue, Oct 18, 2016 at 1:30 PM,wrote: > On Tue, Oct 18, 2016 at 1:25 PM, wrote: >> On Mon, Oct 17, 2016 at 1:01 PM, Pierre Haessig >> wrote: >>> Hi, >>> >>> >>> Le 16/10/2016 à 11:52, Hanno Klemm a écrit : When I have similar situations, I usually interpolate between the valid values. I assume there are a lot of use cases for convolutions but I have difficulties imagining that ignoring a missing value and, for the purpose of the computation, treating it as zero is useful in many of them. >>> When estimating the autocorrelation of a signal, it make sense to drop >>> missing pairs of values. Only in this use case, it opens the question of >>> correcting or not correcting for the number of missing elements when >>> computing the mean. I don't remember what R function "acf" is doing. > > as aside: statsmodels has now an option for acf and similar > > missing : str > A string in ['none', 'raise', 'conservative', 'drop'] > specifying how the NaNs > are to be treated. aside to the aside: statsmodels was just catching up in this The original for masked array acf including correct counting of "valid" terms is https://github.com/pierregm/scikits.timeseries/blob/master/scikits/timeseries/lib/avcf.py (which I looked at way before statsmodels had any acf) Josef > > Josef > >>> >>> >>> Also, coming back to the initial question, I feel that it is necessary >>> that the name "mask" (or "na" or similar) appears in the parameter name. >>> Otherwise, people will wonder : "what on earth is contagious/being >>> propagated" >>> >>> just thinking of yet another keyword name : ignore_masked (or drop_masked) >>> >>> If I remember well, in R it is dropna. It would be nice if the boolean >>> switch followed the same logic. >>> >>> Now of course the convolution function is more general than just >>> autocorrelation... >> >> I think "drop" or "ignore" is too generic, for correlation it would be >> for example ignore pairs versus ignore cases. >> >> To me propagate sounds ok to me, but something with `valid` might be >> more explicit for convolution or `correlate`, however `valid` also >> refers to the end points, so maybe valid_na or valid_masked=True >> >> Josef >> >>> >>> best, >>> Pierre >>> >>> >>> ___ >>> NumPy-Discussion mailing list >>> NumPy-Discussion@scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to name "contagious" keyword in np.ma.convolve
On Tue, Oct 18, 2016 at 1:25 PM,wrote: > On Mon, Oct 17, 2016 at 1:01 PM, Pierre Haessig > wrote: >> Hi, >> >> >> Le 16/10/2016 à 11:52, Hanno Klemm a écrit : >>> When I have similar situations, I usually interpolate between the valid >>> values. I assume there are a lot of use cases for convolutions but I have >>> difficulties imagining that ignoring a missing value and, for the purpose >>> of the computation, treating it as zero is useful in many of them. >> When estimating the autocorrelation of a signal, it make sense to drop >> missing pairs of values. Only in this use case, it opens the question of >> correcting or not correcting for the number of missing elements when >> computing the mean. I don't remember what R function "acf" is doing. as aside: statsmodels has now an option for acf and similar missing : str A string in ['none', 'raise', 'conservative', 'drop'] specifying how the NaNs are to be treated. Josef >> >> >> Also, coming back to the initial question, I feel that it is necessary >> that the name "mask" (or "na" or similar) appears in the parameter name. >> Otherwise, people will wonder : "what on earth is contagious/being >> propagated" >> >> just thinking of yet another keyword name : ignore_masked (or drop_masked) >> >> If I remember well, in R it is dropna. It would be nice if the boolean >> switch followed the same logic. >> >> Now of course the convolution function is more general than just >> autocorrelation... > > I think "drop" or "ignore" is too generic, for correlation it would be > for example ignore pairs versus ignore cases. > > To me propagate sounds ok to me, but something with `valid` might be > more explicit for convolution or `correlate`, however `valid` also > refers to the end points, so maybe valid_na or valid_masked=True > > Josef > >> >> best, >> Pierre >> >> >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to name "contagious" keyword in np.ma.convolve
On Mon, Oct 17, 2016 at 1:01 PM, Pierre Haessigwrote: > Hi, > > > Le 16/10/2016 à 11:52, Hanno Klemm a écrit : >> When I have similar situations, I usually interpolate between the valid >> values. I assume there are a lot of use cases for convolutions but I have >> difficulties imagining that ignoring a missing value and, for the purpose of >> the computation, treating it as zero is useful in many of them. > When estimating the autocorrelation of a signal, it make sense to drop > missing pairs of values. Only in this use case, it opens the question of > correcting or not correcting for the number of missing elements when > computing the mean. I don't remember what R function "acf" is doing. > > > Also, coming back to the initial question, I feel that it is necessary > that the name "mask" (or "na" or similar) appears in the parameter name. > Otherwise, people will wonder : "what on earth is contagious/being > propagated" > > just thinking of yet another keyword name : ignore_masked (or drop_masked) > > If I remember well, in R it is dropna. It would be nice if the boolean > switch followed the same logic. > > Now of course the convolution function is more general than just > autocorrelation... I think "drop" or "ignore" is too generic, for correlation it would be for example ignore pairs versus ignore cases. To me propagate sounds ok to me, but something with `valid` might be more explicit for convolution or `correlate`, however `valid` also refers to the end points, so maybe valid_na or valid_masked=True Josef > > best, > Pierre > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Scipy installation on Window with mingw32
Hi, A few comments: - you really really want to use a scientific Python distribution to avoid these issues on Windows. see http://scipy.org/install.html - we used to build scipy .exe installers with mingw32 but don't do that anymore because it's just too much of a pain. IIRC the last release we did that for was 0.16.0, with the toolchain in https://github.com/numpy/numpy-vendor. - I don't recognize the error; looks not specific to recent changes in scipy so there's probably something in your environment not set up quite right. Cheers, Ralf On Tue, Oct 18, 2016 at 8:18 PM, Shreyank Amartya < shreyank.amar...@itcinfotech.com> wrote: > Hi, > > > > I am trying install to theano which also requires numpy and scipy on > windows 7 with mingw32 compilers. > > I have successfully installed numpy using mingw32 but however when trying > to install scipy I get this error: > > > > Looking for python27.dll > > Building msvcr library: "c:\python27\libs\libmsvcr90.a" (from > C:\Windows\win > > sxs\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.21022.8_ > none_750b37ff97f4f68b\ > > msvcr90.dll) > > objdump.exe: C:\Windows\winsxs\amd64_microsoft.vc90.crt_ > 1fc8b3b9a1e18e3b_9.0 > > .21022.8_none_750b37ff97f4f68b\msvcr90.dll: File format not recognized > > Traceback (most recent call last): > > File "", line 1, in > > File "c:\users\22193\appdata\local\temp\pip-build-d3f_pb\scipy\ > setup.py", > > line 415, in > > setup_package() > > File "c:\users\22193\appdata\local\temp\pip-build-d3f_pb\scipy\ > setup.py", > > line 411, in setup_package > > setup(**metadata) > > File "c:\python27\lib\site-packages\numpy\distutils\core.py", line > 169, in > > setup > > return old_setup(**new_attr) > > File "c:\python27\lib\distutils\core.py", line 151, in setup > > dist.run_commands() > > File "c:\python27\lib\distutils\dist.py", line 953, in run_commands > > self.run_command(cmd) > > File "c:\python27\lib\distutils\dist.py", line 972, in run_command > > cmd_obj.run() > > File "c:\python27\lib\site-packages\numpy\distutils\command\install.py", > l > > ine 62, in run > > r = self.setuptools_run() > > File "c:\python27\lib\site-packages\numpy\distutils\command\install.py", > l > > ine 36, in setuptools_run > > return distutils_install.run(self) > > File "c:\python27\lib\distutils\command\install.py", line 563, in > run > > self.run_command('build') > > File "c:\python27\lib\distutils\cmd.py", line 326, in run_command > > self.distribution.run_command(command) > > File "c:\python27\lib\distutils\dist.py", line 972, in run_command > > cmd_obj.run() > > File "c:\python27\lib\site-packages\numpy\distutils\command\build.py", > lin > > e 47, in run > > old_build.run(self) > > File "c:\python27\lib\distutils\command\build.py", line 127, in run > > self.run_command(cmd_name) > > File "c:\python27\lib\distutils\cmd.py", line 326, in run_command > > self.distribution.run_command(command) > > File "c:\python27\lib\distutils\dist.py", line 972, in run_command > > cmd_obj.run() > > File "c:\python27\lib\site-packages\numpy\distutils\ > command\build_src.py", > > line 147, in run > > self.build_sources() > > File "c:\python27\lib\site-packages\numpy\distutils\ > command\build_src.py", > > line 164, in build_sources > > self.build_extension_sources(ext) > > File "c:\python27\lib\site-packages\numpy\distutils\ > command\build_src.py", > > line 323, in build_extension_sources > > sources = self.generate_sources(sources, ext) > > File "c:\python27\lib\site-packages\numpy\distutils\ > command\build_src.py", > > line 376, in generate_sources > > source = func(extension, build_dir) > > File "scipy\spatial\setup.py", line 35, in get_qhull_misc_config > > if config_cmd.check_func('open_memstream', decl=True, call=True): > > File "c:\python27\lib\site-packages\numpy\distutils\command\config.py", > li > > ne 312, in check_func > > self._check_compiler() > > File "c:\python27\lib\site-packages\numpy\distutils\command\config.py", > li > > ne 39, in _check_compiler > > old_config._check_compiler(self) > > File "c:\python27\lib\distutils\command\config.py", line 102, in > _check_co > > mpiler > > dry_run=self.dry_run, force=1) > > File "c:\python27\lib\site-packages\numpy\distutils\ccompiler.py", > line 59 > > 6, in new_compiler > > compiler = klass(None, dry_run, force) > > File "c:\python27\lib\site-packages\numpy\distutils\ > mingw32ccompiler.py", > > line 96, in __init__ > > msvcr_success = build_msvcr_library() > > File "c:\python27\lib\site-packages\numpy\distutils\ > mingw32ccompiler.py", > > line 360, in build_msvcr_library > > generate_def(dll_file, def_file) > > File
[Numpy-discussion] Scipy installation on Window with mingw32
Hi, I am trying install to theano which also requires numpy and scipy on windows 7 with mingw32 compilers. I have successfully installed numpy using mingw32 but however when trying to install scipy I get this error: Looking for python27.dll Building msvcr library: "c:\python27\libs\libmsvcr90.a" (from C:\Windows\win sxs\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.21022.8_none_750b37ff97f4f68b\ msvcr90.dll) objdump.exe: C:\Windows\winsxs\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0 .21022.8_none_750b37ff97f4f68b\msvcr90.dll: File format not recognized Traceback (most recent call last): File "", line 1, in File "c:\users\22193\appdata\local\temp\pip-build-d3f_pb\scipy\setup.py", line 415, in setup_package() File "c:\users\22193\appdata\local\temp\pip-build-d3f_pb\scipy\setup.py", line 411, in setup_package setup(**metadata) File "c:\python27\lib\site-packages\numpy\distutils\core.py", line 169, in setup return old_setup(**new_attr) File "c:\python27\lib\distutils\core.py", line 151, in setup dist.run_commands() File "c:\python27\lib\distutils\dist.py", line 953, in run_commands self.run_command(cmd) File "c:\python27\lib\distutils\dist.py", line 972, in run_command cmd_obj.run() File "c:\python27\lib\site-packages\numpy\distutils\command\install.py", l ine 62, in run r = self.setuptools_run() File "c:\python27\lib\site-packages\numpy\distutils\command\install.py", l ine 36, in setuptools_run return distutils_install.run(self) File "c:\python27\lib\distutils\command\install.py", line 563, in run self.run_command('build') File "c:\python27\lib\distutils\cmd.py", line 326, in run_command self.distribution.run_command(command) File "c:\python27\lib\distutils\dist.py", line 972, in run_command cmd_obj.run() File "c:\python27\lib\site-packages\numpy\distutils\command\build.py", lin e 47, in run old_build.run(self) File "c:\python27\lib\distutils\command\build.py", line 127, in run self.run_command(cmd_name) File "c:\python27\lib\distutils\cmd.py", line 326, in run_command self.distribution.run_command(command) File "c:\python27\lib\distutils\dist.py", line 972, in run_command cmd_obj.run() File "c:\python27\lib\site-packages\numpy\distutils\command\build_src.py", line 147, in run self.build_sources() File "c:\python27\lib\site-packages\numpy\distutils\command\build_src.py", line 164, in build_sources self.build_extension_sources(ext) File "c:\python27\lib\site-packages\numpy\distutils\command\build_src.py", line 323, in build_extension_sources sources = self.generate_sources(sources, ext) File "c:\python27\lib\site-packages\numpy\distutils\command\build_src.py", line 376, in generate_sources source = func(extension, build_dir) File "scipy\spatial\setup.py", line 35, in get_qhull_misc_config if config_cmd.check_func('open_memstream', decl=True, call=True): File "c:\python27\lib\site-packages\numpy\distutils\command\config.py", li ne 312, in check_func self._check_compiler() File "c:\python27\lib\site-packages\numpy\distutils\command\config.py", li ne 39, in _check_compiler old_config._check_compiler(self) File "c:\python27\lib\distutils\command\config.py", line 102, in _check_co mpiler dry_run=self.dry_run, force=1) File "c:\python27\lib\site-packages\numpy\distutils\ccompiler.py", line 59 6, in new_compiler compiler = klass(None, dry_run, force) File "c:\python27\lib\site-packages\numpy\distutils\mingw32ccompiler.py", line 96, in __init__ msvcr_success = build_msvcr_library() File "c:\python27\lib\site-packages\numpy\distutils\mingw32ccompiler.py", line 360, in build_msvcr_library generate_def(dll_file, def_file) File "c:\python27\lib\site-packages\numpy\distutils\mingw32ccompiler.py", line 274, in generate_def raise ValueError("Symbol table not found") ValueError: Symbol table not found Command "c:\python27\python.exe -u -c "import setuptools, tokenize;__file__='c:\ \users\\22193\\appdata\\local\\temp\\pip-build-d3f_pb\\scipy\\setup.py';exec(com pile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __f ile__, 'exec'))" install --record c:\users\22193\appdata\local\temp\pip-wxyrfu-r ecord\install-record.txt --single-version-externally-managed --compile" failed w ith error code 1 in c:\users\22193\appdata\local\temp\pip-build-d3f_pb\scipy\ C:\Python27\Scripts> I realize this is due to some problem with resolving symbols in python27.dll. Is there a workaround for this? I need this to work on this setup as this is my workstation at office and I cannot install Ubuntu which would have been way easier. Things I have tried: I used to get an