Re: units issues
On 2017-05-24 20:07, Doug Henderson wrote:
> , On 24 May 2017 at 18:16, Brian Inglis wrote:
>> I would appreciate it if anyone has any experience with, or can point me
>> to, any python3 code in any Cygwin package which works Windows proxy
>> authentication?
> I had a glance at the source. I recall adapting it to load SQL tables
> with units data. I recall writing code very similar to that in the
> units_cur{2|3} python sources to keep the table current.
> units_cur2 is the python2 version, units_cur3 is the python3 version
> of the updater.
> It is using the urllib and ElementTree packages which are parts of the
> standard python release.
> A possibly more featureful replacement for urllib is the requests
> package, available in the cygwin packages python2-requests and
> python3-requests. See http://docs.python-requests.org/en/master/ for
> the documentation. It claims "HTTP(S) Proxy Support" however I have
> not used that feature.
> There are two similar lines that use urllib:
> ... urllib.request.urlopen('URL1>') ...
> ... urllib.request.urlopen('?f=xml') ...
> Using requests, they become:
> ... requests.get('').text ...
> ... requests.get('', params={'f':'xml'}).text ...
> Some additional code is required to support proxies, but I can't help
> much there.
> See http://docs.python-requests.org/en/master/user/advanced/#proxies for info.
Hi Doug,
Thanks for the pointers which document things better than urllib.
It looks like it should be possible to support proxies by setting
environment variables:
{HTTP,HTTPS}_PROXY={http,https,socks5}://user:password@host:port/
I will check this out further to see whether that requires urllib2,
urllib3, or requests, which are packaged for cygwin, and follow up with
the OP on the issue, to see what works.
--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
Re: units issues
, On 24 May 2017 at 18:16, Brian Inglis wrote:
> I would appreciate it if anyone has any experience with, or can point me
> to, any python3 code in any Cygwin package which works Windows proxy
> authentication?
I had a glance at the source. I recall adapting it to load SQL tables
with units data. I recall writing code very similar to that in the
units_cur{2|3} python sources to keep the table current.
units_cur2 is the python2 version, units_cur3 is the python3 version
of the updater.
It is using the urllib and ElementTree packages which are parts of the
standard python release.
A possibly more featureful replacement for urllib is the requests
package, available in the cygwin packages python2-requests and
python3-requests. See http://docs.python-requests.org/en/master/ for
the documentation. It claims "HTTP(S) Proxy Support" however I have
not used that feature.
There are two similar lines that use urllib:
... urllib.request.urlopen('URL1>') ...
... urllib.request.urlopen('?f=xml') ...
Using requests, they become:
... requests.get('').text ...
... requests.get('', params={'f':'xml'}).text ...
Some additional code is required to support proxies, but I can't help
much there.
See http://docs.python-requests.org/en/master/user/advanced/#proxies for info.
HTH
Doug
--
Doug Henderson, Calgary, Alberta, Canada - from gmail.com
Re: units issues
On 2017-05-24 12:13, Achim Gratz wrote: > Brian Inglis writes: >> Updating the currencies only when setup is run seems to me to be >> insufficient if users want to use current currency conversions. > > Well yes. But that means that each user should be able to update, not > just the administrator. > >> Would the best approach be to punt on running the update script at all, >> install a null /usr/share/units/currency.units file, announce and >> document that if currency conversions are desirable, the user should >> arrange to run the update script, either from the command line, profile, >> cron job, or Task Scheduler? > > In that case the file should not be packaged at all and it should live > in /usr/var somewplace, I think. But if you think a bit further you'll > see that this wouldn't be sufficient when the user cannot write there. > >> If we kept the postinstall script, we could change it to run only if the >> currency.units file is non-null and drop the time check: >> [ -s /usr/share/units/currency.units ] && /usr/bin/units_cur >> or keep the time check, using find -mtime +0, or date and stat. > > Again, if you package that updater, it'd make sense to create a > sub-package just for that and provide any customization as to which > update style is preferred (the things you mentioned are all sensible in > some situation, just not together). The longer I think about it the > less convincing the idea that a postinstalll script should do the actual > updating, although it might be one of the options you offer. The updater is part of the installed package. Running it in postinstall was added by the original cygwin packager. I am now leaning heavily towards a /usr/share/doc/Cygwin/units.README documenting the commands to automate currency updates properly, the possible approaches to implementing the commands, leaving the mechanism to the user to implement, and conspicuously referring to that doc in future upgrade notices. I will try again to contact upstream via bug-un...@gnu.org this time, and work on fixing the issues with the updater site cert expiry, by retrying with cert checking disabled, and see if I can come up with another fix to work with Windows proxy authentication. I would appreciate it if anyone has any experience with, or can point me to, any python3 code in any Cygwin package which works Windows proxy authentication? -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
Re: units issues
Brian Inglis writes: > Updating the currencies only when setup is run seems to me to be > insufficient if users want to use current currency conversions. Well yes. But that means that each user should be able to update, not just the administrator. > Would the best approach be to punt on running the update script at all, > install a null /usr/share/units/currency.units file, announce and > document that if currency conversions are desirable, the user should > arrange to run the update script, either from the command line, profile, > cron job, or Task Scheduler? In that case the file should not be packaged at all and it should live in /usr/var somewplace, I think. But if you think a bit further you'll see that this wouldn't be sufficient when the user cannot write there. > If we kept the postinstall script, we could change it to run only if the > currency.units file is non-null and drop the time check: > [ -s /usr/share/units/currency.units ] && /usr/bin/units_cur > or keep the time check, using find -mtime +0, or date and stat. Again, if you package that updater, it'd make sense to create a sub-package just for that and provide any customization as to which update style is preferred (the things you mentioned are all sensible in some situation, just not together). The longer I think about it the less convincing the idea that a postinstalll script should do the actual updating, although it might be one of the options you offer. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
Re: units issues
Buchbinder, Barry (NIH/NIAID) [E] writes: > I would prefer that by default updates happen automatically and those > who do not want automatic updates do something to stop them from > happening. That there should be updates was never debated, only the machanism of doing so is discussed here. > For instance, someone who does not want updates makes the > definitions file read-only and have the script check for write > permission and exit or skip updating if the file cannot be written. A > zero length read-only file works if one is worried about someone using > stale conversion factors. An environmental variable whose existence > marks no-update might be another possibility. No again. In general, it is a no-no to alter or remove files that came with the package (besides, the user might not even be able to, depending on how Cygwin gets installed). Cygwin isn't very fussy about such errors at the moment, but that shouldn't be an excuse to become sloppy. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for KORG EX-800 and Poly-800MkII V0.9: http://Synth.Stromeko.net/Downloads.html#KorgSDada
Re: units issues
On 2017-05-24 10:26, Buchbinder, Barry (NIH/NIAID) [E] wrote: > Brian Inglis sent the following at Tuesday, May 23, 2017 10:37 PM >> On 2017-05-23 17:55, Doug Henderson wrote: >>> On 23 May 2017 at 15:49, Brian Inglis wrote: Updating the currencies only when setup is run seems to me to be insufficient if users want to use current currency conversions. >> It's a command line utility from GNU with currency conversion factors >> in a separate definition file included from the main file, updated by >> a Python script, which downloads an RSS XML file of current (Euro) >> rates from a free source with a permissive licence, and converts it to >> definitions acceptable to the utility, overwriting the existing file. >> >> The main issues are that, as currently implemented, currency rates are >> updated automatically by a postinstall script only when setup is run; >> setup may be running in an environment without external access, so the >> postinstall script will generate an error; users may not want or care >> about currency updates; and the postinstall script uses find to avoid >> updating if there is no currency file, or it has been updated recently. >> >> One option to deal with this is update the package to install a >> zero length currency definitions file, so currency conversions are >> not defined, but the program has no issues, and drop the permanent >> postinstall script to perform updates. Then announce and document that >> users who want updated currency conversion rates need to run the update >> script from the command line, a profile script, cron job, or Windows >> Scheduled Task, as is desirable if they use currency conversions. > I would prefer that by default updates happen automatically and those > who do not want automatic updates do something to stop them from > happening. For instance, someone who does not want updates makes the > definitions file read-only and have the script check for write > permission and exit or skip updating if the file cannot be written. A > zero length read-only file works if one is worried about someone using > stale conversion factors. An environmental variable whose existence > marks no-update might be another possibility. I agree with Achim that currency updates should not be done at postinstall, as the user could be doing an automated install with no external access, and we don't know that. With no guaranteed automatic execution facility, we don't even have a mechanism, how should updates should be done automatically and how can we provide this? Alternative suggestions for automated updates are welcome and why I am asking here. For other services, we leave it to users to figure out how to start them up and shut them down before setup is run. We could install an /etc/profile.d script which asks the user first time thru if they want daily currency updates during login over the network, and then either null or update currencies, and after do or do not update currencies. We could document how updates could be run: run the command from the shell, or add the necessary code to: some profile or profile.d script; command alias or function wrapper in some profile script; script which uses units currencies, cron job, or Windows Scheduled Task. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
RE: units issues
Brian Inglis sent the following at Tuesday, May 23, 2017 10:37 PM > >On 2017-05-23 17:55, Doug Henderson wrote: > >> On 23 May 2017 at 15:49, Brian Inglis wrote: >> >>> Updating the currencies only when setup is run seems to me to be >>> insufficient if users want to use current currency conversions. >> >> Currencies needs to be split to a different package from non-currency >> units. Non-currency units is very static, whereas most currencies >> changes daily. In the absence of updates, non-currency units are still >> useful; but currencies are incorrect in most cases, and increasingly >> so as time pass. >> >> It is difficult to predict user requirements based on current package >> dependencies as non-currency units and currencies are likely used by >> almost disjoint sets of packages. Users of non-currency units should >> not be burdened by the complexities of currencies. But if they do need >> currencies, they should be up to date. >> >> A possible solution is to check the currency of the currency data on >> library initialization or each use by comparing the current date with >> the data's date of update. When the data is stale, automatically >> update it if possible, or notify the user of the problem. > >It's a command line utility from GNU with currency conversion factors >in a separate definition file included from the main file, updated by >a Python script, which downloads an RSS XML file of current (Euro) >rates from a free source with a permissive licence, and converts it to >definitions acceptable to the utility, overwriting the existing file. > >The main issues are that, as currently implemented, currency rates are >updated automatically by a postinstall script only when setup is run; >setup may be running in an environment without external access, so the >postinstall script will generate an error; users may not want or care >about currency updates; and the postinstall script uses find to avoid >updating if there is no currency file, or it has been updated recently. > >One option to deal with this is update the package to install a >zero length currency definitions file, so currency conversions are >not defined, but the program has no issues, and drop the permanent >postinstall script to perform updates. Then announce and document that >users who want updated currency conversion rates need to run the update >script from the command line, a profile script, cron job, or Windows >Scheduled Task, as is desirable if they use currency conversions. > >There are also issues with the Python update script, as the currency >source site cert expired recently, causing the update and postinstall >scripts to fail, with no workaround other than a replacement or patch; >and a poster has problems using the update script with Windows proxies, >which I have addressed to the upstream maintainer for discussion about >approaches. If there is no response, I will create a Cygwin update patch >and submit it upstream, but there has been no visible response to issues >raised on the GNU site. I would prefer that by default updates happen automatically and those who do not want automatic updates do something to stop them from happening. For instance, someone who does not want updates makes the definitions file read-only and have the script check for write permission and exit or skip updating if the file cannot be written. A zero length read-only file works if one is worried about someone using stale conversion factors. An environmental variable whose existence marks no-update might be another possibility. Best wishes, - Barry Disclaimer: Statements made herein are not made on behalf of NIAID.
Re: units issues
On 2017-05-23 17:55, Doug Henderson wrote: > On 23 May 2017 at 15:49, Brian Inglis wrote: > >> Updating the currencies only when setup is run seems to me to be >> insufficient if users want to use current currency conversions. > > Currencies needs to be split to a different package from non-currency > units. Non-currency units is very static, whereas most currencies > changes daily. In the absence of updates, non-currency units are still > useful; but currencies are incorrect in most cases, and increasingly > so as time pass. > > It is difficult to predict user requirements based on current package > dependencies as non-currency units and currencies are likely used by > almost disjoint sets of packages. Users of non-currency units should > not be burdened by the complexities of currencies. But if they do need > currencies, they should be up to date. > > A possible solution is to check the currency of the currency data on > library initialization or each use by comparing the current date with > the data's date of update. When the data is stale, automatically > update it if possible, or notify the user of the problem. It's a command line utility from GNU with currency conversion factors in a separate definition file included from the main file, updated by a Python script, which downloads an RSS XML file of current (Euro) rates from a free source with a permissive licence, and converts it to definitions acceptable to the utility, overwriting the existing file. The main issues are that, as currently implemented, currency rates are updated automatically by a postinstall script only when setup is run; setup may be running in an environment without external access, so the postinstall script will generate an error; users may not want or care about currency updates; and the postinstall script uses find to avoid updating if there is no currency file, or it has been updated recently. One option to deal with this is update the package to install a zero length currency definitions file, so currency conversions are not defined, but the program has no issues, and drop the permanent postinstall script to perform updates. Then announce and document that users who want updated currency conversion rates need to run the update script from the command line, a profile script, cron job, or Windows Scheduled Task, as is desirable if they use currency conversions. There are also issues with the Python update script, as the currency source site cert expired recently, causing the update and postinstall scripts to fail, with no workaround other than a replacement or patch; and a poster has problems using the update script with Windows proxies, which I have addressed to the upstream maintainer for discussion about approaches. If there is no response, I will create a Cygwin update patch and submit it upstream, but there has been no visible response to issues raised on the GNU site. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
Re: units issues
On 23 May 2017 at 15:49, Brian Inglis wrote: > Updating the currencies only when setup is run seems to me to be > insufficient if users want to use current currency conversions. Currencies needs to be split to a different package from non-currency units. Non-currency units is very static, whereas most currencies changes daily. In the absence of updates, non-currency units are still useful; but currencies are incorrect in most cases, and increasingly so as time pass. It is difficult to predict user requirements based on current package dependencies as non-currency units and currencies are likely used by almost disjoint sets of packages. Users of non-currency units should not be burdened by the complexities of currencies. But if they do need currencies, they should be up to date. A possible solution is to check the currency of the currency data on library initialization or each use by comparing the current date with the data's date of update. When the data is stale, automatically update it if possible, or notify the user of the problem. Doug -- Doug Henderson, Calgary, Alberta, Canada - from gmail.com
Re: units issues
On 2017-05-23 11:28, Achim Gratz wrote: > Brian Inglis writes: >> The easiest approaches to this would be: >> - rename or delete postinstall script which might upset cygcheck or >> setup remove >> - null /usr/share/units/currency.units, as it is required and produces >> an error message if not available, but if it is empty, everything works. > > Nope. The user should not have to muck with packaged files at all. > >> It is not very useful if not up to date (I run the update daily), and if >> not up to date, might as well be empty, but some opinions may differ. > > The issue is that you cannot assume that postinstall scripts are able to > access the network at all. Where this isn't possible the administrator > will have to find a different way of keeping those files up-to-date > (which should also not be packaged), but that's something to maybe just > document. But you will have to provide some way of letting the user > specify if that updating is OK. If you want this to be possible during > the initial install you might even need to provide another sub-package > whose only purpose is to confer this decision (it could be empty and > just doing a simple postinstall action). > >> In the postinstall script find which ensures updates happen at most daily: > > I've asked specifically to structure any perpetual postinstall script in > a way that it doesn't do any non-necessary work, be it forking or > otherwise unless it is going to actually do something useful. A simple > file test that exits the postinstall script when the file is or isn't > there satisfies that constraint, running commands and scripts doesn't. Updating the currencies only when setup is run seems to me to be insufficient if users want to use current currency conversions. Would the best approach be to punt on running the update script at all, install a null /usr/share/units/currency.units file, announce and document that if currency conversions are desirable, the user should arrange to run the update script, either from the command line, profile, cron job, or Task Scheduler? If we kept the postinstall script, we could change it to run only if the currency.units file is non-null and drop the time check: [ -s /usr/share/units/currency.units ] && /usr/bin/units_cur or keep the time check, using find -mtime +0, or date and stat. I have followed up with the upstream maintainer AdrianM at GNU dot org about the Python currency update script issues with no response yet. Does anyone have contact or know a better address to try? PM if so. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
Re: units issues
Brian Inglis writes: > The easiest approaches to this would be: > - rename or delete postinstall script which might upset cygcheck or > setup remove > - null /usr/share/units/currency.units, as it is required and produces > an error message if not available, but if it is empty, everything works. Nope. The user should not have to muck with packaged files at all. > It is not very useful if not up to date (I run the update daily), and if > not up to date, might as well be empty, but some opinions may differ. The issue is that you cannot assume that postinstall scripts are able to access the network at all. Where this isn't possible the administrator will have to find a different way of keeping those files up-to-date (which should also not be packaged), but that's something to maybe just document. But you will have to provide some way of letting the user specify if that updating is OK. If you want this to be possible during the initial install you might even need to provide another sub-package whose only purpose is to confer this decision (it could be empty and just doing a simple postinstall action). > In the postinstall script find which ensures updates happen at most daily: I've asked specifically to structure any perpetual postinstall script in a way that it doesn't do any non-necessary work, be it forking or otherwise unless it is going to actually do something useful. A simple file test that exits the postinstall script when the file is or isn't there satisfies that constraint, running commands and scripts doesn't. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
Re: units issues
On the cygwin list, Achim Gratz asked: > Can you please make that configurable by some file in /var/lib/units > that will prevent the perpetual postinstall script from even forking > if the user decides that this should not be updated? Configuration of units is currently supported only in /usr/share/units/*.units files which are defaulted by the utility, specified on the command line, included from one of those, or specified in ~/.units, and conditions may be specified depending only on either locale or an environment variable. The easiest approaches to this would be: - rename or delete postinstall script which might upset cygcheck or setup remove - null /usr/share/units/currency.units, as it is required and produces an error message if not available, but if it is empty, everything works. It is not very useful if not up to date (I run the update daily), and if not up to date, might as well be empty, but some opinions may differ. In the postinstall script find which ensures updates happen at most daily: /usr/bin/find /usr/share/units/currency.units -mtime +0 -exec /usr/bin/units_cur \; I could add a -size +0 qualifier, which would skip the update if the file is empty, and avoid creating a config flag file in directory /var/lib/units/, which is not currently created or used by the package. I will ask upstream about disabling currency updates by the python script, as that would be a preferred configurable approach. In any case, I will raise handling an expired cert in the python update script upstream, and suggest a configurable override. Please respond with any advice on how disabling pi updates and expired certs are or should be dealt with in postinstall scripts or python (I posted a temporary replacement workaround script and change to the original on the cygwin list, so the technical workaround is known and handled). and Ian Lambert reported: > In /var/log/setup.log.full I see: > Error connecting to currency server. not known> > Running /usr/bin/units_cur or /etc/postinstall/zp_units_cur.sh I get > Error connecting to currency server. HTTP Error 407: Proxy Authentication > Required > I'm behind a proxy requiring username, password, and restrictions on > user-agents... I will raise handling proxies upstream, but once again, any advice is welcome, including technical approaches that work with Windows proxies under Cygwin in python3 (I am not a "python guy", and resort to google and SO, although some of those "suggestions" just don't work, as I found looking for ways to handle the expired cert). I posted responses that I'd take the issues upstream, follow up here, and report back, or provide a package update. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
