Re: [gentoo-user] USE flags handling
On Mon, 4 Aug 2014 04:39:45 +0100, Stroller wrote: I have to confess, I don't follow desktop development to know myself what Phonon means. The page linked to in the package description for media-libs/phonon is, unsurprisingly, uninformative: https://projects.kde.org/projects/kdesupport/phonon http://en.wikipedia.org/wiki/Phonon_(software) -- Neil Bothwick A bit of tolerance is worth a megabyte of flaming. -- Henry Spencer signature.asc Description: PGP signature
Re: [gentoo-user] USE flags handling
On Mon, 4 Aug 2014 08:26:59 +0100, Neil Bothwick wrote: I have to confess, I don't follow desktop development to know myself what Phonon means. The page linked to in the package description for media-libs/phonon is, unsurprisingly, uninformative: https://projects.kde.org/projects/kdesupport/phonon http://en.wikipedia.org/wiki/Phonon_(software) Or follow the links in the kde.org page and after four or so clicks you end up with the information you expected to find in the first place https://userbase.kde.org/Phonon -- Neil Bothwick IBM - Incredibly Bastardized Multitasking... signature.asc Description: PGP signature
[gentoo-user] Re: Recommendations for scheduler
J. Roeleveld jo...@antarean.org wrote: With the kind of schedules I am working with (and I believe Alan will also end up with), restarting the whole process from the start can lead to issues. Finding out how far the process got before the service crashed can become rather complex. I am not sure whether I understand this correctly: schedule has not a problem to display which tasks have finished/failed/are still running at any time. Of course, a finer granulation than tasks are not possible (how far has a certain task got?) because this would require knowledge about the task and how to check it - you need to be able to split your tasks into more shell commands to make a finer granulation available for schedule. You can just rerun your driving script with the effect that the tasks which already are finished/failed will actually not be restarted, but the behaviour is as if they would finish immediately and report that they are finished/failed. (When you plan to do this, I would suggest to schedule things like sleep as separate tasks, too, and not build them into the driving script.) If there is an unexpected problem, and e.g. you want to re-run a failed task anyway, you can just re-queue your new task on the same place as there was the previous task, e.g. schedule remove jobnr schedule -j jobnr queue commmand to do your task Then the old job (and its state) is replaced by the new queued job, and your (identical as before) driving script will start it instead of assuming that the job is already finished. In order to avoid races, I would recommend to do the above only while your driving script is not running (e.g., you can put it in the background with ctrl-z if you have written it in (...) or if it is really a classical script, and then continue it with fg; or you even stop it completely with Ctrl-c and re-run it, depending on what you want): The problem is that between the above two commands the jobs after jobnr are renumbered. Alternatively, you can insert your new job at the end of the joblist and then use something like (untested) schedule -jjobnr insert 0 jobnr+1:-1 schedule remove 0 to to re-sort your job list: The insert is race-free, and having added a job at the end for some time will hopefully not disturb anything.
Re: [gentoo-user] Re: Recommendations for scheduler
On 4 August 2014 10:41:04 CEST, Martin Vaeth mar...@mvath.de wrote: J. Roeleveld jo...@antarean.org wrote: With the kind of schedules I am working with (and I believe Alan will also end up with), restarting the whole process from the start can lead to issues. Finding out how far the process got before the service crashed can become rather complex. I am not sure whether I understand this correctly: The schedules I am used to dealing with easily span 8 - 14 hours with occasionally even over a week. These schedules then also can't be restarted from the beginning when they stop halfway through without risking massive consistency problems in the final data. And then multiple of those starting at random times with occasionally a whole bunch of the same schedule put into the queue with dependencies to the previous run. If, during that time, one of the machines has a hardware failure or the scheduling process crashes on one or more of the servers, the last state needs to be recoverable. If you have to clean up the environment and bring it back to a state where you can restart the schedules, it saves time if you know which commands and tasks were actually running at the time. For this, the schedules, queues and current state for each node needs to be stored on persistent storage. Hope this clarifies it all a bit. -- Joost -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
[gentoo-user] Re: Recommendations for scheduler
J. Roeleveld jo...@antarean.org wrote: These schedules then also can't be restarted from the beginning when they stop halfway through without risking massive consistency problems in the final data. So you have a command which might break due to hardware error and cannot be rerun. I cannot see how any general-purpose scheduler might help you here: You either need to be able to split your command into several (sequential) commands or you need something adapted for your particular command. And then multiple of those starting at random times with occasionally a whole bunch of the same schedule put into the queue with dependencies to the previous run. That's not a problem. Only if the granularity of one command is not fine enough, it becomes a problem. If, during that time, one of the machines has a hardware failure or the scheduling process crashes on one or more of the servers, the last state needs to be recoverable. One must distinguish two cases: 1. The machine running schedule-server has a hardware failure. (Let us assume tha schedule-server does not have a software failure - otherwise, you have problems anyway.) 2. Some other machine has a hardware failure. Case 2. is not bad (as concerns the scheduling): Of course, the machine will not report that it completed the job, and you will have to think how to complete the job. But it is clear that in such exceptional cases you have to interfere manually in some sense. In order to deal with case 1., you can regularly (e.g. each minute) dump the output of schedule list (possibly suppressing non-important data through the options to keep it short). One could add a logging option to decrease the possible race of 1 minute, but in case of hardware failure a possible race cannot be excluded anyway. In case 1. you manually have to re-queue the jobs and think what to do with the already started jobs. However, I cannot imagine that this occurs so frequently that this exceptional case becomes something one should seriously think about.
Re: [gentoo-user] Re: Recommendations for scheduler
On Monday, August 04, 2014 10:11:41 AM Martin Vaeth wrote: J. Roeleveld jo...@antarean.org wrote: These schedules then also can't be restarted from the beginning when they stop halfway through without risking massive consistency problems in the final data. So you have a command which might break due to hardware error and cannot be rerun. I cannot see how any general-purpose scheduler might help you here: You either need to be able to split your command into several (sequential) commands or you need something adapted for your particular command. A general-purpose scheduler can work, as they do exist. (With a price tag) In the OSS world, there is, to my knowledge, none. Yours seems to be the most promising as it looks like the missing features shouldn't be too difficult to add. The commands are relatively simple, but they deal with large amounts of data. I am talking about ETL processes that, due to the amount of data being processed, can easily take several hours per step. If, during one of these steps, the database or ETL process suffers a crash, the activities of the ETL process need to be rolled back to the point where you can restart it. I am not talking about simple schedules related to day-to-day maintenance of a few servers. And then multiple of those starting at random times with occasionally a whole bunch of the same schedule put into the queue with dependencies to the previous run. That's not a problem. Only if the granularity of one command is not fine enough, it becomes a problem. If nothing happens, it can all be stuck into a single script and the end result will be the same. Problems start because the real world is not 100% reliable. If, during that time, one of the machines has a hardware failure or the scheduling process crashes on one or more of the servers, the last state needs to be recoverable. One must distinguish two cases: 1. The machine running schedule-server has a hardware failure. (Let us assume tha schedule-server does not have a software failure - otherwise, you have problems anyway.) 2. Some other machine has a hardware failure. Case 2. is not bad (as concerns the scheduling): Of course, the machine will not report that it completed the job, and you will have to think how to complete the job. But it is clear that in such exceptional cases you have to interfere manually in some sense. Agreed, this happens more often then you might think. In order to deal with case 1., you can regularly (e.g. each minute) dump the output of schedule list (possibly suppressing non-important data through the options to keep it short). Or all the necessary information is kept in-sync on persistent storage. This would then also allow easy fail-over if the master-schedule-node fails. A 2nd machine could quickly take over. One could add a logging option to decrease the possible race of 1 minute, but in case of hardware failure a possible race cannot be excluded anyway. In case 1. you manually have to re-queue the jobs and think what to do with the already started jobs. However, I cannot imagine that this occurs so frequently that this exceptional case becomes something one should seriously think about. As I mentioned above, with BI infrastructure (large databases, complex ETL processes, interactive report services,), the scheduler is busy 24/7. The amount of tasks, schedules, dependencies, states, that needs to kept track off can easily lead to unforeseen issues and bugs.
[gentoo-user] Re: Recommendations for scheduler
J. Roeleveld jo...@antarean.org wrote: So you have a command which might break due to hardware error and cannot be rerun. I cannot see how any general-purpose scheduler might help you here: You either need to be able to split your command into several (sequential) commands or you need something adapted for your particular command. A general-purpose scheduler can work, as they do exist. I doubt that they can solve your problem. Let me repeat: You have a single program which accesses the database in a complex way and somewhere in the course of accessing it, the machine (or program) crashes. No general-purpose program can recover from this: You need particular knowledge of the database and the program if you even want to have a *chance* to recover from such a situation. A program with such a particular knowledge can hardly be called general-purpose. If, during one of these steps, the database or ETL process suffers a crash, the activities of the ETL process need to be rolled back to the point where you can restart it. I agree, but you need particular knowledge of the database and your tasks to do this which is far beyond the job of a scheduler. As already mentioned by someone in this thread, your problem needs to be solved on the level of the database (using snapshopt capabilities etc.) In order to deal with case 1., you can regularly (e.g. each minute) dump the output of schedule list (possibly suppressing non-important data through the options to keep it short). Or all the necessary information is kept in-sync on persistent storage. This would then also allow easy fail-over if the master-schedule-node fails No, it wouldn't, since jobs just finishing and wanting to report their status cannot do this when there is no server. You would need a rather involved protocol to deal with such situations dynamically. It can certainly be done, but it is not something which can easily be added as a feature: If this is required, it has to be the fundamental concept from the very beginning and everything else has to follow this first aim. You need different protocols than TCP sockets, to start with; something like dbus over IP with servers being able to announce their new presence, etc.
Re: [gentoo-user] Re: Recommendations for scheduler
On 04/08/2014 15:31, Martin Vaeth wrote: J. Roeleveld jo...@antarean.org wrote: So you have a command which might break due to hardware error and cannot be rerun. I cannot see how any general-purpose scheduler might help you here: You either need to be able to split your command into several (sequential) commands or you need something adapted for your particular command. A general-purpose scheduler can work, as they do exist. I doubt that they can solve your problem. Let me repeat: You have a single program which accesses the database in a complex way and somewhere in the course of accessing it, the machine (or program) crashes. No general-purpose program can recover from this: You need particular knowledge of the database and the program if you even want to have a *chance* to recover from such a situation. A program with such a particular knowledge can hardly be called general-purpose. Joost, Either make the ETL tool pick up where it stopped and continue as it is the only that knows what it was doing and how far it got. Or, wrap the entire script in a single transaction. -- Alan McKinnon alan.mckin...@gmail.com
[gentoo-user] Python and man problems
Hello all, I encounter some problems with python compiling and man errors. 1- Python : I can't emerge some python packages (ie pyorbit, libmpeg2, libbonobo-python, etc...). Even the installed python packages can't emerge anymore. Here is the error message for libgnome-python-2.28.1-r1 (USE=-examples PYTHON_TARGETS=-python2_7%) *ERROR: dev-python/libgnome-python-2.28.1-r1::gentoo failed (configure phase): * No supported Python implementation in PYTHON_TARGETS. * * Call stack: * ebuild.sh, line 93: Called src_configure * environment, line 4118: Called gnome-python-common-r1_src_configure * environment, line 2078: Called python_parallel_foreach_impl 'gnome2_src_configure' '--disable-allbindings' '--enable-gnome' '--enable-gnomeui' * environment, line 3947: Called _python_obtain_impls * environment, line 702: Called _python_validate_useflags * environment, line 758: Called die * The specific snippet of code: * die No supported Python implementation in PYTHON_TARGETS. * Is it a locale problem ? Here is my /etc/env.d/02locale LANG=fr_FR.utf8 LC_ALL= When i run locale, i get : $ locale LANG=fr_FR.UTF-8 LC_CTYPE=fr_FR.UTF-8 LC_NUMERIC=fr_FR.UTF-8 LC_TIME=fr_FR.UTF-8 LC_COLLATE=fr_FR.UTF-8 LC_MONETARY=fr_FR.UTF-8 LC_MESSAGES=fr_FR.UTF-8 LC_PAPER=fr_FR.UTF-8 LC_NAME=fr_FR.UTF-8 LC_ADDRESS=fr_FR.UTF-8 LC_TELEPHONE=fr_FR.UTF-8 LC_MEASUREMENT=fr_FR.UTF-8 LC_IDENTIFICATION=fr_FR.UTF-8 LC_ALL= 2- man does not work anymore ; i get the error message : ie : $man emerge (sorry, it's in french) *man emerge sh: most : commande introuvable Erreur pendant l'ex�cution du formatage ou de l'affichage. Le syst�me retourne pour (cd /usr/share/man (echo .ll 11.1i; echo .nr LL 11.1i; echo .pl 1100i; /bin/bzip2 -c -d '/usr/share/man/man1/emerge.1.bz2'; echo .\\\; echo .pl \n(nlu+10) | /usr/bin/gtbl | /usr/bin/nroff -mandoc | most) l'erreur 127. Il n'y a pas de page de manuel pour emerge. * I recompiled the sys-apps/man package with success but the error is still there. My configuration : $emerge --info Portage 2.2.8-r1 (default/linux/amd64/13.0/desktop, gcc-4.7.3, glibc-2.19-r1, 3.14.14-gentoo x86_64) = System uname: Linux-3.14.14-gentoo-x86_64-Intel-R-_Core-TM-2_Duo_CPU_E6850_@_3.00GHz-with-gentoo-2.2 KiB Mem: 4051020 total, 1896448 free KiB Swap:2048280 total, 2048280 free Timestamp of tree: Sun, 03 Aug 2014 08:15:01 + ld GNU ld (GNU Binutils) 2.23.2 app-shells/bash: 4.2_p45 dev-java/java-config: 2.2.0 dev-lang/python: 2.7.7, 3.3.5-r1 dev-util/cmake: 2.8.12.2-r1 dev-util/pkgconfig: 0.28-r1 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.12.4 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.13, 2.69 sys-devel/automake: 1.10.3, 1.11.6, 1.12.6, 1.13.4 sys-devel/binutils: 2.23.2 sys-devel/gcc:4.7.3-r1 sys-devel/gcc-config: 1.7.3 sys-devel/libtool:2.4.2-r1 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.13 (virtual/os-headers) sys-libs/glibc: 2.19-r1 Repositories: gentoo ACCEPT_KEYWORDS=amd64 ACCEPT_LICENSE=* -@EULA CBUILD=x86_64-pc-linux-gnu CFLAGS=-O2 -march=native -pipe CHOST=x86_64-pc-linux-gnu CONFIG_PROTECT=/etc /usr/share/config /usr/share/gnupg/qualified.txt CONFIG_PROTECT_MASK=/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/splash /etc/terminfo CXXFLAGS=-O2 -march=native -pipe DISTDIR=/usr/portage/distfiles EMERGE_DEFAULT_OPTS=--autounmask=n FCFLAGS=-O2 -pipe FEATURES=assume-digests binpkg-logs distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync FFLAGS=-O2 -pipe GENTOO_MIRRORS=http://distfiles.gentoo.org; LANG=fr_FR.utf8 LC_ALL= LDFLAGS=-Wl,-O1 -Wl,--as-needed MAKEOPTS=-j2 PKGDIR=/usr/portage/packages PORTAGE_CONFIGROOT=/ PORTAGE_RSYNC_OPTS=--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages PORTAGE_TMPDIR=/var/tmp PORTDIR=/usr/portage PORTDIR_OVERLAY= SYNC=rsync://rsync.europe.gentoo.org/gentoo-portage USE=X a52 aac acl acpi alsa amd64 berkdb bluetooth branding bzip2 cairo cdda cdr cli consolekit cracklib crypt cups cxx dbus dri dts dvd dvdr emboss encode exif fam firefox flac fortran gdbm gif gpm gtk iconv ipv6 jpeg lcms ldap libnotify mad mmx mng modules mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp pam pango pcre pdf png policykit ppds qt3support qt4 readline sdl session spell sse sse2 ssl startup-notification svg tcpd tiff truetype udev udisks unicode upower usb vorbis wxwidgets x264 xcb xml xv xvid zlib ABI_X86=64
Re: [gentoo-user] Python and man problems
On 04/08/2014 15:46, Roger Cahn wrote: Hello all, I encounter some problems with python compiling and man errors. 1- Python : I can't emerge some python packages (ie pyorbit, libmpeg2, libbonobo-python, etc...). Even the installed python packages can't emerge anymore. Here is the error message for libgnome-python-2.28.1-r1 (USE=-examples PYTHON_TARGETS=-python2_7%) Your PYTHON_TARGETS setting does not make sense. If you want python modules to target python-2.7 then define: PYTHON_TARGETS=python2_7 Otherwise, define python3.3 (or even both, space separated). *ERROR: dev-python/libgnome-python-2.28.1-r1::gentoo failed (configure phase): * No supported Python implementation in PYTHON_TARGETS. * * Call stack: * ebuild.sh, line 93: Called src_configure * environment, line 4118: Called gnome-python-common-r1_src_configure * environment, line 2078: Called python_parallel_foreach_impl 'gnome2_src_configure' '--disable-allbindings' '--enable-gnome' '--enable-gnomeui' * environment, line 3947: Called _python_obtain_impls * environment, line 702: Called _python_validate_useflags * environment, line 758: Called die * The specific snippet of code: * die No supported Python implementation in PYTHON_TARGETS. * As above. Is it a locale problem ? Here is my /etc/env.d/02locale LANG=fr_FR.utf8 LC_ALL= When i run locale, i get : $ locale LANG=fr_FR.UTF-8 LC_CTYPE=fr_FR.UTF-8 LC_NUMERIC=fr_FR.UTF-8 LC_TIME=fr_FR.UTF-8 LC_COLLATE=fr_FR.UTF-8 LC_MONETARY=fr_FR.UTF-8 LC_MESSAGES=fr_FR.UTF-8 LC_PAPER=fr_FR.UTF-8 LC_NAME=fr_FR.UTF-8 LC_ADDRESS=fr_FR.UTF-8 LC_TELEPHONE=fr_FR.UTF-8 LC_MEASUREMENT=fr_FR.UTF-8 LC_IDENTIFICATION=fr_FR.UTF-8 LC_ALL= 2- man does not work anymore ; i get the error message : ie : $man emerge (sorry, it's in french) You can make it English by prefixing the command with LC_MESSAGES=C. *man emerge sh: most : commande introuvable Erreur pendant l'ex�cution du formatage ou de l'affichage. Le syst�me retourne pour (cd /usr/share/man (echo .ll 11.1i; echo .nr LL 11.1i; echo .pl 1100i; /bin/bzip2 -c -d '/usr/share/man/man1/emerge.1.bz2'; echo .\\\; echo .pl \n(nlu+10) | /usr/bin/gtbl | /usr/bin/nroff -mandoc | most) l'erreur 127. Il n'y a pas de page de manuel pour emerge. * I recompiled the sys-apps/man package with success but the error is still there. most ... command not found. I've never heard of such a command. It looks as though PAGER=most is defined in your environment. In Gentoo, it is normally set as /usr/bin/less. --Kerin
Re: [gentoo-user] Python and man problems
On 04/08/2014 17:50, Kerin Millar wrote: On 04/08/2014 15:46, Roger Cahn wrote: snip *man emerge sh: most : commande introuvable Erreur pendant l'ex�cution du formatage ou de l'affichage. Le syst�me retourne pour (cd /usr/share/man (echo .ll 11.1i; echo .nr LL 11.1i; echo .pl 1100i; /bin/bzip2 -c -d '/usr/share/man/man1/emerge.1.bz2'; echo .\\\; echo .pl \n(nlu+10) | /usr/bin/gtbl | /usr/bin/nroff -mandoc | most) l'erreur 127. Il n'y a pas de page de manuel pour emerge. * I recompiled the sys-apps/man package with success but the error is still there. most ... command not found. I've never heard of such a command. It looks as though PAGER=most is defined in your environment. In Gentoo, it is normally set as /usr/bin/less. It transpires that sys-apps/most is in portage. Emerging this package should fix it, if you intend to continue using it as the PAGER. --Kerin
Re: [gentoo-user] Python and man problems
On 08/04/2014 05:46 PM, Roger Cahn wrote: Hello all, I encounter some problems with python compiling and man errors. 1- Python : I can't emerge some python packages (ie pyorbit, libmpeg2, libbonobo-python, etc...). Even the installed python packages can't emerge anymore. Here is the error message for libgnome-python-2.28.1-r1 (USE=-examples PYTHON_TARGETS=-python2_7%) *ERROR: dev-python/libgnome-python-2.28.1-r1::gentoo failed (configure phase): * No supported Python implementation in PYTHON_TARGETS. * * Call stack: * ebuild.sh, line 93: Called src_configure * environment, line 4118: Called gnome-python-common-r1_src_configure * environment, line 2078: Called python_parallel_foreach_impl 'gnome2_src_configure' '--disable-allbindings' '--enable-gnome' '--enable-gnomeui' * environment, line 3947: Called _python_obtain_impls * environment, line 702: Called _python_validate_useflags * environment, line 758: Called die * The specific snippet of code: * die No supported Python implementation in PYTHON_TARGETS. * Is it a locale problem ? Here is my /etc/env.d/02locale LANG=fr_FR.utf8 LC_ALL= When i run locale, i get : $ locale LANG=fr_FR.UTF-8 LC_CTYPE=fr_FR.UTF-8 LC_NUMERIC=fr_FR.UTF-8 LC_TIME=fr_FR.UTF-8 LC_COLLATE=fr_FR.UTF-8 LC_MONETARY=fr_FR.UTF-8 LC_MESSAGES=fr_FR.UTF-8 LC_PAPER=fr_FR.UTF-8 LC_NAME=fr_FR.UTF-8 LC_ADDRESS=fr_FR.UTF-8 LC_TELEPHONE=fr_FR.UTF-8 LC_MEASUREMENT=fr_FR.UTF-8 LC_IDENTIFICATION=fr_FR.UTF-8 LC_ALL= 2- man does not work anymore ; i get the error message : ie : $man emerge (sorry, it's in french) *man emerge sh: most : commande introuvable Erreur pendant l'ex�cution du formatage ou de l'affichage. Le syst�me retourne pour (cd /usr/share/man (echo .ll 11.1i; echo .nr LL 11.1i; echo .pl 1100i; /bin/bzip2 -c -d '/usr/share/man/man1/emerge.1.bz2'; echo .\\\; echo .pl \n(nlu+10) | /usr/bin/gtbl | /usr/bin/nroff -mandoc | most) l'erreur 127. Il n'y a pas de page de manuel pour emerge. * I recompiled the sys-apps/man package with success but the error is still there. My configuration : $emerge --info Portage 2.2.8-r1 (default/linux/amd64/13.0/desktop, gcc-4.7.3, glibc-2.19-r1, 3.14.14-gentoo x86_64) = System uname: Linux-3.14.14-gentoo-x86_64-Intel-R-_Core-TM-2_Duo_CPU_E6850_@_3.00GHz-with-gentoo-2.2 KiB Mem: 4051020 total, 1896448 free KiB Swap:2048280 total, 2048280 free Timestamp of tree: Sun, 03 Aug 2014 08:15:01 + ld GNU ld (GNU Binutils) 2.23.2 app-shells/bash: 4.2_p45 dev-java/java-config: 2.2.0 dev-lang/python: 2.7.7, 3.3.5-r1 dev-util/cmake: 2.8.12.2-r1 dev-util/pkgconfig: 0.28-r1 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.12.4 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.13, 2.69 sys-devel/automake: 1.10.3, 1.11.6, 1.12.6, 1.13.4 sys-devel/binutils: 2.23.2 sys-devel/gcc:4.7.3-r1 sys-devel/gcc-config: 1.7.3 sys-devel/libtool:2.4.2-r1 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.13 (virtual/os-headers) sys-libs/glibc: 2.19-r1 Repositories: gentoo ACCEPT_KEYWORDS=amd64 ACCEPT_LICENSE=* -@EULA CBUILD=x86_64-pc-linux-gnu CFLAGS=-O2 -march=native -pipe CHOST=x86_64-pc-linux-gnu CONFIG_PROTECT=/etc /usr/share/config /usr/share/gnupg/qualified.txt CONFIG_PROTECT_MASK=/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/splash /etc/terminfo CXXFLAGS=-O2 -march=native -pipe DISTDIR=/usr/portage/distfiles EMERGE_DEFAULT_OPTS=--autounmask=n FCFLAGS=-O2 -pipe FEATURES=assume-digests binpkg-logs distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync FFLAGS=-O2 -pipe GENTOO_MIRRORS=http://distfiles.gentoo.org; LANG=fr_FR.utf8 LC_ALL= LDFLAGS=-Wl,-O1 -Wl,--as-needed MAKEOPTS=-j2 PKGDIR=/usr/portage/packages PORTAGE_CONFIGROOT=/ PORTAGE_RSYNC_OPTS=--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages PORTAGE_TMPDIR=/var/tmp PORTDIR=/usr/portage PORTDIR_OVERLAY= SYNC=rsync://rsync.europe.gentoo.org/gentoo-portage USE=X a52 aac acl acpi alsa amd64 berkdb bluetooth branding bzip2 cairo cdda cdr cli consolekit cracklib crypt cups cxx dbus dri dts dvd dvdr emboss encode exif fam firefox flac fortran gdbm gif gpm gtk iconv ipv6 jpeg lcms ldap libnotify mad mmx mng modules mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp pam pango pcre pdf png policykit ppds qt3support qt4 readline sdl session spell sse sse2 ssl
Re: [gentoo-user] Python and man problems
On Mon, 4 August 2014, at 5:52 pm, Kerin Millar kerfra...@fastmail.co.uk wrote: ... most ... command not found. I've never heard of such a command. It looks as though PAGER=most is defined in your environment. In Gentoo, it is normally set as /usr/bin/less. It transpires that sys-apps/most is in portage. Emerging this package should fix it, if you intend to continue using it as the PAGER. most is the best manpager I've found found - the default syntax highlighting is perfect for man pages. You don't have to make it your default pager, you can set it separately: $ env | grep -i page PAGER=/usr/bin/less MANPAGER=/usr/bin/most $ Stroller.
Re: [gentoo-user] Re: Recommendations for scheduler
On 4 August 2014 15:35:41 CEST, Alan McKinnon alan.mckin...@gmail.com wrote: On 04/08/2014 15:31, Martin Vaeth wrote: J. Roeleveld jo...@antarean.org wrote: So you have a command which might break due to hardware error and cannot be rerun. I cannot see how any general-purpose scheduler might help you here: You either need to be able to split your command into several (sequential) commands or you need something adapted for your particular command. A general-purpose scheduler can work, as they do exist. I doubt that they can solve your problem. Let me repeat: You have a single program which accesses the database in a complex way and somewhere in the course of accessing it, the machine (or program) crashes. No general-purpose program can recover from this: You need particular knowledge of the database and the program if you even want to have a *chance* to recover from such a situation. A program with such a particular knowledge can hardly be called general-purpose. Joost, Either make the ETL tool pick up where it stopped and continue as it is the only that knows what it was doing and how far it got. Or, wrap the entire script in a single transaction. Alan, That would be the ideal solution. However, a single transaction dealing with around 500,000,000 rows will get me shot by the DBAs :) (Never mind that the performance of this will be such that having it all done by an office full of secretaries might be quicker.) Having the ETL process clever enough to be able to pick up from any point requires a degree of forward thinking and planning that is never done in real life. I would love to design it like that as it isn't too difficult. But I always get brought into these projects when implementing these structures will require a full rewrite and getting the original architects to admit their design can't be made restartable without human intervention. At which point the business simply says it is acceptable to have people do a manual rollback and restart the schedules from wherever it went wrong. I'm sure your wife has similar experiences as this is why these projects are always late to deliver and over budget. -- Joost -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: [gentoo-user] Re: Recommendations for scheduler
On 4 August 2014 15:31:40 CEST, Martin Vaeth mar...@mvath.de wrote: J. Roeleveld jo...@antarean.org wrote: So you have a command which might break due to hardware error and cannot be rerun. I cannot see how any general-purpose scheduler might help you here: You either need to be able to split your command into several (sequential) commands or you need something adapted for your particular command. A general-purpose scheduler can work, as they do exist. I doubt that they can solve your problem. Let me repeat: You have a single program which accesses the database in a complex way and somewhere in the course of accessing it, the machine (or program) crashes. No general-purpose program can recover from this: You need particular knowledge of the database and the program if you even want to have a *chance* to recover from such a situation. A program with such a particular knowledge can hardly be called general-purpose. The scheduler needs to be able to show which process failed/didn't finish. Then humans need to ensure that part finishes/reruns properly. Then humans need to be able to mark the failed process as succeeded. At which point the scheduler continues with the schedule(s) If, during one of these steps, the database or ETL process suffers a crash, the activities of the ETL process need to be rolled back to the point where you can restart it. I agree, but you need particular knowledge of the database and your tasks to do this which is far beyond the job of a scheduler. As already mentioned by someone in this thread, your problem needs to be solved on the level of the database (using snapshopt capabilities etc.) Or human intervention. Which requires a clear indication of where it went wrong and allows a simple action to continue the schedule from where it was after these humans solved the issues and ensure consistency. In order to deal with case 1., you can regularly (e.g. each minute) dump the output of schedule list (possibly suppressing non-important data through the options to keep it short). Or all the necessary information is kept in-sync on persistent storage. This would then also allow easy fail-over if the master-schedule-node fails No, it wouldn't, since jobs just finishing and wanting to report their status cannot do this when there is no server. You would need a rather involved protocol to deal with such situations dynamically. It can certainly be done, but it is not something which can easily be added as a feature: If this is required, it has to be the fundamental concept from the very beginning and everything else has to follow this first aim. You need different protocols than TCP sockets, to start with; something like dbus over IP with servers being able to announce their new presence, etc. I think it's doable with standard networking protocols. But, either you have a master server which controls everything. Or you have a master process which has failover functionality using classical distributed software techniques. These emails are actually quite useful as I am getting a clear pucture in my head on how I could approach this properly. Thanks, Joost -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: [gentoo-user] Re: Recommendations for scheduler
On 04/08/2014 21:46, J. Roeleveld wrote: On 4 August 2014 15:35:41 CEST, Alan McKinnon alan.mckin...@gmail.com wrote: On 04/08/2014 15:31, Martin Vaeth wrote: J. Roeleveld jo...@antarean.org wrote: So you have a command which might break due to hardware error and cannot be rerun. I cannot see how any general-purpose scheduler might help you here: You either need to be able to split your command into several (sequential) commands or you need something adapted for your particular command. A general-purpose scheduler can work, as they do exist. I doubt that they can solve your problem. Let me repeat: You have a single program which accesses the database in a complex way and somewhere in the course of accessing it, the machine (or program) crashes. No general-purpose program can recover from this: You need particular knowledge of the database and the program if you even want to have a *chance* to recover from such a situation. A program with such a particular knowledge can hardly be called general-purpose. Joost, Either make the ETL tool pick up where it stopped and continue as it is the only that knows what it was doing and how far it got. Or, wrap the entire script in a single transaction. Alan, That would be the ideal solution. You have the same concerns I do - how do you make a transaction around 500 million rows. So I asked the in-house expert - Mrs Alan :-) However, a single transaction dealing with around 500,000,000 rows will get me shot by the DBAs :) (Never mind that the performance of this will be such that having it all done by an office full of secretaries might be quicker.) She reckons an ETL job *must* be self-contained; if it isn't then it's broken by design. It must be idempotent too, which can be as simple as Truncate, Load, Commit Having the ETL process clever enough to be able to pick up from any point requires a degree of forward thinking and planning that is never done in real life. I would love to design it like that as it isn't too difficult. But I always get brought into these projects when implementing these structures will require a full rewrite and getting the original architects to admit their design can't be made restartable without human intervention. I agree with that design actually - it's the job of the hardware and OS guys to make stuff reliable that the application layer can rely on. When a SAN connection goes away, it usually comes back and the app layer just carries on (never mind that it retried 100 times meanwhile). Sometimes this doesn't work out. The easiest, cheapest and quickest way to handle it is to just restart the whole job from the beginning. This offends the engineer in us sometimes, but it really is the best way and all of Unix is built on this very idea :-) If the SAn goes away too often and it causes issues, the manybe the best approach is to get the SAN and facilities guys to get their act together At which point the business simply says it is acceptable to have people do a manual rollback and restart the schedules from wherever it went wrong. Exactly. One of the few cases where business has the correct idea. There's only some many pennies to spend and so many dollars to be delivered. I'm sure your wife has similar experiences as this is why these projects are always late to deliver and over budget. She says her projects are subject to the same universal inviolate rule as mine: time and cost is always best engineering estimate times pi We learn to deal with it. Which brings us back to Martin's initial statement: a scheduler cannot deal with any of this, the job itself must. It's an unpredictable event and schedulers can only deal with predictable events -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] Python and man problems
On Mon, 2014-08-04 at 19:00 +0100, Stroller wrote: most is the best manpager I've found found - the default syntax highlighting is perfect for man pages. Thanks for the tip! I hadn't come across `most`, and now since you pointed to it I've set it as my global MANPAGER. Cheers ;) -- wraeth wra...@wraeth.id.au signature.asc Description: This is a digitally signed message part
