Hi, If you upload a file, there will be a link presented in the "message bar" right below the menus with the number (id) of the upload. Clicking this number will show you a table with the progress.
If you have missed that you could click on "Jobs" in the top menu bar, as for the options choose "my recent jobs", it should show what is currently ongoing. Kind regards, Michael From: Kline, Charles [mailto:charles.kl...@lmco.com] Sent: Donnerstag, 31. Januar 2019 00:53 To: Jaeger, Michael C. (CT RDA SSI); Michael C. Jaeger Cc: Stangel, Dan; fossol...@fossology.org Subject: RE: EXTERNAL: Re: [FOSSology] Quick Question Hi, Is there a way to monitor (whether it be via the fossology GUI or on the fossology server) the progress of the upload of a file? Is there a log or something that can be monitored? In a previous test with a 2G file it took ~10 min to upload. I don't know if this is a linear thing where a 4G file will take 20 minutes, but it would certainly be nice if one could tell how far along the upload was so you don't abort the test prematurely because you felt you had given it enough time. Thanks in advance!!!! Charlie From: Jaeger, Michael C. [mailto:michael.c.jae...@siemens.com] Sent: Tuesday, January 22, 2019 7:36 AM To: Kline, Charles (US) <charles.kl...@lmco.com>; Michael C. Jaeger <m...@mcj.de> Cc: Stangel, Dan <dan.stan...@hpe.com>; fossol...@fossology.org Subject: EXTERNAL: RE: EXTERNAL: Re: [FOSSology] Just curious...question about your installation... Hello, uploading 15GB in a single upload is not advisable in any case, since it will lead to speed issues when browsing the source in an aggregated view. 15GB of compressed source code will be like ... 100GB of uncompressed files? that will be that will be maybe like >1000 packages into a single upload? maybe magnitude of 1 mio files in one upload? -> that will lead to endlessly running queries in the aggregated view unless you are really good at tuning your postgresql server. I would highly recommend to consider splitting your 15GB upload into 30 packages or so. Kind regards, Michael From: fossology@lists.fossology.org<mailto:fossology@lists.fossology.org> [mailto:fossology@lists.fossology.org] On Behalf Of Kline, Charles Sent: Montag, 21. Januar 2019 20:10 To: Michael C. Jaeger Cc: Stangel, Dan; fossol...@fossology.org<mailto:fossol...@fossology.org> Subject: Re: EXTERNAL: Re: [FOSSology] Just curious...question about your installation... Hi Michael, Yeah, this could certainly be an issue...they would like to be able to process a 15G file and I don't think that is going to happen...if we could get 4G that would be nice, but based on the info below that may not even be possible. I'll be testing tonight with the following php.ini properties updated to 4096M but I'm not holding out much hope... Maximum allowed size for uploaded files. upload_max_filesize = 2048M Maximum size of POST data that PHP will accept. post_max_size = 2048M Excerpt from a 'lscpu': Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit CPU(s): 4 If I do a 'cat /proc/meminfo': MemTotal: 8056932 kB MemFree: 430432 kB And not much running right now...results of 'top -u fossy', seems like we might be up against the limit now... top - 13:29:18 up 8 days, 20:01, 1 user, load average: 1.00, 1.38, 1.81 Tasks: 210 total, 2 running, 208 sleeping, 0 stopped, 0 zombie Cpu(s): 3.3%us, 0.8%sy, 0.0%ni, 94.8%id, 1.1%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 8056932k total, 7627692k used, 429240k free, 17668k buffers Swap: 4193276k total, 157492k used, 4035784k free, 5471620k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17259 fossy 20 0 99.1m 30m 2820 R 92.3 0.4 10:01.71 nomos 7075 fossy 20 0 836m 2960 1668 S 0.0 0.0 1:48.17 fo_scheduler Charlie -----Original Message----- From: Michael C. Jaeger [mailto:m...@mcj.de] Sent: Friday, January 18, 2019 5:29 AM To: Kline, Charles (US) <charles.kl...@lmco.com<mailto:charles.kl...@lmco.com>> Cc: Stangel, Dan <dan.stan...@hpe.com<mailto:dan.stan...@hpe.com>>; fossol...@fossology.org<mailto:fossol...@fossology.org> Subject: EXTERNAL: Re: [FOSSology] Just curious...question about your installation... Hello, I think as for the PHP is it the question how much RAM you have (and ... do you run a 32-bit OS?) restarting apache depends on your linux distro, in some cases sudo service apache2 restart or a tool from the apache web server software could also help (which could be installed on your system, I do not know) sudo apachectl restart in other cases google will help. Reading into apachectl can help you to do a more careful handling, such as apachectl configtest apachectl graceful ... Kind regards, Michael > On 16. Jan 2019, at 20:07, Kline, Charles > <charles.kl...@lmco.com<mailto:charles.kl...@lmco.com>> wrote: > > Just curious...what you have in your pkp.ini file for these properties...or I > guess I should ask can you process files greater than 2G? > ; Maximum allowed size for uploaded files. > upload_max_filesize = 2048M > > ; Maximum size of POST data that PHP will accept. > post_max_size = 2048M > > Oh, do you know how to restart Apache??? Trying to figure that out. > > Charlie > > -----Original Message----- > From: Kline, Charles (US) > Sent: Wednesday, January 9, 2019 2:31 PM > To: 'Stangel, Dan' <dan.stan...@hpe.com<mailto:dan.stan...@hpe.com>>; > fossol...@fossology.org<mailto:fossol...@fossology.org> > Subject: RE: Question on version 2.5.0 > > Thanks so much Dan!!!! > > -----Original Message----- > From: Stangel, Dan [mailto:dan.stan...@hpe.com] > Sent: Wednesday, January 9, 2019 1:52 PM > To: Kline, Charles (US) > <charles.kl...@lmco.com<mailto:charles.kl...@lmco.com>>; > fossol...@fossology.org<mailto:fossol...@fossology.org> > Subject: EXTERNAL: RE: Question on version 2.5.0 > > Hi Charlie, > > Ah, the joys of a large enterprise IT organization. You ask a lot of good > questions, and I would probably defer to other experts on this list for most > of them, but I definitely can say that the delete agent has to run > exclusively, by design, and that it can be rather slow. So your users' > experiences are not surprising, and I'm not sure there's a good solution. > Upgrading to a newer version may resolve some of these issues, but not sure > what the team has done around delagent in the past few releases. > > As for maintenance tasks, most of these should be pretty reliable, but to > your point, I'm not sure what impact they have on running jobs. You would > definitely want to run the vacuum analyze job somewhat routinely, for the > health and well-being of Postgres. I think in our configuration it is set up > as a cron job. > > Dan > > > From: Kline, Charles [mailto:charles.kl...@lmco.com] > Sent: Wednesday, January 09, 2019 11:06 AM > To: Stangel, Dan <dan.stan...@hpe.com<mailto:dan.stan...@hpe.com>>; > fossol...@fossology.org<mailto:fossol...@fossology.org> > Subject: RE: Question on version 2.5.0 > > Dan, > At your convenience if you had any thoughts on the below I would appreciate > it....having said that I'm aware that I'm kinda taking advantage of you at > this point so please feel free to decline. I'm sure you have plenty on your > plate. > > Back in Nov (I was out for surgery) the users reported the following: > To the two server admins who kinda support the fossology server from one of > the main fossology users: (apparently user was trying to do some > deletes.assumption is that 'regular' fossology activity was taking place) Can > one of you folks take a look at the Fossology server? > It's just kind of sitting there doing nothing. > > Admin responded that postgresql and fossology had been restarted. > > Subsequent feedback from user. > I tried deletes again, but it's not processing them. It's not even starting > the job. > And subsequently.from another super user. > > I had a big job running that didn't finish until 11/14/2018 @ 01:13 AM. > > As near as I can tell the delete jobs will wait for current jobs to complete > their current step, then pause the job(s), then the delete job(s) executes, > then the other jobs continue on with their next step. > > Your deletion jobs have completed last night around 9:54 pm just around the > time my job completed the ununpack step. > > > > This kicked off talk (from a server admin who I have never worked with) of > updating fossology to the latest version and/or moving it to another server. > This was met with some push back.they finally decided to wait until I > returned to work.which wasn't until 12/14.I talked to the one super user > trying to get a sense of where things stood.she insisted that there were > still issues with trying to do deletes during the week and that they were > doing them on the weekends. > > Meeting was held on 1/7 to discuss. Two main issues. > 1. The large upload file size, which is being addressed. > 2. The issue with doing the deletes and things freezing up. > > In regards to issue 2, I definitely didn't get a sense that the super user > was particularly interested in upgrading fossology and in regards to getting > a new server we now have a mandate that anything new has to be in the > cloud.and at this point that is not an easy process. > > Based on some of my research I ran into the Exclusive and Nokill settings.as > seen in https://github.com/fossology/fossology/wiki/Job-Scheduler and if I > look at /app/fossology/fossology_2.5/share/fossology I see the following.not > sure if any of this is relevant to the above. > Ununpack > ; Directives: > ; EXCLUSIVE: the agent cannot run concurrently with any other agent. > special[] = NOKILL > > buckets > ; Directives: > ; EXCLUSIVE: the agent cannot run concurrently with any other agent. > special[]= > > copyright > ; Directives: > ; EXCLUSIVE: the agent cannot run concurrently with any other agent. > special[] = > > nomos > ; Directives: > ; EXCLUSIVE: the agent cannot run concurrently with any other agent. > special[] = > > pkgagent > ; Directives: > ; EXCLUSIVE: the agent cannot run concurrently with any other agent. > special[] = > > delagent.conf > ; Directives: > ; EXCLUSIVE: the agent cannot run concurrently with any other agent. > special[] = EXCLUSIVE > special[] = NOKILL > > > Also, wondering what your experience is with running any of the maintenance > tasks.any comments would be appreciated. > > I am a little hesitant running these since I am not 100% sure what they do or > impact to the application when they are running.the ones I have considered > running are "Remove uploads with no pfiles", "Remove orphaned temp tables" > and "Vacuum Analyze the database". > > I am retiring later this year and hope to scrape this off my shoe sooner than > later. > > Thanks in advance for your time and consideration. > > Charlie > > -----Original Message----- > From: Kline, Charles (US) > Sent: Wednesday, January 9, 2019 11:44 AM > To: 'Stangel, Dan' <dan.stan...@hpe.com<mailto:dan.stan...@hpe.com>>; > fossol...@fossology.org<mailto:fossol...@fossology.org> > Subject: RE: Question on version 2.5.0 > > Dan, > Thanks so much for your response...to be honest I didn't think I would get > one. > > Upon further diffing I tracked down /etc/php.ini which looks to what needs to > be updated. I was being told that only post_max_size needed to be updated but > totally agree that upload_max_filesize also should be updated. Trick now > would be to get sudo to enable me to do this or find an admin who can copy an > updated version I made into /etc. > > In regards to restarting Apache...again dumb question, but does apache > get restarted when the scheduler is restarted or much it be restarted > separately? Frankly I have never done anything with apache. (and the > big unfortunately is when I was given fossology to support I was told > not to spend any time on it learning anything but just to respond to > issues as they arise...so every time there is an issue it's an > adventure!) > > My plan would be to initially increase the size of these 2 properties to > 4096M, restart the scheduler(fossology), and test. If OK, update the > properties to a size to accommodate the large file they are trying to process > and repeat the above. I don't think I'll leave it at that size, 16384M, but > would probably put it back to 4096M. > > Thanks!!!!! > Charlie > > -----Original Message----- > From: Stangel, Dan [mailto:dan.stan...@hpe.com] > Sent: Wednesday, January 9, 2019 11:33 AM > To: Kline, Charles (US) > <charles.kl...@lmco.com<mailto:charles.kl...@lmco.com>>; > fossol...@fossology.org<mailto:fossol...@fossology.org> > Subject: EXTERNAL: RE: Question on version 2.5.0 > > Hi Charlie, > > On 2018-01-08, Kline, Charles wrote: >> Yeah, I know...why are we still using 2.5.0...long story. > > Don't feel too bad - we're still on 2.6.2 for our production instance > (again, long story.. Suffice it to say these releases have a long > lifespan) > >> Question....users want to process/upload files greater than 2GB which >> my users are saying is the largest then can process. Trying to >> figure out the correct place to update this value. >> >> In >> /app/fossology/fossology_2.5/share/fossology/lib/php/common-scheduler. >> php >> there is the following... >> common-scheduler.php: \param $MaxSize - optional max read size, >> default is 2048 common-scheduler.php:function >> fo_scheduler_read($SchedObj, $MaxSize=2048) I assume this is in MB >> >> But in the fossology online info I am seeing this...assuming these >> are values for the latest version of fossology...in the link they >> talk about updating php.ini for >> apache...(http://archive15.fossology.org/projects/fossology/wiki/SysC >> o >> nfig/annotate/9) >> (...) >> So trying to figure out where I actually need to make an update to >> change the size of the file that can be uploaded/processed by the users. > > You should only need to update the upload_max_filesize and > post_max_size parameters in your php.ini file. In our case for > example, we have > > ; Maximum allowed size for uploaded files. > ; http://php.net/upload-max-filesize > upload_max_filesize = 4096M > > ; Maximum size of POST data that PHP will accept. > ; http://php.net/post-max-size > post_max_size = 4096M > > Once you have updated php.ini, just be sure to restart Apache to pick up the > changes. Then try to do a > 2G upload from file (ideally from a client that > isn't too far away, network-wise) to verify that it works. > > Dan > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#3214): https://lists.fossology.org/g/fossology/message/3214 Mute This Topic: https://lists.fossology.org/mt/29603806/21656 Group Owner: fossology+ow...@lists.fossology.org Unsubscribe: https://lists.fossology.org/g/fossology/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
