Hello, I think your memory limit might be wrong (1024M ?) as the PHP code might need to store the file in memory. In general it should larger than postmaxsize / uploadmaxsize, so 4100 might be good value here.
Kind regards, Michael > On 13. Feb 2019, at 21:12, Kline, Charles <charles.kl...@lmco.com> wrote: > > Hi Guys, > Copying my server admin...Ed Shaver... > > Grrrrrrr!!! We are still unable to upload a file >2G. Apparently the user > can see it uploading and it gets to 100% (using Chrome) but then nothing...a > job ID never gets generated so obviously there is never anything to see for > the job under Show Jobs. > > We have done the following: > Updated /etc/php.ini as follows: > - post_max_size updated from 2048M to 4096M > - upload_max_filesize updated from 2018M to 4096M > - memory_limit updated from 702M to 1024M > Note: on the Upload a New File screen we do see the following line right > about #1 where you select the folder for storing the uploaded file..." Your > FOSSology server has imposed a maximum upload file size of 4096M bytes." > > The RAM has been updated from 8G to 16G. > [crkline@averhart /etc 113]% cat /proc/meminfo > MemTotal: 16329788 kB > MemFree: 7762536 kB > Buffers: 208940 kB > Cached: 6587228 kB > SwapCached: 0 kB > SwapTotal: 4193276 kB > SwapFree: 4193276 kB > > This is from the current 'top -u fossy'. As you can see TOTAL MEM=16,329,788k > and the USED= 8,494,540k with FREE=7,835,248k > top - 14:20:04 up 10:19, 2 users, load average: 0.00, 0.07, 0.22 > Tasks: 203 total, 1 running, 202 sleeping, 0 stopped, 0 zombie > Cpu(s): 0.1%us, 0.2%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st > Mem: 16329788k total, 8494540k used, 7835248k free, 207872k buffers > Swap: 4193276k total, 0k used, 4193276k free, 6518428k cached > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 2833 fossy 20 0 826m 2784 1956 S 0.0 0.0 0:00.94 fo_scheduler > > Fossoloy and Tomcat have been restarted a number of times and the server has > been rebooted. > > What am I missing?!?! > What would cause things just to stop after the file has uploaded...assuming > of course it has successfully uploaded which it would be nice to be able to > verify. > Is there any place I can look in fossology to try to see what is/isn't > happening? > > Any help would be appreciated. > Please note I will be out of office commencing today at 1600 returning Monday > morning at 0730. > > Thanks in advance... > Charlie > > Note: on the Upload a New File screen we do see the following line right > about #1 where you select the folder for storing the uploaded file..." Your > FOSSology server has imposed a maximum upload file size of 4096M bytes." > > > > > > > > -----Original Message----- > From: Kline, Charles (US) > Sent: Monday, February 4, 2019 4:07 PM > To: 'Michael C. Jaeger' <m...@mcj.de> > Cc: 'Jaeger, Michael C.' <michael.c.jae...@siemens.com>; 'Stangel, Dan' > <dan.stan...@hpe.com>; 'fossol...@fossology.org' <fossol...@fossology.org> > Subject: RE: EXTERNAL: Re: [FOSSology] Quick Question > > Any idea where the file is uploaded to (RAM or a specific location) from > which it is then unpacked? Will it unpack if it doesn't see enough RAM or > Storage to do the unpack? We just have no clue as to how an uploaded file is > processed from the perspective of where is it uploaded to...where is it > unpacked to before processing...etc. > > > charlie > > -----Original Message----- > From: Kline, Charles (US) > Sent: Monday, February 4, 2019 3:23 PM > To: 'Michael C. Jaeger' <m...@mcj.de> > Cc: 'Jaeger, Michael C.' <michael.c.jae...@siemens.com>; 'Stangel, Dan' > <dan.stan...@hpe.com>; 'fossol...@fossology.org' <fossol...@fossology.org> > Subject: RE: EXTERNAL: Re: [FOSSology] Quick Question > > Hi guys...this is killing me! > Another question on uploads...currently php.ini has been updated setting > upload_max_filesize and post_max_size to 4096M and on the Upload a New File > screen I see " FOSSology server has imposed a maximum upload file size of > 4096M bytes."...which is interesting because last week after making the > property updates I still saw it indicating "2048M bytes". I thought maybe a > server reboot had been done while I was out...but using 'uptime' indicates > the server has been up for 22 days. > > Anyways...the user tried uploading a 2.8G file using Chrome. When initiated > she saw the %complete in the lower left hand corner and the spinning thing at > the upper left hand corner which they always see. The upload reached 100% > and the spinning stopped but no job id was generated with the link you could > click on despite waiting a while. We are at a loss as to what is going > on...mostly from our general ignorance. Not sure if at this point it is > determining there isn't sufficient RAM to process the uploaded file (I am > currently trying to get the RAM increased on this box from 8G to 24G) or > what. There would appear to be plenty of actual storage available. Any > thoughts on what is going on at this point and why we are not processing the > file that appeared to upload successfully? > > Charlie > > -----Original Message----- > From: Kline, Charles (US) > Sent: Thursday, January 31, 2019 8:26 AM > To: 'Michael C. Jaeger' <m...@mcj.de> > Cc: 'Jaeger, Michael C.' <michael.c.jae...@siemens.com>; 'Stangel, Dan' > <dan.stan...@hpe.com>; 'fossol...@fossology.org' <fossol...@fossology.org> > Subject: RE: EXTERNAL: Re: [FOSSology] Quick Question > > Tried an upload using Chrome...I like that! Thanks...I have passed this on so > someone can test a large file using Chrome. > > Hey, any thoughts on this...this was posed by one of our administrators..." > The location where the file is uploaded to, does that have enough space to > read the file. I don’t know if fossoloy uploads to a local area first and > then puts the file in the define location so I can’t say where this location > would be." > > While trying to test last night with >2G files I was running 'top -u > fossy'...as usual the only running was fo_scheduler...but this also show MEM > and SWAP....the Available memory showed 8G...the used showed 7.xG with ~500M > freeI never saw the free space fall much below this and I often see these > numbers. Later when testing a very very small file I did see the Used in the > 5G range. > > Charlie > > -----Original Message----- > From: Kline, Charles (US) > Sent: Thursday, January 31, 2019 8:14 AM > To: 'Michael C. Jaeger' <m...@mcj.de> > Cc: Jaeger, Michael C. <michael.c.jae...@siemens.com>; Stangel, Dan > <dan.stan...@hpe.com>; fossol...@fossology.org > Subject: RE: EXTERNAL: Re: [FOSSology] Quick Question > > Hi Michael, > Yeah, big thing is we are trying to ascertain whether or not the file is > actually uploaded...certainly the test file I can see uploaded as a job > number was assigned and the ununpack started. I'll check out Chrome. > > charlie > > -----Original Message----- > From: Michael C. Jaeger [mailto:m...@mcj.de] > Sent: Thursday, January 31, 2019 7:10 AM > To: Kline, Charles (US) <charles.kl...@lmco.com> > Cc: Jaeger, Michael C. <michael.c.jae...@siemens.com>; Stangel, Dan > <dan.stan...@hpe.com>; fossol...@fossology.org > Subject: Re: EXTERNAL: Re: [FOSSology] Quick Question > > Hello, > > looks good. it appears also to me that a part of your waiting time appears to > be the upload / transfer time. Not all browsers show the upload progress, > Chrome does in the lower left corner. On some OSes you can see the network > traffic. I think this is maybe also another effect when dealing with VLP- > "very large packages" Maybe FOSSology could improve on this by having an > upload status dialogue to inform the user more precisely. > > I think the last two screenshots show the progress info after the file has > been uploaded to the FOSSology server. > > Kind regards, Michael > >> On 31. Jan 2019, at 09:53, Kline, Charles <charles.kl...@lmco.com> wrote: >> >> Hmmm, not sure I’m following you Michael….here’s some screenshots from >> a test I did… <image003.png><image011.jpg> CLICK UPLOAD… >> <image012.png><image004.png> Then I will see this…but this is after >> the uploaded has been completed and the ununpack has started as shown >> below… <image014.png><image008.png> <image015.png><image010.png> I >> guess I was hoping there was a file or log on the fossology server that one >> could tail to see the progress or at least repeatedly do a listing to see >> the file growth. >> >> Charlie >> >> From: Jaeger, Michael C. [mailto:michael.c.jae...@siemens.com] >> Sent: Thursday, January 31, 2019 1:50 AM >> To: Kline, Charles (US) <charles.kl...@lmco.com>; Michael C. Jaeger >> <m...@mcj.de> >> Cc: Stangel, Dan <dan.stan...@hpe.com>; fossol...@fossology.org >> Subject: RE: EXTERNAL: Re: [FOSSology] Quick Question >> >> Hi, >> >> If you upload a file, there will be a link presented in the “message bar” >> right below the menus with the number (id) of the upload. Clicking this >> number will show you a table with the progress. >> >> If you have missed that you could click on “Jobs” in the top menu bar, as >> for the options choose “my recent jobs”, it should show what is currently >> ongoing. >> >> Kind regards, Michael >> >> From: Kline, Charles [mailto:charles.kl...@lmco.com] >> Sent: Donnerstag, 31. Januar 2019 00:53 >> To: Jaeger, Michael C. (CT RDA SSI); Michael C. Jaeger >> Cc: Stangel, Dan; fossol...@fossology.org >> Subject: RE: EXTERNAL: Re: [FOSSology] Quick Question >> >> Hi, >> Is there a way to monitor (whether it be via the fossology GUI or on the >> fossology server) the progress of the upload of a file? Is there a log or >> something that can be monitored? In a previous test with a 2G file it took >> ~10 min to upload. I don’t know if this is a linear thing where a 4G file >> will take 20 minutes, but it would certainly be nice if one could tell how >> far along the upload was so you don’t abort the test prematurely because you >> felt you had given it enough time. >> >> Thanks in advance!!!! >> >> Charlie >> >> From: Jaeger, Michael C. [mailto:michael.c.jae...@siemens.com] >> Sent: Tuesday, January 22, 2019 7:36 AM >> To: Kline, Charles (US) <charles.kl...@lmco.com>; Michael C. Jaeger >> <m...@mcj.de> >> Cc: Stangel, Dan <dan.stan...@hpe.com>; fossol...@fossology.org >> Subject: EXTERNAL: RE: EXTERNAL: Re: [FOSSology] Just curious...question >> about your installation... >> >> Hello, >> >> uploading 15GB in a single upload is not advisable in any case, since it >> will lead to speed issues when browsing the source in an aggregated view. >> >> 15GB of compressed source code will be like … 100GB of uncompressed files? >> that will be that will be maybe like >1000 packages into a single upload? >> maybe magnitude of 1 mio files in one upload? -> that will lead to endlessly >> running queries in the aggregated view unless you are really good at tuning >> your postgresql server. >> >> I would highly recommend to consider splitting your 15GB upload into 30 >> packages or so. >> >> Kind regards, Michael >> >> >> >> From: fossology@lists.fossology.org >> [mailto:fossology@lists.fossology.org] On Behalf Of Kline, Charles >> Sent: Montag, 21. Januar 2019 20:10 >> To: Michael C. Jaeger >> Cc: Stangel, Dan; fossol...@fossology.org >> Subject: Re: EXTERNAL: Re: [FOSSology] Just curious...question about your >> installation... >> >> Hi Michael, >> Yeah, this could certainly be an issue...they would like to be able to >> process a 15G file and I don’t think that is going to happen…if we >> could get 4G that would be nice, but based on the info below that may >> not even be possible. I’ll be testing tonight with the following >> php.ini properties updated to 4096M but I’m not holding out much hope… >> >> Maximum allowed size for uploaded files. >> upload_max_filesize = 2048M >> Maximum size of POST data that PHP will accept. >> post_max_size = 2048M >> >> Excerpt from a 'lscpu': >> Architecture: x86_64 >> CPU op-mode(s): 32-bit, 64-bit >> CPU(s): 4 >> >> If I do a ‘cat /proc/meminfo’: >> MemTotal: 8056932 kB >> MemFree: 430432 kB >> >> And not much running right now…results of ‘top –u fossy’, seems like >> we might be up against the limit now… top - 13:29:18 up 8 days, 20:01, 1 >> user, load average: 1.00, 1.38, 1.81 >> Tasks: 210 total, 2 running, 208 sleeping, 0 stopped, 0 zombie >> Cpu(s): 3.3%us, 0.8%sy, 0.0%ni, 94.8%id, 1.1%wa, 0.0%hi, 0.1%si, >> 0.0%st >> Mem: 8056932k total, 7627692k used, 429240k free, 17668k buffers >> Swap: 4193276k total, 157492k used, 4035784k free, 5471620k cached >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> 17259 fossy 20 0 99.1m 30m 2820 R 92.3 0.4 10:01.71 nomos >> 7075 fossy 20 0 836m 2960 1668 S 0.0 0.0 1:48.17 fo_scheduler >> >> Charlie >> >> -----Original Message----- >> From: Michael C. Jaeger [mailto:m...@mcj.de] >> Sent: Friday, January 18, 2019 5:29 AM >> To: Kline, Charles (US) <charles.kl...@lmco.com> >> Cc: Stangel, Dan <dan.stan...@hpe.com>; fossol...@fossology.org >> Subject: EXTERNAL: Re: [FOSSology] Just curious...question about your >> installation... >> >> Hello, >> >> I think as for the PHP is it the question how much RAM you have (and >> ... do you run a 32-bit OS?) >> >> restarting apache depends on your linux distro, in some cases >> >> sudo service apache2 restart >> >> or a tool from the apache web server software could also help (which >> could be installed on your system, I do not know) >> >> sudo apachectl restart >> >> in other cases google will help. Reading into apachectl can help you >> to do a more careful handling, such as >> >> apachectl configtest >> apachectl graceful >> >> ... >> >> Kind regards, Michael >> >>> On 16. Jan 2019, at 20:07, Kline, Charles <charles.kl...@lmco.com> wrote: >>> >>> Just curious...what you have in your pkp.ini file for these properties...or >>> I guess I should ask can you process files greater than 2G? >>> ; Maximum allowed size for uploaded files. >>> upload_max_filesize = 2048M >>> >>> ; Maximum size of POST data that PHP will accept. >>> post_max_size = 2048M >>> >>> Oh, do you know how to restart Apache??? Trying to figure that out. >>> >>> Charlie >>> >>> -----Original Message----- >>> From: Kline, Charles (US) >>> Sent: Wednesday, January 9, 2019 2:31 PM >>> To: 'Stangel, Dan' <dan.stan...@hpe.com>; fossol...@fossology.org >>> Subject: RE: Question on version 2.5.0 >>> >>> Thanks so much Dan!!!! >>> >>> -----Original Message----- >>> From: Stangel, Dan [mailto:dan.stan...@hpe.com] >>> Sent: Wednesday, January 9, 2019 1:52 PM >>> To: Kline, Charles (US) <charles.kl...@lmco.com>; >>> fossol...@fossology.org >>> Subject: EXTERNAL: RE: Question on version 2.5.0 >>> >>> Hi Charlie, >>> >>> Ah, the joys of a large enterprise IT organization. You ask a lot of good >>> questions, and I would probably defer to other experts on this list for >>> most of them, but I definitely can say that the delete agent has to run >>> exclusively, by design, and that it can be rather slow. So your users' >>> experiences are not surprising, and I'm not sure there's a good solution. >>> Upgrading to a newer version may resolve some of these issues, but not sure >>> what the team has done around delagent in the past few releases. >>> >>> As for maintenance tasks, most of these should be pretty reliable, but to >>> your point, I'm not sure what impact they have on running jobs. You would >>> definitely want to run the vacuum analyze job somewhat routinely, for the >>> health and well-being of Postgres. I think in our configuration it is set >>> up as a cron job. >>> >>> Dan >>> >>> >>> From: Kline, Charles [mailto:charles.kl...@lmco.com] >>> Sent: Wednesday, January 09, 2019 11:06 AM >>> To: Stangel, Dan <dan.stan...@hpe.com>; fossol...@fossology.org >>> Subject: RE: Question on version 2.5.0 >>> >>> Dan, >>> At your convenience if you had any thoughts on the below I would appreciate >>> it....having said that I'm aware that I'm kinda taking advantage of you at >>> this point so please feel free to decline. I'm sure you have plenty on your >>> plate. >>> >>> Back in Nov (I was out for surgery) the users reported the following: >>> To the two server admins who kinda support the fossology server from one of >>> the main fossology users: (apparently user was trying to do some >>> deletes.assumption is that 'regular' fossology activity was taking place) >>> Can one of you folks take a look at the Fossology server? >>> It's just kind of sitting there doing nothing. >>> >>> Admin responded that postgresql and fossology had been restarted. >>> >>> Subsequent feedback from user. >>> I tried deletes again, but it's not processing them. It's not even starting >>> the job. >>> And subsequently.from another super user. >>> >>> I had a big job running that didn't finish until 11/14/2018 @ 01:13 AM. >>> >>> As near as I can tell the delete jobs will wait for current jobs to >>> complete their current step, then pause the job(s), then the delete job(s) >>> executes, then the other jobs continue on with their next step. >>> >>> Your deletion jobs have completed last night around 9:54 pm just around the >>> time my job completed the ununpack step. >>> >>> >>> >>> This kicked off talk (from a server admin who I have never worked with) of >>> updating fossology to the latest version and/or moving it to another >>> server. This was met with some push back.they finally decided to wait until >>> I returned to work.which wasn't until 12/14.I talked to the one super user >>> trying to get a sense of where things stood.she insisted that there were >>> still issues with trying to do deletes during the week and that they were >>> doing them on the weekends. >>> >>> Meeting was held on 1/7 to discuss. Two main issues. >>> 1. The large upload file size, which is being addressed. >>> 2. The issue with doing the deletes and things freezing up. >>> >>> In regards to issue 2, I definitely didn't get a sense that the super user >>> was particularly interested in upgrading fossology and in regards to >>> getting a new server we now have a mandate that anything new has to be in >>> the cloud.and at this point that is not an easy process. >>> >>> Based on some of my research I ran into the Exclusive and Nokill >>> settings.as seen in >>> https://github.com/fossology/fossology/wiki/Job-Schedulerand if I look at >>> /app/fossology/fossology_2.5/share/fossology I see the following.not sure >>> if any of this is relevant to the above. >>> Ununpack >>> ; Directives: >>> ; EXCLUSIVE: the agent cannot run concurrently with any other agent. >>> special[] = NOKILL >>> >>> buckets >>> ; Directives: >>> ; EXCLUSIVE: the agent cannot run concurrently with any other agent. >>> special[]= >>> >>> copyright >>> ; Directives: >>> ; EXCLUSIVE: the agent cannot run concurrently with any other agent. >>> special[] = >>> >>> nomos >>> ; Directives: >>> ; EXCLUSIVE: the agent cannot run concurrently with any other agent. >>> special[] = >>> >>> pkgagent >>> ; Directives: >>> ; EXCLUSIVE: the agent cannot run concurrently with any other agent. >>> special[] = >>> >>> delagent.conf >>> ; Directives: >>> ; EXCLUSIVE: the agent cannot run concurrently with any other agent. >>> special[] = EXCLUSIVE >>> special[] = NOKILL >>> >>> >>> Also, wondering what your experience is with running any of the maintenance >>> tasks.any comments would be appreciated. >>> >>> I am a little hesitant running these since I am not 100% sure what they do >>> or impact to the application when they are running.the ones I have >>> considered running are "Remove uploads with no pfiles", "Remove orphaned >>> temp tables" and "Vacuum Analyze the database". >>> >>> I am retiring later this year and hope to scrape this off my shoe sooner >>> than later. >>> >>> Thanks in advance for your time and consideration. >>> >>> Charlie >>> >>> -----Original Message----- >>> From: Kline, Charles (US) >>> Sent: Wednesday, January 9, 2019 11:44 AM >>> To: 'Stangel, Dan' <dan.stan...@hpe.com>; fossol...@fossology.org >>> Subject: RE: Question on version 2.5.0 >>> >>> Dan, >>> Thanks so much for your response...to be honest I didn't think I would get >>> one. >>> >>> Upon further diffing I tracked down /etc/php.ini which looks to what needs >>> to be updated. I was being told that only post_max_size needed to be >>> updated but totally agree that upload_max_filesize also should be updated. >>> Trick now would be to get sudo to enable me to do this or find an admin who >>> can copy an updated version I made into /etc. >>> >>> In regards to restarting Apache...again dumb question, but does >>> apache get restarted when the scheduler is restarted or much it be >>> restarted separately? Frankly I have never done anything with >>> apache. (and the big unfortunately is when I was given fossology to >>> support I was told not to spend any time on it learning anything but >>> just to respond to issues as they arise...so every time there is an >>> issue it's an >>> adventure!) >>> >>> My plan would be to initially increase the size of these 2 properties to >>> 4096M, restart the scheduler(fossology), and test. If OK, update the >>> properties to a size to accommodate the large file they are trying to >>> process and repeat the above. I don't think I'll leave it at that size, >>> 16384M, but would probably put it back to 4096M. >>> >>> Thanks!!!!! >>> Charlie >>> >>> -----Original Message----- >>> From: Stangel, Dan [mailto:dan.stan...@hpe.com] >>> Sent: Wednesday, January 9, 2019 11:33 AM >>> To: Kline, Charles (US) <charles.kl...@lmco.com>; >>> fossol...@fossology.org >>> Subject: EXTERNAL: RE: Question on version 2.5.0 >>> >>> Hi Charlie, >>> >>> On 2018-01-08, Kline, Charles wrote: >>>> Yeah, I know...why are we still using 2.5.0...long story. >>> >>> Don't feel too bad - we're still on 2.6.2 for our production >>> instance (again, long story.. Suffice it to say these releases have >>> a long >>> lifespan) >>> >>>> Question....users want to process/upload files greater than 2GB which >>>> my users are saying is the largest then can process. Trying to >>>> figure out the correct place to update this value. >>>> >>>> In >>>> /app/fossology/fossology_2.5/share/fossology/lib/php/common-scheduler. >>>> php >>>> there is the following... >>>> common-scheduler.php: \param $MaxSize - optional max read size, >>>> default is 2048 common-scheduler.php:function >>>> fo_scheduler_read($SchedObj, $MaxSize=2048) I assume this is in MB >>>> >>>> But in the fossology online info I am seeing this...assuming these >>>> are values for the latest version of fossology...in the link they >>>> talk about updating php.ini for >>>> apache...(http://archive15.fossology.org/projects/fossology/wiki/Sy >>>> sC >>>> o >>>> nfig/annotate/9) >>>> (...) >>>> So trying to figure out where I actually need to make an update to >>>> change the size of the file that can be uploaded/processed by the users. >>> >>> You should only need to update the upload_max_filesize and >>> post_max_size parameters in your php.ini file. In our case for >>> example, we have >>> >>> ; Maximum allowed size for uploaded files. >>> ; http://php.net/upload-max-filesize >>> upload_max_filesize = 4096M >>> >>> ; Maximum size of POST data that PHP will accept. >>> ; http://php.net/post-max-size >>> post_max_size = 4096M >>> >>> Once you have updated php.ini, just be sure to restart Apache to pick up >>> the changes. Then try to do a > 2G upload from file (ideally from a client >>> that isn't too far away, network-wise) to verify that it works. >>> >>> Dan >>> >>> >>> >>> >> >> > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#3230): https://lists.fossology.org/g/fossology/message/3230 Mute This Topic: https://lists.fossology.org/mt/29603806/21656 Group Owner: fossology+ow...@lists.fossology.org Unsubscribe: https://lists.fossology.org/g/fossology/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
