Hi Peter,
Thanks for the detailed description of your upload issue, it helps :)
One big issue with 13GB files is the 32bits limit for an Integer. You need
to make sure all your JVMs are 64 bits, and the web server you use (which
one?) is not using "int" for content-length.
Anyway, I think that the issue you encounter is on the Ant/Ivy client memory
usage. From our experience the HTTP client used by Ivy is not using the JVM
heap during stream efficiently (huge buffer and re-read for checksum
calculations).
We have quite good success with deploying very large files using our Ivy
plugin. You can easily use it with the build server integrations: Jenkins,
Team City, Bamboo, or follow the instructions from Tomer for a standalone
run:
"
Please follow these instructions to have build info and license checks work
on a standalone Ivy build. Please note that these instructions are per the
current Artifactory version and may change in future releases, so using the
plugin is still the best and supported option.
1. Download the following jar
http://repo.jfrog.org/artifactory/libs-snapshots-local/org/jfrog/buildinfo/build-info-extractor-ivy/1.1.x-SNAPSHOT/build-info-extractor-ivy-1.1.x-SNAPSHOT-uber.jar
and
place it on your ANT_HOME/lib folder.
2. Create a properties file similar to the attached one. This will be your
manual configuration file for how the Ivy build communicate with
Artifactory.
3. Point at the properties file by appending to your ANT_OPTS environment
variable: -DbuildInfoConfig.propertiesFile=/PATH/TO/buildinfo.properties and
a
-listener org.jfrog.build.extractor.listener.ArtifactoryBuildListener
4. Run an ANT task that publishes your project to a *local* repository. The
ANT build will be intercepted and publishing to Artifactory should happen as
a side effect, including license analysis of all your artifacts and
dependencies.
Hopefully this will get you going :)
Tomer Cohen
JFrog Support
"
Hope it helps,
Fred Simon
JFrog Support
On Fri, Mar 25, 2011 at 10:38 PM, pkline
<[email protected]>wrote:
> So basically our configuration is as follows:
>
> We upload various sized tar files into artifactory from our Ivy build
> system. The smallest being about 200K and the largest up until now was
> 1GB.
> Now we have a new product in our pipeline and the tar file is 13GB. I have
> followed all of the recommended steps (at least I think I did).
>
> 1) Moved from derby to mysql filesystem for the repo.
> artifactory.jcr.configDir=repo/filesystem-mysql
> 2) nuked my data directory
> 3) modified /etc/mysql/my.cfg and changed to various sizes but it now
> stands
> at
> max_allowed_packet = 20480M
> 4) Restarted everything multiple times. I can upload files around 1GB but
> no go on the large one.
>
>
> As you can see the java ant task is taking up some serious resources when
> looking in top:
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 6273 peter_kl 18 -2 8501m 7.4g 9404 S 322 23.4 99:40.37 java
>
> Output from Ant/Ivy publish command:
> [ivy:publish] options = status=release pubdate=Fri Mar 25 15:23:36 EDT
> 2011 validate=true resolveDynamicRevisions=true merge=true resolveId=null
> pubBranch=null
> [ivy:publish] pre 1.3 ivy file: using exactOrRegexp as default matcher
> [ivy:publish] delivering ivy file to
> /ivyart/root/projects/rootfs/.artifact/ivy
> [ivy:publish] deliver done (1730ms)
> [ivy:publish] :: publishing :: com.an.tos#rootfs
> [ivy:publish] validate = true
> [ivy:publish] pre 1.3 ivy file: using exactOrRegexp as default matcher
>
>
> When I get to the last line of the publish, if I do a dstat on the system
> to
> see if its writing lots of data the disk is practially idle the whole time.
> I ran this for 40 minutes before quitting and never saw anything that lead
> me to believe it was actually copying the large tar file anywhere. Now I
> know this works just fine because I can see my other tar files in
> artifactory.
>
> TIA. -Peter
>
>
>
> --
> View this message in context:
> http://forums.jfrog.org/Cannot-upload-13GB-file-into-Artifactory-tp6209115p6209115.html
> Sent from the Artifactory - Users mailing list archive at Nabble.com.
>
>
> ------------------------------------------------------------------------------
> Enable your software for Intel(R) Active Management Technology to meet the
> growing manageability and security demands of your customers. Businesses
> are taking advantage of Intel(R) vPro (TM) technology - will your software
> be a part of the solution? Download the Intel(R) Manageability Checker
> today! http://p.sf.net/sfu/intel-dev2devmar
> _______________________________________________
> Artifactory-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>
--
Chief Architect
JFrog Ltd
http://www.jfrog.org/
http://twitter.com/freddy33
------------------------------------------------------------------------------
Enable your software for Intel(R) Active Management Technology to meet the
growing manageability and security demands of your customers. Businesses
are taking advantage of Intel(R) vPro (TM) technology - will your software
be a part of the solution? Download the Intel(R) Manageability Checker
today! http://p.sf.net/sfu/intel-dev2devmar
_______________________________________________
Artifactory-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/artifactory-users
