The connection abort here is actually on a download request. 2.1.3 had a bug
that can lead to this timeout so upgrading may improve things for you.
Also, can you check that the connection/socket timeout configured in the
Hudson plugin is not too short? It should be at least 60 seconds for big
builds. Depending on the version of the plugin the value would be in either
milliseconds or seconds in latest versions (the label and help should
indicate the unit - we take care of updating the value automatically upon
upgrading the plugin).
Yoav
On Thu, Jul 8, 2010 at 5:05 AM, Clarence <[email protected]> wrote:
> Also what I forgot to mention that my artifactory is serving around 9600
> artifacts. I
>
> When our nightly builds run the call to artifactory is
> http://server:8585/artifactory/.
>
>
> To answer your ulimit question it is set to 1024.
>
>
> Here's the error in my artifactory.log file from just a build
>
> 2010-07-07 19:55:28,112 [TP-Processor10] [WARN ]
> (o.a.w.s.HttpArtifactoryResponse:126) - Failed to commit http response
> (null).
> org.apache.catalina.connector.ClientAbortException: null
> at
> org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:370)
> [na:na]
> at
> org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:432) [na:na]
> at
> org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:321)
> [na:na]
> at
> org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:295)
> [na:na]
> at
> org.apache.catalina.connector.Response.flushBuffer(Response.java:545)
> [na:na]
> at
> org.apache.catalina.connector.ResponseFacade.flushBuffer(ResponseFacade.java:277)
> [na:na]
> at
> javax.servlet.ServletResponseWrapper.flushBuffer(ServletResponseWrapper.java:166)
> [servlet-api.jar:2.4.public_draft]
> at
> org.artifactory.webapp.servlet.HttpArtifactoryResponse.flush(HttpArtifactoryResponse.java:124)
> [artifactory-web-application-2.1.3.jar:na]
> at
> org.artifactory.webapp.servlet.HttpArtifactoryResponse.sendOk(HttpArtifactoryResponse.java:118)
> [artifactory-web-application-2.1.3.jar:na]
> at
> org.artifactory.request.ArtifactoryResponseBase.sendStream(ArtifactoryResponseBase.java:59)
> [artifactory-common-2.1.3.jar:na]
> at
> org.artifactory.request.RequestResponseHelper.sendBodyResponse(RequestResponseHelper.java:66)
> [artifactory-core-2.1.3.jar:na]
> at
> org.artifactory.engine.DownloadServiceImpl.respond(DownloadServiceImpl.java:183)
> [artifactory-core-2.1.3.jar:na]
> at
> org.artifactory.engine.DownloadServiceImpl.process(DownloadServiceImpl.java:122)
> [artifactory-core-2.1.3.jar:na]
> at sun.reflect.GeneratedMethodAccessor87.invoke(Unknown Source)
> [na:na]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> [na:1.5.0_21]
> at java.lang.reflect.Method.invoke(Method.java:592) [na:1.5.0_21]
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
> [spring-aop-2.5.6.jar:2.5.6]
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
> [spring-aop-2.5.6.jar:2.5.6]
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
> [spring-aop-2.5.6.jar:2.5.6]
> at
> org.artifactory.request.aop.RequestAdvice.invoke(RequestAdvice.java:66)
> [artifactory-core-2.1.3.jar:na]
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
> [spring-aop-2.5.6.jar:2.5.6]
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> [spring-aop-2.5.6.jar:2.5.6]
> at $Proxy36.process(Unknown Source) [na:na]
> at
> org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:112)
> [artifactory-web-application-2.1.3.jar:na]
> at
> org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:76)
> [artifactory-web-application-2.1.3.jar:na]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
> [catalina.jar:na]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> [catalina.jar:na]
> at
> org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:191)
> [artifactory-web-application-2.1.3.jar:na]
> at
> org.artifactory.webapp.servlet.AccessFilter.useAnonymousIfPossible(AccessFilter.java:174)
> [artifactory-web-application-2.1.3.jar:na]
> at
> org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:115)
> [artifactory-web-application-2.1.3.jar:na]
> at
> org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:91)
> [artifactory-web-application-2.1.3.jar:na]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
> [catalina.jar:na]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> [catalina.jar:na]
> at
> org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:54)
> [artifactory-web-application-2.1.3.jar:na]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
> [catalina.jar:na]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> [catalina.jar:na]
> at
> org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:54)
> [artifactory-web-application-2.1.3.jar:na]
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
> [catalina.jar:na]
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> [catalina.jar:na]
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
> [catalina.jar:na]
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
> [catalina.jar:na]
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> [catalina.jar:na]
> at
> org.jstripe.tomcat.probe.Tomcat55AgentValve.invoke(Tomcat55AgentValve.java:20)
> [Tomcat55AgentValve.class:na]
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
> [catalina.jar:na]
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
> [catalina.jar:na]
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
> [catalina.jar:na]
> at
> org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
> [tomcat-ajp.jar:na]
> at
> org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
> [tomcat-ajp.jar:na]
> at
> org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
> [tomcat-ajp.jar:na]
> at
> org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
> [tomcat-ajp.jar:na]
> at
> org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
> [tomcat-ajp.jar:na]
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
> [tomcat-util.jar:5.1]
> at java.lang.Thread.run(Thread.java:595) [na:1.5.0_21]
> Caused by: java.net.SocketException: Connection reset
> at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
> [na:1.5.0_21]
> at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> [na:1.5.0_21]
> at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:538)
> [tomcat-ajp.jar:na]
> at
> org.apache.jk.common.JkInputStream.doWrite(JkInputStream.java:162) [na:na]
> at org.apache.coyote.Response.doWrite(Response.java:560) [na:na]
> at
> org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:365)
> [na:na]
> ... 52 common frames omitted
>
> Clarence
>
> On Wed, Jul 7, 2010 at 5:04 PM, Frederic Simon <[email protected]> wrote:
>
>> This looks good but may generate quite a load of threads.
>> Did you try without the AJP connector?
>> Do you know the "ulimit -n" of the tomcat process?
>> Did you see any error in the artifactory.log file?
>>
>> Fred Simon
>> www.jfrog.org
>>
>> On Jul 7, 2010 11:05 PM, "Clarence" <[email protected]> wrote:
>> > Here is my server.xml file.
>> >
>> >
>> > <Service name="Catalina">
>> > <Connector port="8585" maxHttpHeaderSize="8192"
>> > maxThreads="400" minSpareThreads="125" maxSpareThreads="200"
>> > enableLookups="false" redirectPort="8443" acceptCount="200"
>> > connectionTimeout="0" maxKeepAliveRequests="1"
>> > disableUploadTimeout="true" />
>> >
>> > <!-- This is here for compatibility only, not required -->
>> > <Connector port="8509" protocol="AJP/1.3" enableLookups="false"
>> > redirectPort="8443" maxThreads="400" minSpareThreads="125"
>> > maxSpareThreads="200" connectionTimeout="60000"/>
>> >
>> > <Engine name="Catalina" defaultHost="localhost">
>> > <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>> > resourceName="UserDatabase" />
>> > <Host name="localhost" appBase="webapps"
>> > unpackWARs="true" autoDeploy="true"
>> > xmlValidation="false" xmlNamespaceAware="false">
>> > </Host>
>> > </Engine>
>> >
>> > </Service>
>> > </Server>
>> >
>> > Clarence
>> >
>> >
>> > On Wed, Jul 7, 2010 at 12:31 PM, Frederic Simon <[email protected]>
>> wrote:
>> >
>> >> Hi,
>> >>
>> >> From memory, a 503 using a direct connection to tomcat HTTP connector
>> means
>> >> that Tomcat server connector cannot accept any more connections.
>> >> I think you should look at the server.xml file in tomcat and configure
>> a
>> >> higher number of threads and may be accept. You can copy your current
>> >> server.xml here if you want.
>> >>
>> >> Your memory and machine looks OK to me.
>> >>
>> >> Hope this help,
>> >> Fred.
>> >>
>> >> On Wed, Jul 7, 2010 at 6:31 PM, Clarence <[email protected]> wrote:
>> >>
>> >>> Hello everyone. I am having no luck trying to resolve an issue with my
>> >>> Artifactory installation. First of all Artifactory is installed on a
>> box
>> >>> with 12 GB of RAM it has Intel Xeon Dual Core Processes 2.27GHz and
>> its
>> >>> running CentOS 5.4. The artifactory installation is using as its
>> >>> jcr:repo/filesystem-mysql. Its deployed via tomcat 5.5.28 using JDK
>> 1.5.0_21
>> >>> as its JAVA_HOME.
>> >>>
>> >>> When the nightly builds are kicked off from Hudson it generates
>> failures
>> >>> because its reporting that when attempting to upload to Artifactory
>> its
>> >>> receiving 503 errors. When the calls are made from hudson its using
>> the URI
>> >>> of http://hostname:8585/artifactory/<repo-location>. I am not using
>> ajp
>> >>> proxy in this direct request to the tomcat. I would like some help in
>> trying
>> >>> to find a root cause as to why this would happen (My opinion is that
>> Hudson
>> >>> process is timing out awaiting for the upload to complete and once
>> >>> artifactory returns it generates a 503 as the client has already
>> closed the
>> >>> connection too early for artifactory to respond.)
>> >>>
>> >>>
>> >>>
>> >>> I have read the mail thread countless times and google 503 and have
>> run
>> >>> into a few comments regarding Poorly configured VM and possible tuning
>> >>> issues with the tomcat or JAVA. Could this be the issue in my case.
>> The
>> >>> Tomcat is deployed out of the box with no major configuration changes.
>> >>> Here's what I set in my setenv.sh file:
>> >>>
>> >>>
>> >>> export JAVA_OPTS="-server -Xms1536m -Xmx1536m -XX:MaxPermSize=512m
>> >>> -XX:ParallelGCThreads=20 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>> >>> -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=75
>> -XX:MaxTenuringThreshold=0
>> >>> -XX:+AggressiveOpts -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
>> >>> -XX:-TraceClassUnloading -Dsun.io.useCanonCaches=false
>> >>> -Dsun.net.client.defaultConnectTimeout=60000
>> >>> -Dsun.net.client.defaultReadTimeout=60000"
>> >>> export JAVA_HOME=/usr/java/jdk1.5.0_21
>> >>>
>> >>>
>> >>>
>> >>> Would upgrading to the latest artifactory help me in resolving this
>> issue?
>> >>>
>> >>> Also I notice when I perform a top on the box I see that most to all
>> of my
>> >>> memory is consumed by the tomcat-artifactory process is from a bad
>> setting
>> >>> or I have setup tomcat incorrectly for artifactory?
>> >>>
>> >>> Here's the top from the machine in question.
>> >>>
>> >>> top - 11:26:36 up 136 days, 15:08, 1 user, load average: 1.17, 1.24,
>> >>> 1.07
>> >>> Tasks: 287 total, 1 running, 286 sleeping, 0 stopped, 0 zombie
>> >>> Cpu(s): 3.9%us, 2.5%sy, 0.0%ni, 93.7%id, 0.0%wa, 0.0%hi, 0.0%si,
>> >>> 0.0%st
>> >>> Mem: 12292916k total, 11265908k used, 1027008k free, 605780k buffers
>> >>> Swap: 16777208k total, 240k used, 16776968k free, 5460072k cached
>> >>>
>> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> >>> COMMAND
>> >>> 8682 tomcat-a 17 0 2446m 1.7g 16m S 101.1 14.6 299:38.82 java
>> >>>
>> >>> Any help would be appreciated thank you in advance
>> >>>
>> >>> Clarence
>> >>>
>> >>>
>> >>>
>> ------------------------------------------------------------------------------
>> >>> This SF.net email is sponsored by Sprint
>> >>> What will you do first with EVO, the first 4G phone?
>> >>> Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
>> >>> _______________________________________________
>> >>> Artifactory-users mailing list
>> >>> [email protected]
>> >>> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>> >>>
>> >>>
>> >>
>> >>
>> >> --
>> >> Co. Founder and Chief Architect
>> >> JFrog Ltd
>> >> http://www.jfrog.org/
>> >> http://twitter.com/freddy33
>> >>
>> >>
>> >>
>> ------------------------------------------------------------------------------
>> >> This SF.net email is sponsored by Sprint
>> >> What will you do first with EVO, the first 4G phone?
>> >> Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
>> >> _______________________________________________
>> >> Artifactory-users mailing list
>> >> [email protected]
>> >> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>> >>
>> >>
>>
>>
>> ------------------------------------------------------------------------------
>> This SF.net email is sponsored by Sprint
>> What will you do first with EVO, the first 4G phone?
>> Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
>> _______________________________________________
>> Artifactory-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>>
>>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Sprint
> What will you do first with EVO, the first 4G phone?
> Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
> _______________________________________________
> Artifactory-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>
>
------------------------------------------------------------------------------
This SF.net email is sponsored by Sprint
What will you do first with EVO, the first 4G phone?
Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
_______________________________________________
Artifactory-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/artifactory-users
