On Wed, Jun 29, 2016 at 3:43 PM Jérémie Faucher-Goulet <[email protected]> wrote:
> In that case, yes you are correct, scm-manager does use JGit. I assume > that the "Remote: " part of the messages means these messages come from the > server? I'm not familiar with Git protocols. > > Yes, we see these failures intermittently on Jenkins. However, I should > perhaps have been more specific about the failure rate. I would guesstimate > that 25% of the builds will fail with a failure to fetch from the server. > With a team of 10-15 developers never getting this issue over the span of > almost a year, I'd say it should have been reproduced/seen by then if it > wasn't specific to Jenkins or the machine it's running on. And developers > uses multiple tools on the client side such as SourceTree, GitKraken, > Egit (Eclipse), and Git (command-line). > > I'm still very much confused by the fact that Jenkins will show a line in > the logs that it will update/fetch a git submodule when the error occurs. > On successful builds, the submodules aren't touched. Could it be that > Jenkinds calls git from within the wrong 'current directory'? > I believe there is some submodule processing by default in the git plugin. At least, there is a checkbox "Disable submodule processing" in the advanced submodule processing section of "Additional Behaviours". You could try that. Mark Waite > Other thing I've noticed: > > Sometimes we will get the error below from git being called manually by > scripts during the build process. Even though it's supposed to be "fatal", > the operations itself looks successful. This was only seen on Windows > however and could relate to a different issue > <https://github.com/git-for-windows/git/issues/766>. > >> This application has requested the Runtime to terminate it in an unusual way. >> Please contact the application's support team for more information. >> Assertion failed! >> >> Program: C:\Program >> Files\Git\mingw64\libexec\git-core\git-credential-store.exe >> File: exec_cmd.c, Line 23 >> >> > Jérémie > > > Le mercredi 29 juin 2016 15:02:32 UTC-4, Mark Waite a écrit : >> >> >> >> On Tuesday, June 28, 2016 at 5:53:54 PM UTC-6, Jérémie Faucher-Goulet >> wrote: >>> >>> Hi Mark, >>> >>> Thanks for taking time answering, >>> >>> What do you mean by "git server" and "git as a server"? >>> >> >> I meant scm-manager in your case. As far as I can tell from reading the >> scm-manager changelog, they use JGit as their internal git implementation. >> I suspect that means they don't use command line git on the server side. >> >> >>> We currently use (and clone via HTTPS) from a private installation of >>> scm-manager <https://www.scm-manager.org/> version 1.41. But I don't >>> think the "git server" is at cause since only Jenkins experiences these >>> issues. >>> Jenkins is currently configured to use Git via command line. The windows >>> machine is running Git 2.8.3. The Linux slave machine is running Git 1.9.1. >>> >>> >> I thought you said that Jenkins only saw those failures intermittently. >> If only intermittently, then Jenkins more frequent use of checkouts may >> mean it is more likely to be seen there than by users doing checkout. >> >> >>> Is the repo big? Not that big, unless you are counting the submodules. >>> My understanding however is that submodules should not be initialized >>> unless we've specified an advanced behaviour in the job. >>> Beside, this is not a clone operation here that fails. The repository >>> already is in the job's workspace from a previous successful build. This is >>> a git fetch. >>> >>> It is my belief the "Finding sources" and "Getting sizes" messages comes >>> from the fact that the "--progress" option was passed to git fetch when >>> Jenkins was doing it's thing. I could be competently wrong too however ;-) >>> >>> FYI. the following 'Additional Behaviours' are used in the job: >>> >>> - Clean before checkout >>> - Check out to a sub-directory >>> - Custom SCM name >>> >>> Best regards, >>> >>> Jérémie >>> >>> Le mardi 28 juin 2016 16:24:40 UTC-4, Mark Waite a écrit : >>>> >>>> Are you trying to clone a huge repo with JGit on the git server? If >>>> so, you might try cloning the same repo with command line git on the >>>> server. >>>> >>>> I don't recognize "Finding sources" or "Getting sizes" as common >>>> messages from command line git as a server. >>>> >>>> Mark Waite >>>> >>>> On Thursday, June 16, 2016 at 10:57:59 AM UTC-6, Jérémie Faucher-Goulet >>>> wrote: >>>>> >>>>> Hello all, >>>>> I've got a nagging problem for which I'd welcome some thoughtful >>>>> expert input... >>>>> >>>>> Once in a while, our builds will fail with the output pasted below in >>>>> the console. This is a build triggered by another job, which itself is >>>>> triggered remotely by Atlassian Fisheye when a commit is pushed to the >>>>> repository. This is part of our CI system. >>>>> >>>>> Building remotely on grs-jslave >>>>> <http://grs-jenkins03.trilliant.local/computer/grs-jslave> (linux openwrt >>>>> cmockery2) in workspace >>>>> /var/grs-jenkins03/workspace/smv6-6lr-core_Develop_Linux >>>>> > git rev-parse --is-inside-work-tree # timeout=10 >>>>> Fetching changes from the remote Git repository >>>>> > git config remote.origin.url >>>>> https://grs-softsvnrepo1.trilliant.local:8081/scm/git/smv6-6lr-core # >>>>> timeout=10 >>>>> Cleaning workspace >>>>> > git rev-parse --verify HEAD # timeout=10 >>>>> Resetting working tree >>>>> > git reset --hard # timeout=10 >>>>> > git clean -fdx # timeout=10 >>>>> Fetching upstream changes from >>>>> https://grs-softsvnrepo1.trilliant.local:8081/scm/git/smv6-6lr-core >>>>> > git --version # timeout=10 >>>>> using .gitcredentials to set credentials >>>>> > git config --local credential.username grfbuild # timeout=10 >>>>> > git config --local credential.helper store >>>>> --file=/tmp/git9025647881849389319.credentials # timeout=10 >>>>> > git -c core.askpass=true fetch --tags --progress >>>>> https://grs-softsvnrepo1.trilliant.local:8081/scm/git/smv6-6lr-core >>>>> +refs/heads/*:refs/remotes/origin/* >>>>> > git config --local --remove-section credential # timeout=10 >>>>> ERROR: Error fetching remote repo 'origin' >>>>> hudson.plugins.git.GitException >>>>> <http://stacktrace.jenkins-ci.org/search?query=hudson.plugins.git.GitException>: >>>>> Failed to fetch from >>>>> https://grs-softsvnrepo1.trilliant.local:8081/scm/git/smv6-6lr-core >>>>> at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:766) >>>>> <http://stacktrace.jenkins-ci.org/search/?query=hudson.plugins.git.GitSCM.fetchFrom&entity=method> >>>>> at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1022) >>>>> <http://stacktrace.jenkins-ci.org/search/?query=hudson.plugins.git.GitSCM.retrieveChanges&entity=method> >>>>> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1053) >>>>> <http://stacktrace.jenkins-ci.org/search/?query=hudson.plugins.git.GitSCM.checkout&entity=method> >>>>> at hudson.scm.SCM.checkout(SCM.java:485) >>>>> <http://stacktrace.jenkins-ci.org/search/?query=hudson.scm.SCM.checkout&entity=method> >>>>> at >>>>> hudson.model.AbstractProject.checkout(AbstractProject.java:1275) >>>>> <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.AbstractProject.checkout&entity=method> >>>>> at >>>>> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:610) >>>>> >>>>> <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout&entity=method> >>>>> at >>>>> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) >>>>> <http://stacktrace.jenkins-ci.org/search/?query=jenkins.scm.SCMCheckoutStrategy.checkout&entity=method> >>>>> at >>>>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:532) >>>>> >>>>> <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.AbstractBuild$AbstractBuildExecution.run&entity=method> >>>>> at hudson.model.Run.execute(Run.java:1741) >>>>> <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.Run.execute&entity=method> >>>>> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) >>>>> <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.FreeStyleBuild.run&entity=method> >>>>> at >>>>> hudson.model.ResourceController.execute(ResourceController.java:98) >>>>> <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.ResourceController.execute&entity=method> >>>>> at hudson.model.Executor.run(Executor.java:408) >>>>> <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.Executor.run&entity=method> >>>>> Caused by: hudson.plugins.git.GitException >>>>> <http://stacktrace.jenkins-ci.org/search?query=hudson.plugins.git.GitException>: >>>>> Command "git -c core.askpass=true fetch --tags --progress >>>>> https://grs-softsvnrepo1.trilliant.local:8081/scm/git/smv6-6lr-core >>>>> +refs/heads/*:refs/remotes/origin/*" returned status code 1: >>>>> stdout: Fetching submodule contrib/exegin-nan >>>>> >>>>> stderr: remote: Counting objects: 1 >>>>> remote: Counting objects: 57, done >>>>> remote: Finding sources: 4% (1/25) >>>>> remote: Finding sources: 8% (2/25) >>>>> remote: Finding sources: 12% (3/25) >>>>> remote: Finding sources: 16% (4/25) >>>>> remote: Finding sources: 20% (5/25) >>>>> remote: Finding sources: 24% (6/25) >>>>> remote: Finding sources: 28% (7/25) >>>>> remote: Finding sources: 32% (8/25) >>>>> remote: Finding sources: 36% (9/25) >>>>> remote: Finding sources: 40% (10/25) >>>>> remote: Finding sources: 44% (11/25) >>>>> remote: Finding sources: 48% (12/25) >>>>> remote: Finding sources: 52% (13/25) >>>>> remote: Finding sources: 56% (14/25) >>>>> remote: Finding sources: 60% (15/25) >>>>> remote: Finding sources: 64% (16/25) >>>>> remote: Finding sources: 68% (17/25) >>>>> remote: Finding sources: 72% (18/25) >>>>> remote: Finding sources: 76% (19/25) >>>>> remote: Finding sources: 80% (20/25) >>>>> remote: Finding sources: 84% (21/25) >>>>> remote: Finding sources: 88% (22/25) >>>>> remote: Finding sources: 92% (23/25) >>>>> remote: Finding sources: 96% (24/25) >>>>> remote: Finding sources: 100% (25/25) >>>>> remote: Finding sources: 100% (25/25) >>>>> remote: Getting sizes: 2% (1/42) >>>>> remote: Getting sizes: 4% (2/42) >>>>> remote: Getting sizes: 7% (3/42) >>>>> remote: Getting sizes: 9% (4/42) >>>>> remote: Getting sizes: 11% (5/42) >>>>> remote: Getting sizes: 14% (6/42) >>>>> remote: Getting sizes: 16% (7/42) >>>>> remote: Getting sizes: 19% (8/42) >>>>> remote: Getting sizes: 21% (9/42) >>>>> remote: Getting sizes: 23% (10/42) >>>>> remote: Getting sizes: 26% (11/42) >>>>> remote: Getting sizes: 28% (12/42) >>>>> remote: Getting sizes: 30% (13/42) >>>>> remote: Getting sizes: 33% (14/42) >>>>> remote: Getting sizes: 35% (15/42) >>>>> remote: Getting sizes: 38% (16/42) >>>>> remote: Getting sizes: 40% (17/42) >>>>> remote: Getting sizes: 42% (18/42) >>>>> remote: Getting sizes: 45% (19/42) >>>>> remote: Getting sizes: 47% (20/42) >>>>> remote: Getting sizes: 50% (21/42) >>>>> remote: Getting sizes: 52% (22/42) >>>>> remote: Getting sizes: 54% (23/42) >>>>> remote: Getting sizes: 57% (24/42) >>>>> remote: Getting sizes: 59% (25/42) >>>>> remote: Getting sizes: 61% (26/42) >>>>> remote: Getting sizes: 64% (27/42) >>>>> remote: Getting sizes: 66% (28/42) >>>>> remote: Getting sizes: 69% (29/42) >>>>> remote: Getting sizes: 71% (30/42) >>>>> remote: Getting sizes: 73% (31/42) >>>>> remote: Getting sizes: 76% (32/42) >>>>> remote: Getting sizes: 78% (33/42) >>>>> remote: Getting sizes: 80% (34/42) >>>>> remote: Getting sizes: 83% (35/42) >>>>> remote: Getting sizes: 85% (36/42) >>>>> remote: Getting sizes: 88% (37/42) >>>>> remote: Getting sizes: 90% (38/42) >>>>> remote: Getting sizes: 92% (39/42) >>>>> remote: Getting sizes: 95% (40/42) >>>>> remote: Getting sizes: 97% (41/42) >>>>> remote: Getting sizes: 100% (42/42) >>>>> remote: Getting sizes: 100% (42/42) >>>>> remote: Total 25 (delta 12), reused 25 (delta 12) >>>>> From https://grs-softsvnrepo1.trilliant.local:8081/scm/git/smv6-6lr-core >>>>> 029fa64..dee5ad4 develop -> origin/develop >>>>> * [new branch] feature/fb-new-exegin-3_6_0 -> >>>>> origin/feature/fb-new-exegin-3_6_0 >>>>> fatal: remote error: Invalid username or password. >>>>> >>>>> at >>>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1710) >>>>> at >>>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1454) >>>>> at >>>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:63) >>>>> at >>>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:314) >>>>> at >>>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152) >>>>> at >>>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145) >>>>> at hudson.remoting.UserRequest.perform(UserRequest.java:121) >>>>> at hudson.remoting.UserRequest.perform(UserRequest.java:49) >>>>> at hudson.remoting.Request$2.run(Request.java:326) >>>>> at >>>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> at ......remote call to grs-jslave(Native Method) >>>>> at >>>>> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1413) >>>>> at hudson.remoting.UserResponse.retrieve(UserRequest.java:221) >>>>> at hudson.remoting.Channel.call(Channel.java:778) >>>>> at >>>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145) >>>>> at sun.reflect.GeneratedMethodAccessor570.invoke(Unknown Source) >>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) >>>>> at java.lang.reflect.Method.invoke(Unknown Source) >>>>> at >>>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131) >>>>> at com.sun.proxy.$Proxy70.execute(Unknown Source) >>>>> at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:764) >>>>> ... 11 more >>>>> ERROR: null >>>>> >>>>> >>>>> Other tidbits or observations: >>>>> >>>>> - It seems to fail when there hasn't been any builds for a while, >>>>> as if something was cached. I've never seen two failures in rapid >>>>> succession. When we get this failure, usually immediately relaunching >>>>> the >>>>> same build will work the second time around. >>>>> - I'm confused by the fact that it seems to try fetching git >>>>> submodules when it fails. We haven't enabled any advanced submodule >>>>> behavior and a successful build doesn't show any submodule fetches >>>>> being >>>>> made. >>>>> - This occurs on both Windows and Linux (We run a similar build on >>>>> a Linux machine configured as a Jenkins slave). >>>>> - From the windows error, I've read somewhere that it might have >>>>> something to do with "wincreds" being configured globally while we use >>>>> "store" locally. But how would that apply to Linux? Furthermore I've >>>>> confirmed that wincred isn't configured on the windows machine. >>>>> >>>>> On a windows build, we get a slightly different output: >>>>> >>>>> ERROR: Error fetching remote repo 'origin' >>>>> hudson.plugins.git.GitException >>>>> <http://stacktrace.jenkins-ci.org/search?query=hudson.plugins.git.GitException>: >>>>> Failed to fetch from >>>>> https://grs-softsvnrepo1.trilliant.local:8081/scm/git/smv6-6lr-core >>>>> at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:766) >>>>> <http://stacktrace.jenkins-ci.org/search/?query=hudson.plugins.git.GitSCM.fetchFrom&entity=method> >>>>> at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1022) >>>>> <http://stacktrace.jenkins-ci.org/search/?query=hudson.plugins.git.GitSCM.retrieveChanges&entity=method> >>>>> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1053) >>>>> <http://stacktrace.jenkins-ci.org/search/?query=hudson.plugins.git.GitSCM.checkout&entity=method> >>>>> at >>>>> org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:129) >>>>> <http://stacktrace.jenkins-ci.org/search/?query=org.jenkinsci.plugins.multiplescms.MultiSCM.checkout&entity=method> >>>>> at hudson.scm.SCM >>>>> <http://stacktrace.jenkins-ci.org/search/?query=hudson.scm.SCM.checkout&entity=method> >>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtHOZUmJZbmesCuaO5fT2DXXm0ODz6RJ04je92LZZ03g-Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
