[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour Component/s: remoting Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemSometimes, after a certain amount of builds, the build never ends on the node. It sends forever the same 20-30 lines.It seems that the problem occures occurs more often when I restart Jenkins while there are some tasks running. But I'm not sure. It also seems that the problem is linked to a high load/network on the master/node. We can see the difference between the timestamper date (put when received by the master) and the log date (written during the Powershell execution)!Capture d'écran de 2016-08-20 16-35-18.png|thumbnail!Some logs files can be bigger than 10GB before I kill the process.(Yes, it's really stored in the JENKINS_HOME)!jekins_10GB_log.png|thumbnail!_I think it's worth mentionning that I have 3 jenkins nodes on the same machine , and that my JENKINS-HOME is located on a network drive (CIFS/SMB) ._h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.The process is terminated, the log file is not bigger than 3MB (and can be seen in the upper left corner of the screenshot)!Capture d'écran de 2016-08-20 17-04-34.png|thumbnail!*Update 1:* It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.*Update 2:* It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop in hudson.remoting.ProxyOutputStream.!Capture d'écran de 2016-08-20 19-02-41.png|thumbnail!*Update 3:* It seems that the Jenkins Master is guilty. I've connected my debugger to this one. The error occurs when it tries to write the log in its JENKINS_HOME. When executing the green line on the following screenshot. !Capture d'écran de 2016-08-20 19-35-36.png|thumbnail! The error is catched in DurableTaskStep$Execution.check, as it seems to be a workspace error. It seems that Jenkins doesn't find the workspace folder, as it's searching the jenkins node workspace in its local file system. C:\\ci\\int12\\ocoint... So it saves the log but interrupt the task, send to the slave that it has interrupted the task. The slave thinks that the logs has not been saved, and resend it to the master, which don't find the node workspace in it's local filesystem, etc.
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemSometimes, after a certain amount of builds, the build never ends on the node. It sends forever the same 20-30 lines.It seems that the problem occures more when I restart Jenkins while there are some tasks running.We can see the difference between the timestamper date (put when received by the master) and the log date (written during the Powershell execution)!Capture d'écran de 2016-08-20 16-35-18.png|thumbnail!Some logs files can be bigger than 10GB before I kill the process.(Yes, it's really stored in the JENKINS_HOME)!jekins_10GB_log.png|thumbnail!_I think it's worth mentionning that I have 3 jenkins nodes on the same machine._h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.The process is terminated, the log file is not bigger than 3MB (and can be seen in the upper left corner of the screenshot)!Capture d'écran de 2016-08-20 17-04-34.png|thumbnail!*Update 1:* It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.*Update 2:* It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop in hudson.remoting.ProxyOutputStream.!Capture d'écran de 2016-08-20 19-02-41.png|thumbnail!*Update 3:* It seems that the Jenkins Master is guilty. I've connected my debugger to this one. The error occurs when it tries to write the log in its JENKINS_HOME. When executing the green line on the following screenshot. !Capture d'écran de 2016-08-20 19-35-36.png|thumbnail! The error is catched in DurableTaskStep$Execution.check, as it seems to be a workspace error. It seems that Jenkins doesn't find the workspace folder, as it's searching the jenkins node workspace in its local file system. C:\\ci\\int12\\ocoint... So it saves the log but interrupt the task, send to the slave that it has interrupted the task. The slave thinks that the logs has not been saved, and resend it to the master, which don't find the node workspace in it's local filesystem, etc. !Capture d'écran de 2016-08-20 19-35-13.png|thumbnail! *Update 4:* It seems that when the master deserializes the response of the node, it returns a null object instead of the logs when it comes to
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemSometimes, after a certain amount of builds, the build never ends on the node. It sends forever the same 20-30 lines.It seems that the problem occures more when I restart Jenkins while there are some tasks running.We can see the difference between the timestamper date (put when received by the master) and the log date (written during the Powershell execution)!Capture d'écran de 2016-08-20 16-35-18.png|thumbnail!Some logs files can be bigger than 10GB before I kill the process.(Yes, it's really stored in the JENKINS_HOME)!jekins_10GB_log.png|thumbnail!_I think it's worth mentionning that I have 3 jenkins nodes on the same machine._h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.The process is terminated, the log file is not bigger than 3MB (and can be seen in the upper left corner of the screenshot)!Capture d'écran de 2016-08-20 17-04-34.png|thumbnail!*Update 1:* It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.*Update 2:* It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop in hudson.remoting.ProxyOutputStream.!Capture d'écran de 2016-08-20 19-02-41.png|thumbnail!*Update 3:* It seems that the Jenkins Master is guilty. I've connected my debugger to this one. The error occurs when it tries to write the log in its JENKINS_HOME. When executing the green line on the following screenshot. !Capture d'écran de 2016-08-20 19-35-36.png|thumbnail! The error is catched in DurableTaskStep$Execution.check, as it seems to be a workspace error. It seems that Jenkins doesn't find the workspace folder, as it's searching the jenkins node workspace in its local file system. C:\\ci\\int12\\ocoint... So it saves the log but interrupt the task, send to the slave that it has interrupted the task. The slave thinks that the logs has not been saved, and resend it to the master, which don't find the node workspace in it's local filesystem, etc. !Capture d'écran de 2016-08-20 19-35-13.png|thumbnail! *Update 4:* It seems that when the master deserializes the response of the node, it returns a null object instead of the logs when it comes to
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemSometimes, after a certain amount of builds, the build never ends on the node. It sends forever the same 20-30 lines.It seems that the problem occures more when I restart Jenkins while there are some tasks running.We can see the difference between the timestamper date (put when received by the master) and the log date (written during the Powershell execution)!Capture d'écran de 2016-08-20 16-35-18.png|thumbnail!Some logs files can be bigger than 10GB before I kill the process.(Yes, it's really stored in the JENKINS_HOME)!jekins_10GB_log.png|thumbnail!_I think it's worth mentionning that I have 3 jenkins nodes on the same machine._h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.The process is terminated, the log file is not bigger than 3MB (and can be seen in the upper left corner of the screenshot)!Capture d'écran de 2016-08-20 17-04-34.png|thumbnail!*Update 1:* It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.*Update 2:* It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop in hudson.remoting.ProxyOutputStream.!Capture d'écran de 2016-08-20 19-02-41.png|thumbnail!*Update 3:* It seems that the Jenkins Master is guilty. I've connected my debugger to this one. The error occurs when it tries to write the log in its JENKINS_HOME. When executing the green line on the following screenshot. !Capture d'écran de 2016-08-20 19-35-36.png|thumbnail! The error is catched in DurableTaskStep$Execution.check, as it seems to be a workspace error. It seems that Jenkins doesn't find the workspace folder, as it's searching the jenkins node workspace in its local file system. C:\\ci\\int12\\ocoint... So it saves the log but interrupt the task, send to the slave that it has interrupted the task. The slave thinks that the logs has not been saved, and resend it to the master, which don't find the node workspace in it's local filesystem, etc. !Capture d'écran de 2016-08-20 19-35-13.png|thumbnail! *Update 4:* It seems that when the master deserializes the response of the node, it returns a null object instead of the logs when it comes to
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemSometimes, after a certain amount of builds, the build never ends on the node. It sends forever the same 20-30 lines.It seems that the problem occures more when I restart Jenkins while there are some tasks running.We can see the difference between the timestamper date (put when received by the master) and the log date (written during the Powershell execution)!Capture d'écran de 2016-08-20 16-35-18.png|thumbnail!Some logs files can be bigger than 10GB before I kill the process.(Yes, it's really stored in the JENKINS_HOME)!jekins_10GB_log.png|thumbnail!_I think it's worth mentionning but that I have 3 jenkins nodes on the same machine._h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.The process is terminated, the log file is not bigger than 3MB (and can be seen in the upper left corner of the screenshot)!Capture d'écran de 2016-08-20 17-04-34.png|thumbnail!*Update 1:* It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.*Update 2:* It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop in hudson.remoting.ProxyOutputStream.!Capture d'écran de 2016-08-20 19-02-41.png|thumbnail!*Update 3:* It seems that the Jenkins Master is guilty. I've connected my debugger to this one. The error occurs when it tries to write the log in its JENKINS_HOME. When executing the green line on the following screenshot. !Capture d'écran de 2016-08-20 19-35-36.png|thumbnail! The error is catched in DurableTaskStep$Execution.check, as it seems to be a workspace error. It seems that Jenkins doesn't find the workspace folder, as it's searching the jenkins node workspace in its local file system. C:\\ci\\int12\\ocoint... So it saves the log but interrupt the task, send to the slave that it has interrupted the task. The slave thinks that the logs has not been saved, and resend it to the master, which don't find the node workspace in it's local filesystem, etc. !Capture d'écran de 2016-08-20 19-35-13.png|thumbnail! *Update 4:* It seems that when the master deserializes the response of the node, it returns a null object instead of the logs when it
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemSometimes, after a certain amount of builds, the build never ends on the node. It sends forever the same 20-30 lines.It seems that the problem occures more when I restart Jenkins while there are some tasks running.We can see the difference between the timestamper date (put when received by the master) and the log date (written during the Powershell execution)!Capture d'écran de 2016-08-20 16-35-18.png|thumbnail!Some logs files can be bigger than 10GB before I kill the process.(Yes, it's really stored in the JENKINS_HOME)!jekins_10GB_log.png|thumbnail!_I think it's worth mentionning but I have 3 jenkins nodes on the same machine._h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.The process is terminated, the log file is not bigger than 3MB (and can be seen in the upper left corner of the screenshot)!Capture d'écran de 2016-08-20 17-04-34.png|thumbnail!*Update 1:* It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.*Update 2:* It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop in hudson.remoting.ProxyOutputStream.!Capture d'écran de 2016-08-20 19-02-41.png|thumbnail!*Update 3:* It seems that the Jenkins Master is guilty. I've connected my debugger to this one. The error occurs when it tries to write the log in its JENKINS_HOME. When executing the green line on the following screenshot. !Capture d'écran de 2016-08-20 19-35-36.png|thumbnail! The error is catched in DurableTaskStep$Execution.check, as it seems to be a workspace error. It seems that Jenkins doesn't find the workspace folder, as it's searching the jenkins node workspace in its local file system. C:\\ci\\int12\\ocoint... So it saves the log but interrupt the task, send to the slave that it has interrupted the task. The slave thinks that the logs has not been saved, and resend it to the master, which don't find the node workspace in it's local filesystem, etc. !Capture d'écran de 2016-08-20 19-35-13.png|thumbnail! *Update 4:* It seems that when the master deserializes the response of the node, it returns a null object instead of the logs when it comes to
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemSometimes, after a certain amount of builds, the build never ends on the node. It sends forever the same 20-30 lines.It seems that the problem occures more when I restart Jenkins while there are some tasks running.We can see the difference between the timestamper date (put when received by the master) and the log date (written during the Powershell execution)!Capture d'écran de 2016-08-20 16-35-18.png|thumbnail!Some logs files can be bigger than 10GB before I kill the process.(Yes, it's really stored in the JENKINS_HOME)!jekins_10GB_log.png|thumbnail!_I think it's worth mentionning but I have 3 jenkins nodes on the same machine._h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.The process is terminated, the log file is not bigger than 3MB (and can be seen in the upper left corner of the screenshot)!Capture d'écran de 2016-08-20 17-04-34.png|thumbnail!*Update 1:* It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.*Update 2:* It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop in hudson.remoting.ProxyOutputStream.!Capture d'écran de 2016-08-20 19-02-41.png|thumbnail!*Update 3:* It seems that the Jenkins Master is guilty. I've connected my debugger to this one. The error occurs when it tries to write the log in its JENKINS_HOME. When executing the green line on the following screenshot. !Capture d'écran de 2016-08-20 19-35-36.png|thumbnail! The error is catched in DurableTaskStep$Execution.check, as it seems to be a workspace error. It seems that Jenkins doesn't find the workspace folder, as it's searching the jenkins node workspace in its local file system. C:\\ci\\int12\\ocoint... So it saves the log but interrupt the task, send to the slave that it has interrupted the task. The slave thinks that the logs has not been saved, and resend it to the master, which don't find the node workspace in it's local filesystem, etc. !Capture d'écran de 2016-08-20 19-35-13.png|thumbnail! *Update 4:* It seems that when the master deserializes the response of the node, it returns a null object instead of the logs when it comes to
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemSometimes, after a certain amount of builds, the build never ends on the node. It sends forever the same 20-30 lines.It seems that the problem occures more when I restart Jenkins while there are some tasks running.We can see the difference between the timestamper date (put when received by the master) and the log date (written during the Powershell execution)!Capture d'écran de 2016-08-20 16-35-18.png|thumbnail!Some logs files can be bigger than 10GB before I kill the process.(Yes, it's really stored in the JENKINS_HOME)!jekins_10GB_log.png|thumbnail!_I think it's worth mentionning but I have 3 jenkins nodes on the same machine._h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.The process is terminated, the log file is not bigger than 3MB (and can be seen in the upper left corner of the screenshot)!Capture d'écran de 2016-08-20 17-04-34.png|thumbnail!*Update 1:* It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.*Update 2:* It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop in hudson.remoting.ProxyOutputStream.!Capture d'écran de 2016-08-20 19-02-41.png|thumbnail!*Update 3:* It seems that the Jenkins Master is guilty. I've connected my debugger to this one. The error occurs when it tries to write the log in its JENKINS_HOME. When executing the green line on the following screenshot. !Capture d'écran de 2016-08-20 19-35-36.png|thumbnail! The error is catched in DurableTaskStep$Execution.check, as it seems to be a workspace error. It seems that Jenkins doesn't find the workspace folder, as it's searching the jenkins node workspace in its local file system. C:\\ci\\int12\\ocoint... So it saves the log but interrupt the task, send to the slave that it has interrupted the task. The slave thinks that the logs has not been saved, and resend it to the master, which don't find the node workspace in it's local filesystem, etc. !Capture d'écran de 2016-08-20 19-35-13.png|thumbnail! *Update 4:* It seems that when the master deserializes the response of the node, it returns a null object instead of the logs when it comes to
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemSometimes, after a certain amount of builds, the build never ends on the node. It sends forever the same 20-30 lines.It seems that the problem occures more when I restart Jenkins while there are some tasks running.We can see the difference between the timestamper date (put when received by the master) and the log date (written during the Powershell execution)!Capture d'écran de 2016-08-20 16-35-18.png|thumbnail!Some logs files can be bigger than 10GB before I kill the process.(Yes, it's really stored in the JENKINS_HOME)!jekins_10GB_log.png|thumbnail!_I think it's worth mentionning but I have 3 jenkins nodes on the same machine._h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.The process is terminated, the log file is not bigger than 3MB (and can be seen in the upper left corner of the screenshot)!Capture d'écran de 2016-08-20 17-04-34.png|thumbnail!*Update 1:* It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.*Update 2:* It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop in hudson.remoting.ProxyOutputStream.!Capture d'écran de 2016-08-20 19-02-41.png|thumbnail!*Update 3:* It seems that the Jenkins Master is guilty. I've connected my debugger to this one. The error occurs when it tries to write the log in its JENKINS_HOME. When executing the green line on the following screenshot. !Capture d'écran de 2016-08-20 19-35-36.png|thumbnail! The error is catched in DurableTaskStep$Execution.check, as it seems to be a workspace error. It seems that Jenkins doesn't find the workspace folder, as it's searching the jenkins node workspace in its local file system. C:\\ci\\int12\\ocoint... So it saves the log but interrupt the task, send to the slave that it has interrupted the task. The slave thinks that the logs has not been saved, and resend it to the master, which don't find the node workspace in it's local filesystem, etc. !Capture d'écran de 2016-08-20 19-35-13.png|thumbnail! *Update 4:* It seems that when the master deserializes the response of the node, it returns a null object instead of the logs when it comes to
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemSometimes, after a certain amount of builds, the build never ends on the node. It sends forever the same 20-30 lines.It seems that the problem occures more when I restart Jenkins while there are some tasks running.We can see the difference between the timestamper date (put when received by the master) and the log date (written during the Powershell execution)!Capture d'écran de 2016-08-20 16-35-18.png|thumbnail!Some logs files can be bigger than 10GB before I kill the process.(Yes, it's really stored in the JENKINS_HOME)!jekins_10GB_log.png|thumbnail! _I think it's worth mentionning but I have 3 jenkins nodes on the same machine._ h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.The process is terminated, the log file is not bigger than 3MB (and can be seen in the upper left corner of the screenshot)!Capture d'écran de 2016-08-20 17-04-34.png|thumbnail!*Update 1:* It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.*Update 2:* It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop in hudson.remoting.ProxyOutputStream.!Capture d'écran de 2016-08-20 19-02-41.png|thumbnail!*Update 3:* It seems that the Jenkins Master is guilty. I've connected my debugger to this one. The error occurs when it tries to write the log in its JENKINS_HOME. When executing the green line on the following screenshot. !Capture d'écran de 2016-08-20 19-35-36.png|thumbnail! The error is catched in DurableTaskStep$Execution.check, as it seems to be a workspace error. It seems that Jenkins doesn't find the workspace folder, as it's searching the jenkins node workspace in its local file system. C:\\ci\\int12\\ocoint... So it saves the log but interrupt the task, send to the slave that it has interrupted the task. The slave thinks that the logs has not been saved, and resend it to the master, which don't find the node workspace in it's local filesystem, etc. !Capture d'écran de 2016-08-20 19-35-13.png|thumbnail! *Update 4:* It seems that when the master deserializes the response of the node, it returns a null object instead of the logs when it comes to
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour Attachment: Capture d'écran de 2016-08-20 23-41-53.png Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemSometimes, after a certain amount of builds, the build never ends on the node. It sends forever the same 20-30 lines.It seems that the problem occures more when I restart Jenkins while there are some tasks running.We can see the difference between the timestamper date (put when received by the master) and the log date (written during the Powershell execution)!Capture d'écran de 2016-08-20 16-35-18.png|thumbnail!Some logs files can be bigger than 10GB before I kill the process.(Yes, it's really stored in the JENKINS_HOME)!jekins_10GB_log.png|thumbnail!h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.The process is terminated, the log file is not bigger than 3MB (and can be seen in the upper left corner of the screenshot)!Capture d'écran de 2016-08-20 17-04-34.png|thumbnail!*Update 1:* It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.*Update 2:* It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop in hudson.remoting.ProxyOutputStream.!Capture d'écran de 2016-08-20 19-02-41.png|thumbnail!*Update 3:* It seems that the Jenkins Master is guilty. I've connected my debugger to this one. The error occurs when it tries to write the log in its JENKINS_HOME. When executing the green line on the following screenshot. !Capture d'écran de 2016-08-20 19-35-36.png|thumbnail! The error is catched in DurableTaskStep$Execution.check, as it seems to be a workspace error. It seems that Jenkins doesn't find the workspace folder, as it's searching the jenkins node workspace in its local file system. C:\\ci\\int12\\ocoint... So it saves the log but interrupt the task, send to the slave that it has interrupted the task. The slave thinks that the logs has not been saved, and resend it to the master, which don't find the node workspace in it's local filesystem, etc. !Capture d'écran de 2016-08-20 19-35-13.png|thumbnail! *Update 4:* It seems that when the master deserializes the response of the node, it returns a null object instead of the logs when it comes to Write Logs... !Capture d'écran de 2016-08-20 23-41-53.png|thumbnail! h1. How
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. Problem The Sometimes, after a certain amount of builds, the build never ends on the node. It sends forever the same 20-30 lines. It seems that the problem occures more when I restart Jenkins while there are some tasks running. We can see the difference between the timestamper date (put when received by the master) and the log date (written during the Powershell execution)!Capture d'écran de 2016-08-20 16-35-18.png|thumbnail!Some logs files can be bigger than 10GB before I kill the process.(Yes, it's really stored in the JENKINS_HOME)!jekins_10GB_log.png|thumbnail!h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.The process is terminated, the log file is not bigger than 3MB (and can be seen in the upper left corner of the screenshot)!Capture d'écran de 2016-08-20 17-04-34.png|thumbnail!*Update 1:* It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.*Update 2:* It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop in hudson.remoting.ProxyOutputStream.!Capture d'écran de 2016-08-20 19-02-41.png|thumbnail!*Update 3:* It seems that the Jenkins Master is guilty. I've connected my debugger to this one. The error occurs when it tries to write the log in its JENKINS_HOME. When executing the green line on the following screenshot. !Capture d'écran de 2016-08-20 19-35-36.png|thumbnail! The error is catched in DurableTaskStep$Execution.check, as it seems to be a workspace error. It seems that Jenkins doesn't find the workspace folder, as it's searching the jenkins node workspace in its local file system. C:\\ci\\int12\\ocoint... So it saves the log but interrupt the task, send to the slave that it has interrupted the task. The slave thinks that the logs has not been saved, and resend it to the master, which don't find the node workspace in it's local filesystem, etc. !Capture d'écran de 2016-08-20 19-35-13.png|thumbnail! h1. How to reproduceI've tried to reproduce this bug for 3 days but didn't achieve to. I had to put my production in debug to inspect the error.As I don't understand why it fails, I can't create a reproduce protocol
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemThe build never ends on the node. It sends forever the same 20-30 lines.We can see the difference between the timestamper date (put when received by the master) and the log date (written during the Powershell execution)!Capture d'écran de 2016-08-20 16-35-18.png|thumbnail!Some logs files can be bigger than 10GB before I kill the process.(Yes, it's really stored in the JENKINS_HOME)!jekins_10GB_log.png|thumbnail!h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.The process is terminated, the log file is not bigger than 3MB (and can be seen in the upper left corner of the screenshot)!Capture d'écran de 2016-08-20 17-04-34.png|thumbnail!*Update 1:* It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.*Update 2:* It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop in hudson.remoting.ProxyOutputStream.!Capture d'écran de 2016-08-20 19-02-41.png|thumbnail!*Update 3: *It seems that the Jenkins Master is guilty. I've connected my debugger to this one. The error occurs when it tries to write the log in its JENKINS_HOME. When executing the green line on the following screenshot. !Capture d'écran de 2016-08-20 19-35-36.png|thumbnail! The error is catched in DurableTaskStep$Execution.check, as it seems to be a workspace error. It seems that Jenkins doesn't find the workspace folder, as it's searching the jenkins node workspace in its local file system. C:\\ci\\int12\\ocoint... So it saves the log but interrupt the task, send to the slave that it has interrupted the task. The slave thinks that the logs has not been saved, and resend it to the master, which don't find the node workspace in it's local filesystem, etc. !Capture d'écran de 2016-08-20 19-35-13.png|thumbnail! h1. How to reproduceI've tried to reproduce this bug for 3 days but didn't achieve to.
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemThe build never ends on the node. It sends forever the same 20-30 lines.We can see the difference between the timestamper date (put when received by the master) and the log date (written during the Powershell execution)!Capture d'écran de 2016-08-20 16-35-18.png|thumbnail!Some logs files can be bigger than 10GB before I kill the process.(Yes, it's really stored in the JENKINS_HOME)!jekins_10GB_log.png|thumbnail!h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.The process is terminated, the log file is not bigger than 3MB (and can be seen in the upper left corner of the screenshot)!Capture d'écran de 2016-08-20 17-04-34.png|thumbnail! * Update 1: * It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now. * Update 2: * It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop in hudson.remoting.ProxyOutputStream.!Capture d'écran de 2016-08-20 19-02-41.png|thumbnail! * Update 3: * It seems that the Jenkins Master is guilty. I've connected my debugger to this one. The error occurs when it tries to write the log in its JENKINS_HOME. When executing the green line on the following screenshot. !Capture d'écran de 2016-08-20 19-35-36.png|thumbnail! The error is catched in DurableTaskStep$Execution.check, as it seems to be a workspace error. It seems that Jenkins doesn't find the workspace folder, as it's searching the jenkins node workspace in its local file system. C:\\ci\\int12\\ocoint... !Capture d'écran de 2016-08-20 19-35-13.png|thumbnail! h1. How to reproduceI've tried to reproduce this bug for 3 days but didn't achieve to.
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour Attachment: Capture d'écran de 2016-08-20 19-35-13.png Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemThe build never ends on the node. It sends forever the same 20-30 lines.We can see the difference between the timestamper date (put when received by the master) and the log date (written during the Powershell execution)!Capture d'écran de 2016-08-20 16-35-18.png|thumbnail!Some logs files can be bigger than 10GB before I kill the process.(Yes, it's really stored in the JENKINS_HOME)!jekins_10GB_log.png|thumbnail!h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.The process is terminated, the log file is not bigger than 3MB (and can be seen in the upper left corner of the screenshot)!Capture d'écran de 2016-08-20 17-04-34.png|thumbnail! Edit Update 1: It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now. Edit Update 2: It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop in hudson.remoting.ProxyOutputStream.!Capture d'écran de 2016-08-20 19-02-41.png|thumbnail! Update 3: It seems that the Jenkins Master is guilty. I've connected my debugger to this one. The error occurs when it tries to write the log in its JENKINS_HOME. !Capture d'écran de 2016-08-20 19-35-36.png|thumbnail! h1. How to reproduceI've tried to reproduce this bug for 3 days but didn't achieve to. Add Comment
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour Attachment: Capture d'écran de 2016-08-20 19-35-36.png Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemThe build never ends on the node. It sends forever the same 20-30 lines. We can see the difference between the timestamper date (put when received by the master) and the log date (written during the Powershell execution) !Capture d'écran de 2016-08-20 16-35-18.png|thumbnail!Some logs files can be bigger than 10GB before I kill the process. (Yes, it's really stored in the JENKINS_HOME) !jekins_10GB_log.png|thumbnail!h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase. The process is terminated, the log file is not bigger than 3MB (and can be seen in the upper left corner of the screenshot) !Capture d'écran de 2016-08-20 17-04-34.png|thumbnail!Edit 1: It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.Edit 2: It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop in hudson . remoting.ProxyOutputStream. !Capture d'écran de 2016-08-20 19-02-41.png|thumbnail!h1. How to reproduceI've tried to reproduce this bug for 3 days but didn't achieve to. Add Comment
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemThe build never ends on the node. It sends forever the same 20-30 lines.!Capture d'écran de 2016-08-20 16-35-18.png|thumbnail!Some logs files can be bigger than 10GB before I kill the process.! jenkins_10GB_log jekins_10GB_log .png|thumbnail!h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.!Capture d'écran de 2016-08-20 17-04-34.png|thumbnail!Edit 1: It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.Edit 2: It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop.!Capture d'écran de 2016-08-20 19-02-41.png|thumbnail!h1. How to reproduceI've tried to reproduce this bug for 3 days but didn't achieve to. Add Comment
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemThe build never ends on the node. It sends forever the same 20-30 lines.!Capture d'écran de 2016-08-20 19 16 - 02 35 - 41 18 .png|thumbnail!Some logs files can be bigger than 10GB before I kill the process.! Capture d'écran de 2016-08-20 19-02-41 jenkins_10GB_log .png|thumbnail!h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase. !Capture d'écran de 2016-08-20 17-04-34.png|thumbnail! Edit 1: It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.Edit 2: It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop.!Capture d'écran de 2016-08-20 19-02-41.png|thumbnail!h1. How to reproduceI've tried to reproduce this bug for 3 days but didn't achieve to. Add Comment
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemThe build never ends on the node. It sends forever the same 20-30 lines. !Capture d'écran de 2016-08-20 19-02-41.png|thumbnail! Some logs files can be bigger than 10GB before I kill the process. !Capture d'écran de 2016-08-20 19-02-41.png|thumbnail! h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.Edit 1: It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.Edit 2: It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop.!Capture d'écran de 2016-08-20 19-02-41.png |thumbnail ! h2. ScreenshotsOn the first one we can see the logs. Especially that the date from timestamper is different from the date of the logs. It's a good indication that we have a problem.On the second screenshot, I've connected netbeans on the node which handle the job. I've stopped the thread and also put a breakpoint on sink.write(buf). When I hit continue, I've seen that this function is always called. We can see some variables, especially the file descriptor and its associated size. The folder of this file is open on the top left of this screenshot. We can see that the size of this file is arround 3MB.On the last screenshot, it's just an example at how big the file becomes before crashing Jenkins in an Out of Memory Exception... h1. How to reproduceI've tried to reproduce this bug for 3 days but didn't achieve to.
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemThe build never ends on the node. It sends forever the same 20-30 lines.Some logs files can be bigger than 10GB before I kill the process.h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.Edit 1: It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now.Edit 2: It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop. !Capture d'écran de 2016-08-20 19-02-41.png! h2. ScreenshotsOn the first one we can see the logs. Especially that the date from timestamper is different from the date of the logs. It's a good indication that we have a problem.On the second screenshot, I've connected netbeans on the node which handle the job. I've stopped the thread and also put a breakpoint on sink.write(buf). When I hit continue, I've seen that this function is always called. We can see some variables, especially the file descriptor and its associated size. The folder of this file is open on the top left of this screenshot. We can see that the size of this file is arround 3MB.On the last screenshot, it's just an example at how big the file becomes before crashing Jenkins in an Out of Memory Exception...h1. How to reproduceI've tried to reproduce this bug for 3 days but didn't achieve to. Add Comment
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour Attachment: Capture d'écran de 2016-08-20 19-02-41.png Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemThe build never ends on the node. It sends forever the same 20-30 lines.Some logs files can be bigger than 10GB before I kill the process.h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase. Edit 1: It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now. Edit 2: It seems that it comes from the network, as I've captured a java.io.InterruptedIOException in this loop. h2. ScreenshotsOn the first one we can see the logs. Especially that the date from timestamper is different from the date of the logs. It's a good indication that we have a problem.On the second screenshot, I've connected netbeans on the node which handle the job. I've stopped the thread and also put a breakpoint on sink.write(buf). When I hit continue, I've seen that this function is always called. We can see some variables, especially the file descriptor and its associated size. The folder of this file is open on the top left of this screenshot. We can see that the size of this file is arround 3MB.On the last screenshot, it's just an example at how big the file becomes before crashing Jenkins in an Out of Memory Exception...h1. How to reproduceI've tried to reproduce this bug for 3 days but didn't achieve to. Add Comment
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemThe build never ends on the node. It sends forever the same 20-30 lines.Some logs files can be bigger than 10GB before I kill the process.h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase. It seems that Jenkins read the whole file. If it fails, it will return 0. I suspect that Jenkins is failing to close the file descriptor. So the lastLocation is not updated. But the data are sent. Jenkins retries to read the file, fail again, etc. That's only a supposition for now. h2. ScreenshotsOn the first one we can see the logs. Especially that the date from timestamper is different from the date of the logs. It's a good indication that we have a problem.On the second screenshot, I've connected netbeans on the node which handle the job. I've stopped the thread and also put a breakpoint on sink.write(buf). When I hit continue, I've seen that this function is always called. We can see some variables, especially the file descriptor and its associated size. The folder of this file is open on the top left of this screenshot. We can see that the size of this file is arround 3MB.On the last screenshot, it's just an example at how big the file becomes before crashing Jenkins in an Out of Memory Exception...h1. How to reproduceI've tried to reproduce this bug for 3 days but didn't achieve to. Add Comment
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemThe build never ends on the node. It sends forever the same 20-30 lines.Some logs files can be bigger than 10GB before I kill the process.h1. Investigationh2. StepsI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase.h2. ScreenshotsOnthe first one we can see the logs. Especially that the date from timestamper is different from the date of the logs. It's a good indication that we have a problem. On the second screenshot, I've connected netbeans on the node which handle the job. I've stopped the thread and also put a breakpoint on sink.write(buf). When I hit continue, I've seen that this function is always called. We can see some variables, especially the file descriptor and its associated size. The folder of this file is open on the top left of this screenshot. We can see that the size of this file is arround 3MB.On the last screenshot, it's just an example at how big the file becomes before crashing Jenkins in an Out of Memory Exception... h1. How to reproduceI've tried to reproduce this bug for 3 days but didn't achieve to. Add Comment
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour h1. ProblemThe build never ends on the node. It sends forever the same 20-30 lines.Some logs files can be bigger than 10GB before I kill the process.h1. Investigation h2. Steps I've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase. (check the attached screenshots above) h2. Screenshots On h1. How to reproduceI've tried to reproduce this bug for 3 days but didn't achieve to. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour Attachment: jekins_10GB_log.png Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour updated an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Change By: Quentin Dufour Attachment: Capture d'écran de 2016-08-20 17-04-34.png Attachment: Capture d'écran de 2016-08-20 16-35-18.png h1. ProblemThe build never ends on the node. It sends forever the same 20-30 lines.Some logs files can be bigger than 10GB before I kill the process.h1. InvestigationI've found on Wireshark that the node keeps sending the same logs forever.So the jenkins master is not (directly) the culprit.After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.javaThe same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase. (check the attached screenshots above) h1. How to reproduceI've tried to reproduce this bug for 3 days but didn't achieve to. Add Comment
[JIRA] (JENKINS-37575) Jenkins node keeps sending the sames logs indefinitely
Title: Message Title Quentin Dufour created an issue Jenkins / JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely Issue Type: Bug Assignee: Jesse Glick Components: durable-task-plugin Created: 2016/Aug/20 9:15 PM Environment: Windows Server 2012 R2 Jenkins 2.7.2 Priority: Major Reporter: Quentin Dufour Problem The build never ends on the node. It sends forever the same 20-30 lines. Some logs files can be bigger than 10GB before I kill the process. Investigation I've found on Wireshark that the node keeps sending the same logs forever. So the jenkins master is not (directly) the culprit. After enabling the debugguer on the slave, I've found that the method FileMonitoringTask$FileMonitoringController$WriteLog.invoke is called in an infinite loop somewhere in this file: durable-task-plugin\src\main\java\org\jenkinsci\plugins\durabletask\FileMonitoringTask.java The same file is read again and again with a lastLocation of 1930670. lastLocation represent the bytes already read. But I don't understand why it doesn't increase. How to reproduce I've tried to reproduce this bug for 3 days but didn't achieve to.
