My analysis of the problem in the past few days also points to the GKE
autoscaler bringing down nodes with scheduled pods. HIVE-30128 is a
promising fix so I merged it a few minutes ago. Let's continue
monitoring the situation to see if additional actions are needed.
For the record, there has been a Kubernetes upgrade on September 20
which may have brought new changes in the autoscaler logic aggravating
the issue.
$ gcloud logging read
'protoPayload.methodName="google.container.internal.ClusterManagerInternal.PatchCluster"
AND protoPayload.metadata.operationType="UPGRADE_MASTER" AND
timestamp>="2026-08-20T00:00:00Z"' --limit=10 --format=json | jq '.[]
|{timestamp:.timestamp,newversion:.protoPayload.metadata.currentMasterVersion,
oldversion:.protoPayload.metadata.previousMasterVersion}'
{
"timestamp": "2026-09-27T23:15:18.476357688Z",
"newversion": "1.35.6-gke.1250000",
"oldversion": "1.35.6-gke.1250000"
}
{
"timestamp": "2026-09-20T23:23:51.890372730Z",
"newversion": "1.35.6-gke.1250000",
"oldversion": "1.34.9-gke.1655001"
}
There were ChannelClosedException issues appearing on master around
September 20 [1] but also before [2, 3] so it's hard to fully blame
just the upgrade.
Best,
Stamatis
[1] https://ci.hive.apache.org/job/hive-precommit/job/master/3094/
[2] https://ci.hive.apache.org/job/hive-precommit/job/master/3063/
[3] https://ci.hive.apache.org/job/hive-precommit/job/master/3006/
On Wed, Oct 14, 2026 at 6:20 PM Raghav Aggarwal
<[email protected]> wrote:
>
> Hi Stamatis,
>
> This issue has been occurring for over three weeks, so it is not a recent
> problem. I have observed it multiple times across my pull requests.
>
> For reference:
>
> - A fix attempt appears to be in progress (HIVE-30128):
> https://github.com/apache/hive/pull/6843
> - A related PR merged 19 days ago demonstrates this pod failure issue:
> https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-6750
>
> Regards,
> Raghav Aggarwal
>
>
> On Wed, Oct 14, 2026 at 5:19 PM Stamatis Zampetakis <[email protected]>
> wrote:
>>
>> Small correction: The PR 6644 and 6872 advance normally. However, in
>> PR#6815 the ChannelClosedException popped up but the underlying pods
>> are long gone so don't really know what happened. I will check a bit
>> further to see if I can determine something.
>>
>> [2026-10-14T10:07:50.257Z] Cannot contact
>> hive-precommit-pr-6815-5-9327p-g96pl-71l7f:
>> hudson.remoting.ChannelClosedException: Channel
>> "hudson.remoting.Channel@5ca733ed:JNLP4-connect connection from
>> 10.0.13.8/10.0.13.8:39556": Remote call on JNLP4-connect connection
>> from 10.0.13.8/10.0.13.8:39556 failed. The channel is closing down or
>> has closed down
>>
>> Best,
>> Stamatis
>>
>> On Wed, Oct 14, 2026 at 12:36 PM Stamatis Zampetakis <[email protected]>
>> wrote:
>> >
>> > Hi Konstantin,
>> >
>> > I just checked the Google Cloud incidents and it appears that there
>> > was a problem affecting VMs in us-central1 where our pods are running.
>> > The incident timeline:
>> > * Started on October 13, 2026 at 3:19:12 PM UTC+1
>> > * Resolved on October 13, 2026 at 6:35:59 PM UTC+1
>> >
>> > At the moment, I see 61 pods up and running normally for the following
>> > PRs: 6644, 6815, 6872. Let's see if the problem persists to
>> > investigate further.
>> >
>> > Best,
>> > Stamatis
>> >
>> > On Wed, Oct 14, 2026 at 12:48 AM Konstantin Bereznyakov
>> > <[email protected]> wrote:
>> > >
>> > > Hi all,
>> > >
>> > > I investigated repeated CI failures on HIVE-30037 (PR #6842
>> > > (https://github.com/apache/hive/pull/6842)) and found matching
>> > > agent/pod failures on unrelated PRs.
>> > >
>> > > All times below are October 13, UTC:
>> > >
>> > > #6842 build #12: split-09 pod failed at 09:24; postprocessing raised
>> > > AgentOfflineException. Archived test reports contain zero
>> > > failures/errors, including the new branch tests.
>> > > PR #6861 (https://github.com/apache/hive/pull/6861), build #8:
>> > > split-15 pod failed at 16:58.
>> > > PR #6871 (https://github.com/apache/hive/pull/6871), build #3: six
>> > > pods failed between 18:58 and 19:17.
>> > > #6842 build #13: pod failed at 19:28 during workspace upload, after
>> > > successful compilation and before testing.
>> > >
>> > > The logs consistently show ChannelClosedException, “Pod just failed,”
>> > > and agent deletion. This appears to be a shared infrastructure issue.
>> > >
>> > > Could someone with CI access check the Kubernetes pod/node events? I
>> > > have saved all four console logs.
>> > >
>> > > Thanks,
>> > > Konstantin
>
>
>
> --
> Thanks,
> Raghav Aggarwal
