Re: [ovs-discuss] ovsdb-server fails to start and report one unreferenced file path doesn't exist
If that is the case, the size of /usr/sbin/ovsdb-server and /usr/local/sbin/ovsdb-server should be exactly the same, isn't it? -Original Message- From: Adrian Moreno Sent: Thursday, April 25, 2024 3:10 PM To: Wang, Fei2 ; Ilya Maximets ; b...@openvswitch.org Subject: Re: [ovs-discuss] ovsdb-server fails to start and report one unreferenced file path doesn't exist On 4/25/24 04:29, Wang, Fei2 via discuss wrote: > The ovs was configured by my colleague, I am trying to help with the mess-up. > Took a deep dive, found two obsdb-server binary as you can see below: > > # ls -al /usr/sbin/ovsdb-server > -rwxr-xr-x 1 root root 16172416 Apr 18 18:59 /usr/sbin/ovsdb-server > gnr-sp-03:~ # ls -al /usr/local/sbin/ovsdb-server -rwxr-xr-x 1 root > root 20026112 Apr 24 11:06 /usr/local/sbin/ovsdb-server > > Mysterious to me where does /usr/sbin/ovsdb-server come from? None of them > can be sourced to a rpm package, and both reports the same version number. I > searched openvswitch keyword in /var/log/zypper.log and > /var/log/zypp/history. Any insight what could be the reason? Appreciate it. > Someone must have built it with "--prefix=/usr/". -- Adrián ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Clustered ovsdb-server 3.1 to 3.2+ upgrade question
Thank you for your answers!
> On 25 Apr 2024, at 13:01, Ilya Maximets wrote:
>
> On 4/25/24 11:51, Vladislav Odintsov wrote:
>>
>>
>>> On 25 Apr 2024, at 12:20, Ilya Maximets wrote:
>>>
>>> On 4/25/24 10:53, Vladislav Odintsov wrote:
Hi Ilya,
I’ve got question regarding upgrade of clustered ovsdb-servers from 3.1 to
3.2+ version.
ovsdb(7) states that the recommended path for upgrade is to upgrade
ovsdb-servers one-by-one
and that after upgrade ovsdb-server should be started with option
--disable-file-no-data-conversion
and after whole cluster is upgradede, it is needed to enable no-data
conversion via appctl.
I’ve ran through code and did some upgrade tests, so my question is:
Do I understand correctly, that if there is no necessity to call schema
conversion after start
and before end of cluster upgrade, so it is allowed just to restart
ovsdb-servers without
--disable-file-no-data-conversion option and this will not have any
downsides?
>>>
>>> Simply re-starting removing the option is enough. There is no need
>>> to enable specifically via appctl in this case.
>>
>> I’m talking about a bit another thing:
>> I want not to start each upgraded ovsdb-server with
>> --disable-file-no-data-conversion at all,
>> because it guaranteed in my case that there will be no schema conversions
>> before full cluster
>> upgrade is finished.
>> So, in this case there will be no need to enable it back via appctl or
>> removing ''disable"
>> option and restart.
>>
>> Am I right, or I’m missing something?
>
> Ah, sorry. If you're sure that there will be no conversion
> before all the servers are upgraded, then it should be fine
> to just upgrade as usual.
>
>>
>>>
>>> Also, we did actually backport parts of the format change to 3.1.
>>> It should be in v3.1.2 release and newer. So, technically, if you're
>>> performing upgrade from v3.1.2+, it should be safe to just upgrade as
>>> usual. ovsdb-servr v3.1.2 understands new database format, though it
>>> can't produce it on its own. See the following commit on branch-3.1:
>>>
>>> 9529e9aa967c ("ovsdb: Allow conversion records with no data in a clustered
>>> storage.")
>>
>> Hmm, nice. But mine source version is 3.0 and this is not applicable :)
>
> The same change should be available in v3.0.5.
>
>>
>>>
>>> It would be painful to document all the combinations of minor versions,
>>> so it's not an officially supported upgrade path, but it is there if
>>> you know what are you doing.
>>
>> Sure.
>>
>>>
>>> Best regards, Ilya Maximets.
>>
>>
>> Regards,
>> Vladislav Odintsov
Regards,
Vladislav Odintsov
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Clustered ovsdb-server 3.1 to 3.2+ upgrade question
On 4/25/24 11:51, Vladislav Odintsov wrote:
>
>
>> On 25 Apr 2024, at 12:20, Ilya Maximets wrote:
>>
>> On 4/25/24 10:53, Vladislav Odintsov wrote:
>>> Hi Ilya,
>>>
>>> I’ve got question regarding upgrade of clustered ovsdb-servers from 3.1 to
>>> 3.2+ version.
>>> ovsdb(7) states that the recommended path for upgrade is to upgrade
>>> ovsdb-servers one-by-one
>>> and that after upgrade ovsdb-server should be started with option
>>> --disable-file-no-data-conversion
>>> and after whole cluster is upgradede, it is needed to enable no-data
>>> conversion via appctl.
>>>
>>> I’ve ran through code and did some upgrade tests, so my question is:
>>> Do I understand correctly, that if there is no necessity to call schema
>>> conversion after start
>>> and before end of cluster upgrade, so it is allowed just to restart
>>> ovsdb-servers without
>>> --disable-file-no-data-conversion option and this will not have any
>>> downsides?
>>
>> Simply re-starting removing the option is enough. There is no need
>> to enable specifically via appctl in this case.
>
> I’m talking about a bit another thing:
> I want not to start each upgraded ovsdb-server with
> --disable-file-no-data-conversion at all,
> because it guaranteed in my case that there will be no schema conversions
> before full cluster
> upgrade is finished.
> So, in this case there will be no need to enable it back via appctl or
> removing ''disable"
> option and restart.
>
> Am I right, or I’m missing something?
Ah, sorry. If you're sure that there will be no conversion
before all the servers are upgraded, then it should be fine
to just upgrade as usual.
>
>>
>> Also, we did actually backport parts of the format change to 3.1.
>> It should be in v3.1.2 release and newer. So, technically, if you're
>> performing upgrade from v3.1.2+, it should be safe to just upgrade as
>> usual. ovsdb-servr v3.1.2 understands new database format, though it
>> can't produce it on its own. See the following commit on branch-3.1:
>>
>> 9529e9aa967c ("ovsdb: Allow conversion records with no data in a clustered
>> storage.")
>
> Hmm, nice. But mine source version is 3.0 and this is not applicable :)
The same change should be available in v3.0.5.
>
>>
>> It would be painful to document all the combinations of minor versions,
>> so it's not an officially supported upgrade path, but it is there if
>> you know what are you doing.
>
> Sure.
>
>>
>> Best regards, Ilya Maximets.
>
>
> Regards,
> Vladislav Odintsov
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Clustered ovsdb-server 3.1 to 3.2+ upgrade question
> On 25 Apr 2024, at 12:20, Ilya Maximets wrote:
>
> On 4/25/24 10:53, Vladislav Odintsov wrote:
>> Hi Ilya,
>>
>> I’ve got question regarding upgrade of clustered ovsdb-servers from 3.1 to
>> 3.2+ version.
>> ovsdb(7) states that the recommended path for upgrade is to upgrade
>> ovsdb-servers one-by-one
>> and that after upgrade ovsdb-server should be started with option
>> --disable-file-no-data-conversion
>> and after whole cluster is upgradede, it is needed to enable no-data
>> conversion via appctl.
>>
>> I’ve ran through code and did some upgrade tests, so my question is:
>> Do I understand correctly, that if there is no necessity to call schema
>> conversion after start
>> and before end of cluster upgrade, so it is allowed just to restart
>> ovsdb-servers without
>> --disable-file-no-data-conversion option and this will not have any
>> downsides?
>
> Simply re-starting removing the option is enough. There is no need
> to enable specifically via appctl in this case.
I’m talking about a bit another thing:
I want not to start each upgraded ovsdb-server with
--disable-file-no-data-conversion at all, because it guaranteed in my case that
there will be no schema conversions before full cluster upgrade is finished.
So, in this case there will be no need to enable it back via appctl or removing
''disable" option and restart.
Am I right, or I’m missing something?
>
> Also, we did actually backport parts of the format change to 3.1.
> It should be in v3.1.2 release and newer. So, technically, if you're
> performing upgrade from v3.1.2+, it should be safe to just upgrade as
> usual. ovsdb-servr v3.1.2 understands new database format, though it
> can't produce it on its own. See the following commit on branch-3.1:
>
> 9529e9aa967c ("ovsdb: Allow conversion records with no data in a clustered
> storage.")
Hmm, nice. But mine source version is 3.0 and this is not applicable :)
>
> It would be painful to document all the combinations of minor versions,
> so it's not an officially supported upgrade path, but it is there if
> you know what are you doing.
Sure.
>
> Best regards, Ilya Maximets.
Regards,
Vladislav Odintsov
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Clustered ovsdb-server 3.1 to 3.2+ upgrade question
On 4/25/24 10:53, Vladislav Odintsov wrote:
> Hi Ilya,
>
> I’ve got question regarding upgrade of clustered ovsdb-servers from 3.1 to
> 3.2+ version.
> ovsdb(7) states that the recommended path for upgrade is to upgrade
> ovsdb-servers one-by-one
> and that after upgrade ovsdb-server should be started with option
> --disable-file-no-data-conversion
> and after whole cluster is upgradede, it is needed to enable no-data
> conversion via appctl.
>
> I’ve ran through code and did some upgrade tests, so my question is:
> Do I understand correctly, that if there is no necessity to call schema
> conversion after start
> and before end of cluster upgrade, so it is allowed just to restart
> ovsdb-servers without
> --disable-file-no-data-conversion option and this will not have any downsides?
Simply re-starting removing the option is enough. There is no need
to enable specifically via appctl in this case.
Also, we did actually backport parts of the format change to 3.1.
It should be in v3.1.2 release and newer. So, technically, if you're
performing upgrade from v3.1.2+, it should be safe to just upgrade as
usual. ovsdb-servr v3.1.2 understands new database format, though it
can't produce it on its own. See the following commit on branch-3.1:
9529e9aa967c ("ovsdb: Allow conversion records with no data in a clustered
storage.")
It would be painful to document all the combinations of minor versions,
so it's not an officially supported upgrade path, but it is there if
you know what are you doing.
Best regards, Ilya Maximets.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Urgent Help needed: OVS 3.2.2 Strange TC DROPs
On 4/23/24 17:39, Gavin McKee wrote: If you look at both traces (non working and working) the thing that stands out to me is this At line 10 in the working file the following entry exists ct_state NEW tcp (SYN_SENT) orig [172.27.16.11.38793 > 172.27.31.189.9100] reply [172.27.31.189.9100 > 172.27.16.11.38793] zone 195 That's quite interesting, indeed. It means that in the working scenario, there is already a conntrack entry for the connection even though it's the TCP SYN and there is no flow in the kernel cache (the packet get's upcalled). Is the client reusing source ports by any chance? The fact that the conntrack info is already available makes the working case skip an upcall. Also, I'm curious about the drop in the TC_INGRESS. Are you using tc-offload? Can you dump the tc filter rules for your port? his doesn't happen in the non working file - I just see the following 3904992932904126 [swapper/125] 0 [kr] queue_userspace_packet #ddf92049d47aeff1d0b662562 (skb 18382861792850905088) n 4 if 38 (enp148s0f0_1) rxif 38 172.27.16.11.42303 > 172.27.31.189.9100 ttl 64 tos 0x0 id 36932 off 0 [DF] len 52 proto TCP (6) flags [S] seq 2266263186 win 42340 upcall_enqueue (miss) (125/3904992932890052) q 3682874017 ret 0 + 3904992932907247 [swapper/125] 0 [kr] ovs_dp_upcall #ddf92049d47aeff1d0b662562 (skb 18382861792850905088) n 5 if 38 (enp148s0f0_1) rxif 38 172.27.16.11.42303 > 172.27.31.189.9100 ttl 64 tos 0x0 id 36932 off 0 [DF] len 52 proto TCP (6) flags [S] seq 2266263186 win 42340 upcall_ret (125/3904992932890052) ret 0 I am wondering if a failure to track the ct_state SYN is causing the returning ACK to drop ? + 3904992936344421 [swapper/125] 0 [tp] skb:kfree_skb #ddf9204d21c48ff1d0b676c330c00 (skb 18382861792850913792) n 3 drop (TC_INGRESS) if 33 (genev_sys_6081) rxif 33 172.27.31.189.9100 > 172.27.16.11.42303 ttl 64 tos 0x0 id 0 off 0 [DF] len 52 proto TCP (6) flags [S.] seq 605271182 ack 2266263187 win 42340 On Mon, 22 Apr 2024 at 18:54, Gavin McKee wrote: Ok @Adrian Moreno @Flavio Leitner Two more detailed Retis traces attached. One is not working - the same session that I can't establish a TCP session to on port 9010 172.27.16.11.42303 > 172.27.31.189.9100 Then I restart Open vSwtich and try again 172.27.16.11.38793 > 172.27.31.189.9100 (this works post restart) It looks to me in the non working example that we - SEND SYN -> exits the tunnel interface genev_sys_6081 via enp148s0f0np0 - exactly as expected RECV ACK -> tcp_gro_receive then -> net:netif_receive_skb where we hit drop (TC_INGRESS) After a restart things seem to be very different Any ideas where to look next ? >> On Mon, 22 Apr 2024 at 12:49, Gavin McKee wrote: Hi Guys, We have had another occurrence of the issue today. In short - we try to open a connection from 172.27.22.90 -> 172.27.31.189 on port 9100. We see the SYN being received and the ACK being sent from the server back to the client 172.27.22.90. The server retransmits the ACK as it's never received by the server. When we run retis on the server side the ACK is being dropped - see (drop (TC_INGRESS) below) retis -p generic collect -c ovs,ct --ovs-track -f 'port 9100' (filtered on the ephemeral TCP port 40735) 3879878550398423 [swapper/125] 0 [tp] net:napi_gro_receive_entry #dc8ba9ec4a5d7ff4167e4dd466000 (skb 18393096588613658112) n 0 if 37 (enp148s0f0_1) + 3879878550401407 [swapper/125] 0 [k] tcp_gro_receive #dc8ba9ec4a5d7ff4167e4dd466000 (skb 18393096588613658112) n 1 if 37 (enp148s0f0_1) 172.27.22.90.40735 > 172.27.31.189.9100 ttl 64 tos 0x0 id 31996 off 0 [DF] len 52 proto TCP (6) flags [S] seq 1864480616 win 42340 + 3879878550402477 [swapper/125] 0 [tp] net:netif_receive_skb #dc8ba9ec4a5d7ff4167e4dd466000 (skb 18393096588613658112) n 2 if 37 (enp148s0f0_1) 172.27.22.90.40735 > 172.27.31.189.9100 ttl 64 tos 0x0 id 31996 off 0 [DF] len 52 proto TCP (6) flags [S] seq 1864480616 win 42340 + 3879878550409874 [swapper/125] 0 [tp] net:net_dev_queue #dc8ba9ec4a5d7ff4167e4dd466000 (skb 18393096588613658112) n 3 if 43 (genev_sys_6081) rxif 37 172.27.22.90.40735 > 172.27.31.189.9100 ttl 64 tos 0x0 id 31996 off 0 [DF] len 52 proto TCP (6) flags [S] seq 1864480616 win 42340 + 3879878550411535 [swapper/125] 0 [tp] net:net_dev_start_xmit #dc8ba9ec4a5d7ff4167e4dd466000 (skb 18393096588613658112) n 4 if 43 (genev_sys_6081) rxif 37 172.27.22.90.40735 > 172.27.31.189.9100 ttl 64 tos 0x0 id 31996 off 0 [DF] len 52 proto TCP (6) flags [S] seq 1864480616 win 42340 + 3879878550419012 [swapper/125] 0 [k] skb_scrub_packet #dc8ba9ec4a5d7ff4167e4dd466000 (skb 18393096588613658112) n 5 if 43 (genev_sys_6081) rxif 37 172.27.22.90.40735 > 172.27.31.189.9100 ttl 64 tos 0x0 id 31996 off 0 [DF] len 52 proto TCP (6) flags [S] seq 1864480616 win 42340 + 3879878550421222 [swapper/125] 0 [k] skb_scrub_packet #dc8ba9ec4a5d7ff4167e4dd466000 (skb 18393096588613658112) n 6 if 43 (gen
[ovs-discuss] Clustered ovsdb-server 3.1 to 3.2+ upgrade question
Hi Ilya, I’ve got question regarding upgrade of clustered ovsdb-servers from 3.1 to 3.2+ version. ovsdb(7) states that the recommended path for upgrade is to upgrade ovsdb-servers one-by-one and that after upgrade ovsdb-server should be started with option --disable-file-no-data-conversion and after whole cluster is upgradede, it is needed to enable no-data conversion via appctl. I’ve ran through code and did some upgrade tests, so my question is: Do I understand correctly, that if there is no necessity to call schema conversion after start and before end of cluster upgrade, so it is allowed just to restart ovsdb-servers without --disable-file-no-data-conversion option and this will not have any downsides? Regards, Vladislav Odintsov ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] ovsdb-server fails to start and report one unreferenced file path doesn't exist
On 4/25/24 04:29, Wang, Fei2 via discuss wrote: The ovs was configured by my colleague, I am trying to help with the mess-up. Took a deep dive, found two obsdb-server binary as you can see below: # ls -al /usr/sbin/ovsdb-server -rwxr-xr-x 1 root root 16172416 Apr 18 18:59 /usr/sbin/ovsdb-server gnr-sp-03:~ # ls -al /usr/local/sbin/ovsdb-server -rwxr-xr-x 1 root root 20026112 Apr 24 11:06 /usr/local/sbin/ovsdb-server Mysterious to me where does /usr/sbin/ovsdb-server come from? None of them can be sourced to a rpm package, and both reports the same version number. I searched openvswitch keyword in /var/log/zypper.log and /var/log/zypp/history. Any insight what could be the reason? Appreciate it. Someone must have built it with "--prefix=/usr/". -- Adrián ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
