Re: [Etherlab-users] Distributed Clocks and Oversampling - slave configuration of the ELM3604-002

[Merkel, Amos]

Hi Gavin,

thanks a lot for your explanations! 

I had the effect with two different PCs with quite different hardware and setup 
and the 4kHz seemed a pretty fixed limit, so I am expecting my problem to be 
more in the area of my DC Configuration instead of the drivers. 
The patchset I know and use, but so far I have avoided using dedicated drivers, 
so maybe this is worth a try as well. 

Right now I do not have access to the hardware - pandemic home office - but I 
will try your suggestions as soon as I can. 

Thanks again! 

-- 
Amos Merkel, M.Sc.
Group leader
Digitalization & Automation

ifas - Institute for Fluid Power Drives and Systems
RWTH Aachen University
Campus-Boulevard 30
52074 Aachen

Tel: +49 241 80-47737
Fax: +49 241 80-647712

www.ifas.rwth-aachen.de | Twitter | Facebook | YouTube | LinkedIn


-Ursprüngliche Nachricht-
Von: Gavin Lambert  
Gesendet: Donnerstag, 27. Mai 2021 01:57
An: Merkel, Amos ; etherlab-users@etherlab.org
Betreff: RE: Distributed Clocks and Oversampling - slave configuration of the 
ELM3604-002

I'm not sure what the Matlab equivalent of this is, but assuming that the slave 
follows the normal DC subordinated cycles mode (which I did glance through the 
datasheet but couldn't verify) then the way that the DC sync0 and sync1 cycle 
times need to be set is a little weird at first glance.

Typically the slave will sample on SYNC0 and report on SYNC1, so you want the 
SYNC1 pulse to match your domain packet cycle time and SYNC0 your desired 
sampling rate, but these are specified a bit oddly.  For example, with a domain 
cycle time of 1ms (1kHz) and a desired sampling rate of 4kHz, you need to set 
the Sync0 cycle time to 25 and the Sync1 cycle time to 75 (not to 
100 as you might have expected).

Some other advice is that for domain cycle times faster than 1ms you will 
definitely need to use the dedicated drivers (not the generic one), and for 
very fast rates you will likely need to use an RTOS (such as Xenomai).  This 
isn't needed if you have a fast sampling rate with a relatively slow domain 
cycle time, but obviously that will increase the data transfer requirements 
each domain cycle.

The unofficial patchset 
(https://sourceforge.net/u/uecasm/etherlab-patches/ci/default/tree/) does 
contain some improvements for DC, especially for subordinated cycles, so if 
you're not already using that then it may be worth a try.



Gavin Lambert
Senior Software Developer TOMRA Fresh Food

 


COMPAC SORTING EQUIPMENT LTD | 4 Henderson Pl | Onehunga | Auckland 1061 | New 
Zealand
Switchboard: +49 2630 96520 | https://www.tomra.com

The information contained in this communication and any attachment is 
confidential and may be legally privileged. It should only be read by the 
person(s) to whom it is addressed. If you have received this communication in 
error, please notify the sender and delete the communication.
From: Merkel, Amos
Sent: Thursday, 27 May 2021 12:59 am
To: etherlab-users@etherlab.org
Subject: [Etherlab-users] Distributed Clocks and Oversampling - slave 
configuration of the ELM3604-002


Hi everyone!

I am trying to get an ELM3604-002 by Beckhoff to work in the etherlab 
environment using Simulink and the generic slave block of the etherlab_lib. So 
far, I managed to get the slave operating and working for lower sample rates, 
but keep getting error states and "DC Invalid Sync Cycle Time" messages in 
dmesg for anything higher than 4kHz. Unfortunately, I do not really understand 
how the DC configuration works in detail, and the ELM3604-002 apparently does 
not support freerun mode, so I am stuck right now. 

Does anyone have an idea how to fix this? I would be most grateful for any 
hint! 
You will find my Matlab function with the slave struct below.

Best regards,

Amos

%% ELM3604-0002 Generic Slave Struct
% by Amos Merkel

%% notes
%    firmware does not support freerun mode %    Output Port 1..4: Value-Vector 
Channel 1..4, %   -length depends on oversampling 
factor %   -vector entry n correspondents to nth value 
of %   timestep, taken at time timestep + % 
  n*Timesteplength/oversampling factor %    Output Port 5..8: 
Status Output of Channel 1..4, if activated %    
-entries are: 1: error 2: Underrange 3: Overrange 4: Diag % 
   5: TxPDOState %    configure osm (in 1..15) and t_sample to your need %  
  check SDO Configuration below and configure to your needs (see Beckhoff 
documentation for %    additional options)

%% struct
function rv=elm3604

osm=3;  % osm:  oversampling mode in range 1..15 for 
the resulting oversampling rates: 1 2 4 5 8 10 16 20 25 32 40 50 64 80 100 
tsample=1/5000;%tsample=1e-3;   % tsample:  sample time in s 
StatusOutput=true;  % StatusOutput: Add Status Outputs for each Channel

osm=max(min(osm,15),1); % limit os to 

Re: [Etherlab-users] Distributed Clocks and Oversampling - slave configuration of the ELM3604-002

[Richard Hacker]

Have a look at el1262.m for a good example of using oversampling; even
the output vector and PDO's are calculated dynamically based on the
oversampling factor. For example, to turn on oversampling rate 4,
choose:

rv.SlaveConfig.dc = [hex2dec('730'),0,-4,-1,1,1,0,-1,0,0];

and you should be fine, instead of calculating and hard coding
CycleTimeSync1 manually! Notice how there is absolutely no reference to
the block's sampling time since it is implicitely calculated when using
factors.

Your calculation does not seem to be wrong though (the value of
CycleTimeSync1=15 is correct), but rather let the block calculate
these values itself. It reduces your problem set by one ;)

Your problem may be related to your hardware as Gavin replies.

On a side note: notice that elm3604.m is a function, so you could
parameterize it using:
function rv=elm3604(osfac)

and then calling
elm3604(4)
from the generic slave's "EtherCAT Slave Configuration struct"
parameter field. Like this you do not have to edit elm3604.m manually
every time you want to change the oversampling factor. Just makes life
a little easier

Richard 

On Wed, 2021-05-26 at 12:59 +, Merkel, Amos wrote:
> Hi everyone!
>  
> I am trying to get an ELM3604-002 by Beckhoff to work in the etherlab
> environment using Simulink and the generic slave block of the
> etherlab_lib.  
> So far, I managed to get the slave operating and working for lower
> sample rates, but keep getting error states and “DC Invalid Sync
> Cycle Time” messages in dmesg for anything higher than 4kHz.
> Unfortunately, I do not really understand how the DC configuration
> works in detail, and the ELM3604-002 apparently does not support
> freerun mode, so I am stuck right now.
>  
> Does anyone have an idea how to fix this? I would be most grateful
> for any hint!
> You will find my Matlab function with the slave struct below.
>  
> Best regards,
>  
> Amos
>  
> %% ELM3604-0002 Generic Slave Struct
> % by Amos Merkel
> %% notes
> %firmware does not support freerun mode
> %Output Port 1..4: Value-Vector Channel 1..4,
> %   -length depends on oversampling factor
> %   -vector entry n correspondents to nth
> value of
> %   timestep, taken at time timestep +
> %   n*Timesteplength/oversampling factor
> %Output Port 5..8: Status Output of Channel 1..4, if activated
> %-entries are: 1: error 2: Underrange 3:
> Overrange 4: Diag
> %5: TxPDOState
> %configure osm (in 1..15) and t_sample to your need
> %check SDO Configuration below and configure to your needs (see
> Beckhoff documentation for
> %additional options)
> %% struct
> function rv=elm3604
> osm=3;  % osm:  oversampling mode in range
> 1..15 for the resulting oversampling rates: 1 2 4 5 8 10 16 20 25 32
> 40 50 64 80 100
> tsample=1/5000;%tsample=1e-3;   % tsample:  sample time
> in s
> StatusOutput=true;  % StatusOutput: Add Status Outputs for each
> Channel
> osm=max(min(osm,15),1); % limit os to 1..100
> osf=[1 2 4 5 8 10 16 20 25 32 40 50 64 80 100]; %oversampling rates
> corresponding to oversampling mode
> os=osf(osm);% actual oversampling factor
> CycleTimeSync1=double(uint32(tsample(1)*(1-1/os)*1e9));
> %% -Slave Configuration--
> rv.SlaveConfig.vendor=2;
> rv.SlaveConfig.product=hex2dec('50219349');
> rv.SlaveConfig.description='ELM3604-0002';
> if os>1
> rv.SlaveConfig.dc=[hex2dec('0730') 0 -os -25 1 1
> CycleTimeSync1 0 0 0 ]; % DC-Setup [Assign_Activate Cycle_Time_Sync0
> Cycle_Time_Sync0_Factor Shift_Time_Sync0 Shift_Time_Sync0_Factor
> Shift_Time_Sync0_Input Cycle_Time_Sync1 Cycle_Time_Sync1_Factor
> Shift_Time_Sync1 Shift_Time_Sync1_Factor]
> 

-- 
Etherlab-users mailing list
Etherlab-users@etherlab.org
https://lists.etherlab.org/mailman/listinfo/etherlab-users

Re: [Etherlab-users] Distributed Clocks and Oversampling - slave configuration of the ELM3604-002

[Gavin Lambert]

I'm not sure what the Matlab equivalent of this is, but assuming that the slave 
follows the normal DC subordinated cycles mode (which I did glance through the 
datasheet but couldn't verify) then the way that the DC sync0 and sync1 cycle 
times need to be set is a little weird at first glance.

Typically the slave will sample on SYNC0 and report on SYNC1, so you want the 
SYNC1 pulse to match your domain packet cycle time and SYNC0 your desired 
sampling rate, but these are specified a bit oddly.  For example, with a domain 
cycle time of 1ms (1kHz) and a desired sampling rate of 4kHz, you need to set 
the Sync0 cycle time to 25 and the Sync1 cycle time to 75 (not to 
100 as you might have expected).

Some other advice is that for domain cycle times faster than 1ms you will 
definitely need to use the dedicated drivers (not the generic one), and for 
very fast rates you will likely need to use an RTOS (such as Xenomai).  This 
isn't needed if you have a fast sampling rate with a relatively slow domain 
cycle time, but obviously that will increase the data transfer requirements 
each domain cycle.

The unofficial patchset 
(https://sourceforge.net/u/uecasm/etherlab-patches/ci/default/tree/) does 
contain some improvements for DC, especially for subordinated cycles, so if 
you're not already using that then it may be worth a try.



Gavin Lambert
Senior Software Developer TOMRA Fresh Food

 


COMPAC SORTING EQUIPMENT LTD | 4 Henderson Pl | Onehunga | Auckland 1061 | New 
Zealand
Switchboard: +49 2630 96520 | https://www.tomra.com

The information contained in this communication and any attachment is 
confidential and may be legally privileged. It should only be read by the 
person(s) to whom it is addressed. If you have received this communication in 
error, please notify the sender and delete the communication.
From: Merkel, Amos
Sent: Thursday, 27 May 2021 12:59 am
To: etherlab-users@etherlab.org
Subject: [Etherlab-users] Distributed Clocks and Oversampling - slave 
configuration of the ELM3604-002


Hi everyone!

I am trying to get an ELM3604-002 by Beckhoff to work in the etherlab 
environment using Simulink and the generic slave block of the etherlab_lib.  
So far, I managed to get the slave operating and working for lower sample 
rates, but keep getting error states and "DC Invalid Sync Cycle Time" messages 
in dmesg for anything higher than 4kHz. Unfortunately, I do not really 
understand how the DC configuration works in detail, and the ELM3604-002 
apparently does not support freerun mode, so I am stuck right now. 

Does anyone have an idea how to fix this? I would be most grateful for any 
hint! 
You will find my Matlab function with the slave struct below.

Best regards,

Amos

%% ELM3604-0002 Generic Slave Struct 
% by Amos Merkel

%% notes 
%    firmware does not support freerun mode
%    Output Port 1..4: Value-Vector Channel 1..4, 
%   -length depends on oversampling factor 
%   -vector entry n correspondents to nth value of
%   timestep, taken at time timestep +
%   n*Timesteplength/oversampling factor
%    Output Port 5..8: Status Output of Channel 1..4, if activated
%    -entries are: 1: error 2: Underrange 3: Overrange 
4: Diag
%    5: TxPDOState
%    configure osm (in 1..15) and t_sample to your need
%    check SDO Configuration below and configure to your needs (see Beckhoff 
documentation for
%    additional options)

%% struct
function rv=elm3604

osm=3;  % osm:  oversampling mode in range 1..15 for 
the resulting oversampling rates: 1 2 4 5 8 10 16 20 25 32 40 50 64 80 100
tsample=1/5000;%tsample=1e-3;   % tsample:  sample time in s
StatusOutput=true;  % StatusOutput: Add Status Outputs for each Channel

osm=max(min(osm,15),1); % limit os to 1..100
osf=[1 2 4 5 8 10 16 20 25 32 40 50 64 80 100]; %oversampling rates 
corresponding to oversampling mode
os=osf(osm);    % actual oversampling factor

CycleTimeSync1=double(uint32(tsample(1)*(1-1/os)*1e9));

%% -Slave Configuration--

rv.SlaveConfig.vendor=2;
rv.SlaveConfig.product=hex2dec('50219349');
rv.SlaveConfig.description='ELM3604-0002';
if os>1
    rv.SlaveConfig.dc=[hex2dec('0730') 0 -os -25 1 1 CycleTimeSync1 0 0 0 
]; % DC-Setup [Assign_Activate Cycle_Time_Sync0 Cycle_Time_Sync0_Factor 
Shift_Time_Sync0 Shift_Time_Sync0_Factor Shift_Time_Sync0_Input 
Cycle_Time_Sync1 Cycle_Time_Sync1_Factor Shift_Time_Sync1 
Shift_Time_Sync1_Factor]

end
%% --SyncManager (Sm) Configuration--

rv.SlaveConfig.sm={ ...
    {0, 0, {}}, ...
    {1, 1, {}},...
    {2, 0, {}},...
    {3, 1, {
    {hex2dec('1a00'), [ % PAI TxPDO-Map Status Ch.1
    hex2dec('6000'), 01, 08;    %   No of Samples   8bit
    hex2dec('6000'), 09, 1; %   Error   1 bit
 

Re: [etherlab-users] Distributed Clocks

[Henry Bausley]

Make sure you are not sending anything like 
ecrt_master_sdo_download/ecrt_master_sdo_upload from additional threads or 
anything else.

We use xenomai but always in kernel mode.  We can control up to 8kHz without 
issues.




On Fri, 2016-04-15 at 00:16 +0200, Richard Hacker wrote:
> On 14.04.2016 23:51, Thomas Bitsky Jr wrote:
> > Over the course of the last few weeks, I have done the following:
> > - Built and installed a Xenomai 2.6.4 kernel.
> > - Switched x86 hardware
> > - Adapted the program to Xenomai
> > - Rewrote the program so that it is completely asynchronous
> > - Fixed the distributed clock code I had written.
> Woa, respect, that is crazy!! I was wondering what had happened to your 
> experiments.
> 
> >
> > The result has been a reduced latency, worst case latency of +/-10 
> > microseconds when not running the program. I am still running the user 
> > space at a rate of 4000Hz. I am using the e1000e ethercat driver.
> >
> >
> > However, this did not solve my problem. When I run live with the EtherCAT 
> > master, I get over runs, skipped packet and eventually a delay so long that 
> > it causes the Synchronized Clock watchdog to barf.
> >
del
> > After scoping, I have confirmed that the delays are always in these two 
> > functions:
> >
> > ecrt_master_receive
> > ecrt_master_send
> Now that is a good clue. Have you instrumented the time just before and 
> just after these calls? Probably ;)
> 
> The Intel chipset is designed for high throughput with as little CPU 
> interaction as possible. It has things like interrupt throttling, 
> buffers and the works - poison for any real time application.
> 
> Although we have not experienced these issues and are successfully using 
> this driver, we try to use the most dumb (!) network cards available: 
> RTL8139 chipset is a good example. Paradoxically, they work best: when 
> they receive a packet of data, it is available immediately; when a 
> packet is sent to the card, it is transmitted immediately - no 
> buffering, no throttling.
> 
> >
> > In fact, I can comment out those functions but leave everything else 
> > intact, including ecrt_domain_process, ecrt_domain_queue and even the 
> > distributed clock functions, and the timing is near perfect. However, if I 
> > add back in ecrt_master_receive/send, they return inconsistently.
> >
> > Can anyone tell me why ecrt_master_receive and ecrt_master_send would take 
> > unreasonably long? Could this be a problem with calling those functions 
> > from user space? Could it be a problem with the e1000e driver? I’m not 
> > exactly sure where to go from here except to try moving the whole program 
> > into the kernel, but, at this point, I’m not sure that will solve the issue.
> >
> > Any insights or thoughts are greatly appreciated.
> >
> > Thanks!
> > Tom
> >
> >
> >
> >
> >
> > On 3/2/16, 10:04 AM, "Henry Bausley"  wrote:
> >
> >>
> >> Are you using the dc_clock_adjust to modify your sleep time?  ie.
> >> something like
> >>
> >> clock_nanosleep(CLOCK_MONOTONIC,TIMER_ABSTIME,>wakeupTime,NULL);
> >> TIMESPEC_ADD_NS(inst->wakeupTime,100 + dc_clock_adjust);
> >>
> >> Does your inst->cycleNs include the clock adjustement?
> >>
> >> I have found the best debug tool for DC is an oscilloscope and a board
> >> that you probe the SOF and sync0 .  If you see SOF wander with respect
> >> to sync0 it won't work correctly.  If they don't you are good.
> >>
> >> On Wed, 2016-03-02 at 14:47 +, Thomas Bitsky Jr wrote:
> >>> Thanks for the response, Richard.
> >>>
> >>> CyclicTest on the hardware:
> >>> user@core:~$ sudo cyclictest --smp -p95 -m
> >>> # /dev/cpu_dma_latency set to 0us
> >>> policy: fifo: loadavg: 0.01 0.04 0.04 1/162 1547
> >>>
> >>> T: 0 ( 1543) P:95 I:1000 C:  26093 Min:  6 Act:   30 Avg:   25 Max:   
> >>>56
> >>> T: 1 ( 1544) P:95 I:1500 C:  17386 Min:  6 Act:   14 Avg:   23 Max:   
> >>>66
> >>> T: 2 ( 1545) P:95 I:2000 C:  13036 Min:  5 Act:   28 Avg:   29 Max:   
> >>>57
> >>> T: 3 ( 1546) P:95 I:2500 C:  10424 Min:  6 Act:   30 Avg:   29 Max:   
> >>>58
> >>>
> >>>
> >>>
> >>> So, the max latency is definitely in the ballpark of what I was saying 
> >>> last night.
> >>>
> >>> Setting cylictest with a break value fails almost instantly.
> >>>
> >>> root@core:/sys/kernel/debug/tracing# cyclictest --smp -p95 -f -b 200
> >>> # /dev/cpu_dma_latency set to 0us
> >>> INFO: debugfs mountpoint: /sys/kernel/debug/tracing/
> >>> policy: fifo: loadavg: 0.00 0.01 0.05 3/165 1703
> >>>
> >>> T: 0 ( 1700) P:95 I:1000 C:  0 Min:100 Act:0 Avg:0 Max:   
> >>> 0
> >>> T: 1 ( 1701) P:95 I:1500 C:  0 Min:100 Act:0 Avg:0 Max:   
> >>> 0
> >>> T: 2 (0) P:95 I:2000 C:  0 Min:100 Act:0 Avg:0 Max:   
> >>> 0
> >>> T: 3 ( 1703) P:95 I:2500 C:  0 Min:100 Act:0 Avg:0 Max:   
> >>> 0
> >>> # Thread Ids: 01700 01701 01702 01703
> >>> # Break thread: 1702
> >>> # Break 

Re: [etherlab-users] Distributed Clocks

[Bilko AS, Oguz Dilmac]

Hi,

We are using e1000e too. We are using RTAI kernel modules instead of 
user space. We are not using any patch or any configuration for the 
e1000e either.
If I only run the latency test it goes around 10us. But when our program 
is running, jitter is around 50us. I suspect somehow the semaphore calls 
are guilty.


Have you tried to reduce the timing? We are operating at 1ms sample 
period. And some times even 2ms.
If I went below 1ms I get too many jitter and warnings from EtherCAT 
library too.


Best regards,
Oguz.

15.4.2016 00:51 tarihinde Thomas Bitsky Jr yazdı:

Over the course of the last few weeks, I have done the following:
- Built and installed a Xenomai 2.6.4 kernel.
- Switched x86 hardware
- Adapted the program to Xenomai
- Rewrote the program so that it is completely asynchronous
- Fixed the distributed clock code I had written.

The result has been a reduced latency, worst case latency of +/-10 microseconds 
when not running the program. I am still running the user space at a rate of 
4000Hz. I am using the e1000e ethercat driver.


However, this did not solve my problem. When I run live with the EtherCAT 
master, I get over runs, skipped packet and eventually a delay so long that it 
causes the Synchronized Clock watchdog to barf.

After scoping, I have confirmed that the delays are always in these two 
functions:

ecrt_master_receive
ecrt_master_send

In fact, I can comment out those functions but leave everything else intact, 
including ecrt_domain_process, ecrt_domain_queue and even the distributed clock 
functions, and the timing is near perfect. However, if I add back in 
ecrt_master_receive/send, they return inconsistently.

Can anyone tell me why ecrt_master_receive and ecrt_master_send would take 
unreasonably long? Could this be a problem with calling those functions from 
user space? Could it be a problem with the e1000e driver? I’m not exactly sure 
where to go from here except to try moving the whole program into the kernel, 
but, at this point, I’m not sure that will solve the issue.

Any insights or thoughts are greatly appreciated.

Thanks!
Tom





On 3/2/16, 10:04 AM, "Henry Bausley"  wrote:


Are you using the dc_clock_adjust to modify your sleep time?  ie.
something like
  
clock_nanosleep(CLOCK_MONOTONIC,TIMER_ABSTIME,>wakeupTime,NULL);

TIMESPEC_ADD_NS(inst->wakeupTime,100 + dc_clock_adjust);

Does your inst->cycleNs include the clock adjustement?

I have found the best debug tool for DC is an oscilloscope and a board
that you probe the SOF and sync0 .  If you see SOF wander with respect
to sync0 it won't work correctly.  If they don't you are good.

On Wed, 2016-03-02 at 14:47 +, Thomas Bitsky Jr wrote:

Thanks for the response, Richard.

CyclicTest on the hardware:
user@core:~$ sudo cyclictest --smp -p95 -m
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.01 0.04 0.04 1/162 1547

T: 0 ( 1543) P:95 I:1000 C:  26093 Min:  6 Act:   30 Avg:   25 Max:  56
T: 1 ( 1544) P:95 I:1500 C:  17386 Min:  6 Act:   14 Avg:   23 Max:  66
T: 2 ( 1545) P:95 I:2000 C:  13036 Min:  5 Act:   28 Avg:   29 Max:  57
T: 3 ( 1546) P:95 I:2500 C:  10424 Min:  6 Act:   30 Avg:   29 Max:  58



So, the max latency is definitely in the ballpark of what I was saying last 
night.

Setting cylictest with a break value fails almost instantly.

root@core:/sys/kernel/debug/tracing# cyclictest --smp -p95 -f -b 200
# /dev/cpu_dma_latency set to 0us
INFO: debugfs mountpoint: /sys/kernel/debug/tracing/
policy: fifo: loadavg: 0.00 0.01 0.05 3/165 1703

T: 0 ( 1700) P:95 I:1000 C:  0 Min:100 Act:0 Avg:0 Max:   0
T: 1 ( 1701) P:95 I:1500 C:  0 Min:100 Act:0 Avg:0 Max:   0
T: 2 (0) P:95 I:2000 C:  0 Min:100 Act:0 Avg:0 Max:   0
T: 3 ( 1703) P:95 I:2500 C:  0 Min:100 Act:0 Avg:0 Max:   0
# Thread Ids: 01700 01701 01702 01703
# Break thread: 1702
# Break value: 217



[snip]
First things first: when you are running RT proggies, you need a stable
clock source and a preemptable kernel!
[/snip]


I am using RT Preempt:
# uname -a
Linux core 3.12.50-rt68 #1 SMP PREEMPT RT Mon Nov 23 18:17:14 CST 2015 i686 
i686 i686 GNU/Linux


Thanks!
Thomas C. Bitsky Jr. | Lead Developer
ADC | automateddesign.com 
P: 630-783-1150 F: 630-783-1159 M: 630-632-6679

Follow ADC news and media:
Facebook  | Twitter 
 | YouTube 









On 3/2/16, 5:30 AM, "etherlab-users on behalf of Richard Hacker" 
 wrote:


Hi John,

Skipped frames are either a timing or a hardware problem (faulty
EtherCAT slaves, cabling, noise). Some slaves don't care, DC slaves puke!

First things first: when you are running RT proggies, you need a stable
clock source and a preemptable 

Re: [etherlab-users] Distributed Clocks

[Thomas Bitsky Jr]

>>Although we have not experienced these issues and are successfully using this 
>>driver,


Is that to say you are using the E1000e driver? Do you modify its configuration 
at all?

Tom

(Sent from my mobile device.)

On Apr 14, 2016, at 5:16 PM, Richard Hacker > 
wrote:

Although we have not experienced these issues and are successfully using this 
driver,
___
etherlab-users mailing list
etherlab-users@etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-users

Re: [etherlab-users] Distributed Clocks

[Richard Hacker]

On 14.04.2016 23:51, Thomas Bitsky Jr wrote:

Over the course of the last few weeks, I have done the following:
- Built and installed a Xenomai 2.6.4 kernel.
- Switched x86 hardware
- Adapted the program to Xenomai
- Rewrote the program so that it is completely asynchronous
- Fixed the distributed clock code I had written.
Woa, respect, that is crazy!! I was wondering what had happened to your 
experiments.




The result has been a reduced latency, worst case latency of +/-10 microseconds 
when not running the program. I am still running the user space at a rate of 
4000Hz. I am using the e1000e ethercat driver.


However, this did not solve my problem. When I run live with the EtherCAT 
master, I get over runs, skipped packet and eventually a delay so long that it 
causes the Synchronized Clock watchdog to barf.

After scoping, I have confirmed that the delays are always in these two 
functions:

ecrt_master_receive
ecrt_master_send
Now that is a good clue. Have you instrumented the time just before and 
just after these calls? Probably ;)


The Intel chipset is designed for high throughput with as little CPU 
interaction as possible. It has things like interrupt throttling, 
buffers and the works - poison for any real time application.


Although we have not experienced these issues and are successfully using 
this driver, we try to use the most dumb (!) network cards available: 
RTL8139 chipset is a good example. Paradoxically, they work best: when 
they receive a packet of data, it is available immediately; when a 
packet is sent to the card, it is transmitted immediately - no 
buffering, no throttling.




In fact, I can comment out those functions but leave everything else intact, 
including ecrt_domain_process, ecrt_domain_queue and even the distributed clock 
functions, and the timing is near perfect. However, if I add back in 
ecrt_master_receive/send, they return inconsistently.

Can anyone tell me why ecrt_master_receive and ecrt_master_send would take 
unreasonably long? Could this be a problem with calling those functions from 
user space? Could it be a problem with the e1000e driver? I’m not exactly sure 
where to go from here except to try moving the whole program into the kernel, 
but, at this point, I’m not sure that will solve the issue.

Any insights or thoughts are greatly appreciated.

Thanks!
Tom





On 3/2/16, 10:04 AM, "Henry Bausley"  wrote:



Are you using the dc_clock_adjust to modify your sleep time?  ie.
something like

clock_nanosleep(CLOCK_MONOTONIC,TIMER_ABSTIME,>wakeupTime,NULL);
TIMESPEC_ADD_NS(inst->wakeupTime,100 + dc_clock_adjust);

Does your inst->cycleNs include the clock adjustement?

I have found the best debug tool for DC is an oscilloscope and a board
that you probe the SOF and sync0 .  If you see SOF wander with respect
to sync0 it won't work correctly.  If they don't you are good.

On Wed, 2016-03-02 at 14:47 +, Thomas Bitsky Jr wrote:

Thanks for the response, Richard.

CyclicTest on the hardware:
user@core:~$ sudo cyclictest --smp -p95 -m
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.01 0.04 0.04 1/162 1547

T: 0 ( 1543) P:95 I:1000 C:  26093 Min:  6 Act:   30 Avg:   25 Max:  56
T: 1 ( 1544) P:95 I:1500 C:  17386 Min:  6 Act:   14 Avg:   23 Max:  66
T: 2 ( 1545) P:95 I:2000 C:  13036 Min:  5 Act:   28 Avg:   29 Max:  57
T: 3 ( 1546) P:95 I:2500 C:  10424 Min:  6 Act:   30 Avg:   29 Max:  58



So, the max latency is definitely in the ballpark of what I was saying last 
night.

Setting cylictest with a break value fails almost instantly.

root@core:/sys/kernel/debug/tracing# cyclictest --smp -p95 -f -b 200
# /dev/cpu_dma_latency set to 0us
INFO: debugfs mountpoint: /sys/kernel/debug/tracing/
policy: fifo: loadavg: 0.00 0.01 0.05 3/165 1703

T: 0 ( 1700) P:95 I:1000 C:  0 Min:100 Act:0 Avg:0 Max:   0
T: 1 ( 1701) P:95 I:1500 C:  0 Min:100 Act:0 Avg:0 Max:   0
T: 2 (0) P:95 I:2000 C:  0 Min:100 Act:0 Avg:0 Max:   0
T: 3 ( 1703) P:95 I:2500 C:  0 Min:100 Act:0 Avg:0 Max:   0
# Thread Ids: 01700 01701 01702 01703
# Break thread: 1702
# Break value: 217



[snip]
First things first: when you are running RT proggies, you need a stable
clock source and a preemptable kernel!
[/snip]


I am using RT Preempt:
# uname -a
Linux core 3.12.50-rt68 #1 SMP PREEMPT RT Mon Nov 23 18:17:14 CST 2015 i686 
i686 i686 GNU/Linux


Thanks!
Thomas C. Bitsky Jr. | Lead Developer
ADC | automateddesign.com 
P: 630-783-1150 F: 630-783-1159 M: 630-632-6679

Follow ADC news and media:
Facebook  | Twitter 
 | YouTube 









On 3/2/16, 5:30 AM, "etherlab-users on behalf of Richard Hacker" 
 wrote:


Hi John,

Skipped 

Re: [etherlab-users] Distributed Clocks

[Matthieu Bec]

Hello Thomas,

These 13/50 are max latency numbers? both systems would qualify as reasonably 
good wrt. rt-preempt.

What I find surprising somehow: even your 'good' system shows so much variance: 
min: 6789, max: 50345
Shouldn't that be much always the same? Maybe something else is happening and 
amplified on the 'bad' system.

Regards,
Matthieu








On 3/8/16, 7:47 PM, "Thomas Bitsky Jr"  wrote:

Matthieu,

[snip]
in other words, you are positive the difference starts with ethercat master?

[/snip]

Yes. I have measured the time around that code block. That is where the 
variance occurs that causes me to have unmatched packets. 

That being said, one machine (i3) has a 13 microsecond latency, the other 
(celeron) has a 50 microsecond latency. If the ethercat userspace API is 
“sleeping” my process, then that would probably explain the problem. 

Are these problems avoided or reduced in the kernel space, or will I have the 
exact same timing?

Thanks!








On 3/8/16, 9:32 PM, "Matthieu Bec"  wrote:

>Hello Thomas,
>
>Both systems give similar cyclictest results? in other words, you are positive 
>the difference starts with ethercat master?
>I'm not clear for example if celeron and i3 fall under the same processor 
>family when you compile your kernel. Still such difference, explanation would 
>seem far fetched.
>
>Regards,
>Matthieu
>
>
>
>
>On 3/8/16, 8:09 AM, "etherlab-users on behalf of Thomas Bitsky Jr" 
> 
>wrote:
>
>Richard,
>
>I put a lot of time into your suggestions the past few days, and here is where 
>I’m at.
>
>[snip]
>we use CLOCK_MONOTONIC for ecrt_master_application_time(), in fact we 
>use it thoughout. Probably not the solution, but worth a try.
>[/snip]
>
>
>As I generally do; thanks for catching that error for me.
>
>[snip]
>Once your cyclic test is stable (~10us max jitter), you may start 
>looking for delays in your code.
>[/snip]
>
>
>
>Semaphore and mutex calls were waking up inconsistently (the rt tests reported 
>the same thing). I re-designed the API to eliminate the need for them in my 
>code. I also reduced the memory footprint. Send, recv and timespec functions 
>are the only system calls remaining.
>
>This brought the rest of the program down to very little execution time. I 
>went through function by function, but couldn’t find the reason for the long 
>execution times in my cleaned up code.
>
>However, I still have a surprisingly long execution time. The functions that 
>cause the problem are from the ethercat master library.
>
>
>ecrt_master_receive(master_);
>   
>   
>
>   ecrt_domain_process(lrwDomainMgr_.domain);  
>   ecrt_domain_process(noLrwWriteDomainMgr_.domain);
>   ecrt_domain_process(noLrwReadDomainMgr_.domain);
>
>
>
>I have tried this on two pieces of hardware. They run the same operating 
>system and configuration, same EtherLAB master version, same Ethernet driver.
>
>System 1 (Intel i3 NUC, 1 Intel e1000e Ethernet port): exec min 6789 .. max 
>50345  (nano seconds)
>System 2 (Intel quad celeoron, 4 x e1000e Ethernet port): exec min 15588 … max 
>323376 (nano seconds)
>(Both systems: Ubuntu Server 14.04LTS, Linux core 3.12.50-rt68 #1 SMP PREEMPT 
>RT)
>
>On system 1, the unmatched packets never happen; for whatever reason, those 
>functions are returning much more rapidly. On the second system, the worst 
>case happens about once per minute, once every two minutes. 
>
>1. Am I unreasonable in thinking these times are excessive? I understand that 
>the system is communicating out to external hardware and there will be a 
>delay, but the difference between the two computers is surprising to me.
>2. Is this a consequence of using the EtherCAT master from user space? I’ve 
>always used it from kernel space before. However, I’ve never done as extensive 
>of timing measurements before. The EtherLAB documentation mentioned that the 
>userspace library only added 1 microsecond of overhead, if I recall. Is this 
>as good as it gets from the user space?
>3. Is there a way to address this problem, or is it completely dependent upon 
>the hardware?
>
>
>Thanks for any insights to the problem, and for all the advice everyone has 
>submitted this far.
>
>
>
>
>
>
>
>On 3/3/16, 1:59 PM, "Richard Hacker"  wrote:
>
>>Hi,
>>
>>we use CLOCK_MONOTONIC for ecrt_master_application_time(), in fact we 
>>use it thoughout. Probably not the solution, but worth a try.
>>
>>- Richard
>>
>>On 03.03.2016 17:08, Thomas Bitsky Jr wrote:
>>>
>>> [snip]
>>> Are you using the dc_clock_adjust to modify your sleep time?  ie.
>>> something like
>>> [/snip]
>>>
>>>
>>> I’ve been trying, but the results are all over the place and I’m obviously 
>>> not understanding what needs to be done with the clock.
>>>
>>> My latest attempt is has the scan time ticking from anywhere between 3930 
>>> and 4100 scans per second (reported by 

Re: [etherlab-users] Distributed Clocks

[Thomas Bitsky Jr]

Matthieu,

[snip]
in other words, you are positive the difference starts with ethercat master?

[/snip]

Yes. I have measured the time around that code block. That is where the 
variance occurs that causes me to have unmatched packets. 

That being said, one machine (i3) has a 13 microsecond latency, the other 
(celeron) has a 50 microsecond latency. If the ethercat userspace API is 
“sleeping” my process, then that would probably explain the problem. 

Are these problems avoided or reduced in the kernel space, or will I have the 
exact same timing?

Thanks!








On 3/8/16, 9:32 PM, "Matthieu Bec"  wrote:

>Hello Thomas,
>
>Both systems give similar cyclictest results? in other words, you are positive 
>the difference starts with ethercat master?
>I'm not clear for example if celeron and i3 fall under the same processor 
>family when you compile your kernel. Still such difference, explanation would 
>seem far fetched.
>
>Regards,
>Matthieu
>
>
>
>
>On 3/8/16, 8:09 AM, "etherlab-users on behalf of Thomas Bitsky Jr" 
> 
>wrote:
>
>Richard,
>
>I put a lot of time into your suggestions the past few days, and here is where 
>I’m at.
>
>[snip]
>we use CLOCK_MONOTONIC for ecrt_master_application_time(), in fact we 
>use it thoughout. Probably not the solution, but worth a try.
>[/snip]
>
>
>As I generally do; thanks for catching that error for me.
>
>[snip]
>Once your cyclic test is stable (~10us max jitter), you may start 
>looking for delays in your code.
>[/snip]
>
>
>
>Semaphore and mutex calls were waking up inconsistently (the rt tests reported 
>the same thing). I re-designed the API to eliminate the need for them in my 
>code. I also reduced the memory footprint. Send, recv and timespec functions 
>are the only system calls remaining.
>
>This brought the rest of the program down to very little execution time. I 
>went through function by function, but couldn’t find the reason for the long 
>execution times in my cleaned up code.
>
>However, I still have a surprisingly long execution time. The functions that 
>cause the problem are from the ethercat master library.
>
>
>ecrt_master_receive(master_);
>   
>   
>
>   ecrt_domain_process(lrwDomainMgr_.domain);  
>   ecrt_domain_process(noLrwWriteDomainMgr_.domain);
>   ecrt_domain_process(noLrwReadDomainMgr_.domain);
>
>
>
>I have tried this on two pieces of hardware. They run the same operating 
>system and configuration, same EtherLAB master version, same Ethernet driver.
>
>System 1 (Intel i3 NUC, 1 Intel e1000e Ethernet port): exec min 6789 .. max 
>50345  (nano seconds)
>System 2 (Intel quad celeoron, 4 x e1000e Ethernet port): exec min 15588 … max 
>323376 (nano seconds)
>(Both systems: Ubuntu Server 14.04LTS, Linux core 3.12.50-rt68 #1 SMP PREEMPT 
>RT)
>
>On system 1, the unmatched packets never happen; for whatever reason, those 
>functions are returning much more rapidly. On the second system, the worst 
>case happens about once per minute, once every two minutes. 
>
>1. Am I unreasonable in thinking these times are excessive? I understand that 
>the system is communicating out to external hardware and there will be a 
>delay, but the difference between the two computers is surprising to me.
>2. Is this a consequence of using the EtherCAT master from user space? I’ve 
>always used it from kernel space before. However, I’ve never done as extensive 
>of timing measurements before. The EtherLAB documentation mentioned that the 
>userspace library only added 1 microsecond of overhead, if I recall. Is this 
>as good as it gets from the user space?
>3. Is there a way to address this problem, or is it completely dependent upon 
>the hardware?
>
>
>Thanks for any insights to the problem, and for all the advice everyone has 
>submitted this far.
>
>
>
>
>
>
>
>On 3/3/16, 1:59 PM, "Richard Hacker"  wrote:
>
>>Hi,
>>
>>we use CLOCK_MONOTONIC for ecrt_master_application_time(), in fact we 
>>use it thoughout. Probably not the solution, but worth a try.
>>
>>- Richard
>>
>>On 03.03.2016 17:08, Thomas Bitsky Jr wrote:
>>>
>>> [snip]
>>> Are you using the dc_clock_adjust to modify your sleep time?  ie.
>>> something like
>>> [/snip]
>>>
>>>
>>> I’ve been trying, but the results are all over the place and I’m obviously 
>>> not understanding what needs to be done with the clock.
>>>
>>> My latest attempt is has the scan time ticking from anywhere between 3930 
>>> and 4100 scans per second (reported by ethercat master command) and does 
>>> nothing to eliminate skipped datagrams. I don’t think I’m fundamentally 
>>> understanding what needs to be done.
>>>
>>> My latest code is below. If anyone has any insights, it would be greatly 
>>> appreciated.
>>>
>>> 
>>>
>>> int64_t  dc_adjust_ns = 0 ;
>>> int32_t  dc_diff_ns = 0;
>>>
>>>
>>>
>>> void calculateClockDrift(void) {
>>>  uint32_t ref_time = 0;
>>>  uint64_t 

Re: [etherlab-users] Distributed Clocks

[Matthieu Bec]

Hello Thomas,

Both systems give similar cyclictest results? in other words, you are positive 
the difference starts with ethercat master?
I'm not clear for example if celeron and i3 fall under the same processor 
family when you compile your kernel. Still such difference, explanation would 
seem far fetched.

Regards,
Matthieu




On 3/8/16, 8:09 AM, "etherlab-users on behalf of Thomas Bitsky Jr" 
 
wrote:

Richard,

I put a lot of time into your suggestions the past few days, and here is where 
I’m at.

[snip]
we use CLOCK_MONOTONIC for ecrt_master_application_time(), in fact we 
use it thoughout. Probably not the solution, but worth a try.
[/snip]


As I generally do; thanks for catching that error for me.

[snip]
Once your cyclic test is stable (~10us max jitter), you may start 
looking for delays in your code.
[/snip]



Semaphore and mutex calls were waking up inconsistently (the rt tests reported 
the same thing). I re-designed the API to eliminate the need for them in my 
code. I also reduced the memory footprint. Send, recv and timespec functions 
are the only system calls remaining.

This brought the rest of the program down to very little execution time. I went 
through function by function, but couldn’t find the reason for the long 
execution times in my cleaned up code.

However, I still have a surprisingly long execution time. The functions that 
cause the problem are from the ethercat master library.


ecrt_master_receive(master_);



ecrt_domain_process(lrwDomainMgr_.domain);  
ecrt_domain_process(noLrwWriteDomainMgr_.domain);
ecrt_domain_process(noLrwReadDomainMgr_.domain);



I have tried this on two pieces of hardware. They run the same operating system 
and configuration, same EtherLAB master version, same Ethernet driver.

System 1 (Intel i3 NUC, 1 Intel e1000e Ethernet port): exec min 6789 .. max 
50345  (nano seconds)
System 2 (Intel quad celeoron, 4 x e1000e Ethernet port): exec min 15588 … max 
323376 (nano seconds)
(Both systems: Ubuntu Server 14.04LTS, Linux core 3.12.50-rt68 #1 SMP PREEMPT 
RT)

On system 1, the unmatched packets never happen; for whatever reason, those 
functions are returning much more rapidly. On the second system, the worst case 
happens about once per minute, once every two minutes. 

1. Am I unreasonable in thinking these times are excessive? I understand that 
the system is communicating out to external hardware and there will be a delay, 
but the difference between the two computers is surprising to me.
2. Is this a consequence of using the EtherCAT master from user space? I’ve 
always used it from kernel space before. However, I’ve never done as extensive 
of timing measurements before. The EtherLAB documentation mentioned that the 
userspace library only added 1 microsecond of overhead, if I recall. Is this as 
good as it gets from the user space?
3. Is there a way to address this problem, or is it completely dependent upon 
the hardware?


Thanks for any insights to the problem, and for all the advice everyone has 
submitted this far.







On 3/3/16, 1:59 PM, "Richard Hacker"  wrote:

>Hi,
>
>we use CLOCK_MONOTONIC for ecrt_master_application_time(), in fact we 
>use it thoughout. Probably not the solution, but worth a try.
>
>- Richard
>
>On 03.03.2016 17:08, Thomas Bitsky Jr wrote:
>>
>> [snip]
>> Are you using the dc_clock_adjust to modify your sleep time?  ie.
>> something like
>> [/snip]
>>
>>
>> I’ve been trying, but the results are all over the place and I’m obviously 
>> not understanding what needs to be done with the clock.
>>
>> My latest attempt is has the scan time ticking from anywhere between 3930 
>> and 4100 scans per second (reported by ethercat master command) and does 
>> nothing to eliminate skipped datagrams. I don’t think I’m fundamentally 
>> understanding what needs to be done.
>>
>> My latest code is below. If anyone has any insights, it would be greatly 
>> appreciated.
>>
>> 
>>
>> int64_t  dc_adjust_ns = 0 ;
>> int32_t  dc_diff_ns = 0;
>>
>>
>>
>> void calculateClockDrift(void) {
>>  uint32_t ref_time = 0;
>>  uint64_t prev_app_time = dc_time_ns;
>>
>>  dc_time_ns = system_time_ns();
>>  // get reference clock time to synchronize master cycle
>>  ecrt_master_reference_clock_time(master_, & ref_time);
>>  dc_diff_ns = (uint32_t) prev_app_time - ref_time;
>>  // calc drift (via un-normalised time diff)
>>  int32_t delta = dc_diff_ns - prev_dc_diff_ns;
>>  prev_dc_diff_ns = dc_diff_ns;
>>
>>
>>  dc_diff_ns =
>>  ((dc_diff_ns + (cycle_ns_ / 2)) % cycle_ns_) - (cycle_ns_ / 2);
>>
>> }
>>
>>
>> #define TIMESPEC_ADD_NS(TS, NS)\
>>  (TS).tv_nsec += (NS);\
>> while ((TS).tv_nsec >= NANOS_PER_SEC) {\
>>  (TS).tv_nsec -= NANOS_PER_SEC;\
>>  (TS).tv_sec++;
>> }
>>
>> #define TIMESPEC2NSEPOCH2000(T)\
>>  

Re: [etherlab-users] Distributed Clocks

[Thomas Bitsky Jr]

I have to get a machine out the door, but that’s totally worth a try and I’ll 
give it a test tomorrow.

Thanks!







On 3/8/16, 4:29 PM, "etherlab-users on behalf of Gavin Lambert" 
 wrote:

>[Repost to list due to incorrect email address.]
>
>On 9 March 2016 05:09, quoth Thomas Bitsky Jr:
>> I have tried this on two pieces of hardware. They run the same 
>> operating system and configuration, same EtherLAB master version, same 
>> Ethernet driver.
>> 
>> System 1 (Intel i3 NUC, 1 Intel e1000e Ethernet port): exec min 6789 
>> .. max
>> 50345  (nano seconds)
>> System 2 (Intel quad celeoron, 4 x e1000e Ethernet port): exec min 
>> 15588 … max 323376 (nano seconds) (Both systems: Ubuntu Server 
>> 14.04LTS, Linux core 3.12.50-rt68 #1 SMP PREEMPT RT)
>> 
>> On system 1, the unmatched packets never happen; for whatever reason, 
>> those functions are returning much more rapidly. On the second system, 
>> the worst case happens about once per minute, once every two minutes.
>> 
>> 1. Am I unreasonable in thinking these times are excessive? I 
>> understand that the system is communicating out to external hardware 
>> and there will be a delay, but the difference between the two computers is 
>> surprising to me.
>> 2. Is this a consequence of using the EtherCAT master from user space? 
>> I’ve always used it from kernel space before. However, I’ve never done 
>> as extensive of timing measurements before. The EtherLAB documentation 
>> mentioned that the userspace library only added 1 microsecond of 
>> overhead, if I recall. Is this as good as it gets from the user space?
>> 3. Is there a way to address this problem, or is it completely 
>> dependent upon the hardware?
>
>I'm not sure if it will help in this case (the issue was discovered when 
>testing something else), but it might be worthwhile trying this patch:
>
>http://lists.etherlab.org/pipermail/etherlab-dev/2015/000475.html
>
>There's some fairly slow watchdog code in the e1000e driver that can execute 
>synchronously inside ecrt_master_receive.  Normally the watchdog is blocked 
>entirely when the port is receiving packets, but since one of your systems has 
>several ports (and presumably not all of them are in use) it's possible that 
>you're being affected by this.  The patch pushes the watchdog back onto a 
>separate worker thread (although it also allows it to run even when receiving 
>packets, for other reasons).
>
>As I said, not sure if it will help here, but seems worth a try at least. :)
>
>
>___
>etherlab-users mailing list
>etherlab-users@etherlab.org
>http://lists.etherlab.org/mailman/listinfo/etherlab-users
___
etherlab-users mailing list
etherlab-users@etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-users

Re: [etherlab-users] Distributed Clocks

[Gavin Lambert]

[Repost to list due to incorrect email address.]

On 9 March 2016 05:09, quoth Thomas Bitsky Jr:
> I have tried this on two pieces of hardware. They run the same 
> operating system and configuration, same EtherLAB master version, same 
> Ethernet driver.
> 
> System 1 (Intel i3 NUC, 1 Intel e1000e Ethernet port): exec min 6789 
> .. max
> 50345  (nano seconds)
> System 2 (Intel quad celeoron, 4 x e1000e Ethernet port): exec min 
> 15588 … max 323376 (nano seconds) (Both systems: Ubuntu Server 
> 14.04LTS, Linux core 3.12.50-rt68 #1 SMP PREEMPT RT)
> 
> On system 1, the unmatched packets never happen; for whatever reason, 
> those functions are returning much more rapidly. On the second system, 
> the worst case happens about once per minute, once every two minutes.
> 
> 1. Am I unreasonable in thinking these times are excessive? I 
> understand that the system is communicating out to external hardware 
> and there will be a delay, but the difference between the two computers is 
> surprising to me.
> 2. Is this a consequence of using the EtherCAT master from user space? 
> I’ve always used it from kernel space before. However, I’ve never done 
> as extensive of timing measurements before. The EtherLAB documentation 
> mentioned that the userspace library only added 1 microsecond of 
> overhead, if I recall. Is this as good as it gets from the user space?
> 3. Is there a way to address this problem, or is it completely 
> dependent upon the hardware?

I'm not sure if it will help in this case (the issue was discovered when 
testing something else), but it might be worthwhile trying this patch:

http://lists.etherlab.org/pipermail/etherlab-dev/2015/000475.html

There's some fairly slow watchdog code in the e1000e driver that can execute 
synchronously inside ecrt_master_receive.  Normally the watchdog is blocked 
entirely when the port is receiving packets, but since one of your systems has 
several ports (and presumably not all of them are in use) it's possible that 
you're being affected by this.  The patch pushes the watchdog back onto a 
separate worker thread (although it also allows it to run even when receiving 
packets, for other reasons).

As I said, not sure if it will help here, but seems worth a try at least. :)


___
etherlab-users mailing list
etherlab-users@etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-users

Re: [etherlab-users] Distributed Clocks

[Thomas Bitsky Jr]

>> clock_nanosleep(CLOCK_MONOTONIC,TIMER_ABSTIME,>wakeupTime,NULL);
>> TIMESPEC_ADD_NS(inst->wakeupTime,100 + dc_clock_adjust);

>> Does your inst->cycleNs include the clock adjustement?

No, I’m not adjusting that clock based on the DC. I can give that a try.

Thanks!
Thomas C. Bitsky Jr. | Lead Developer
ADC | automateddesign.com 
P: 630-783-1150 F: 630-783-1159 M: 630-632-6679

Follow ADC news and media:
Facebook  | Twitter 
 | YouTube 









On 3/2/16, 10:04 AM, "Henry Bausley"  wrote:

>
>Are you using the dc_clock_adjust to modify your sleep time?  ie.
>something like  
>  
>clock_nanosleep(CLOCK_MONOTONIC,TIMER_ABSTIME,>wakeupTime,NULL);
>TIMESPEC_ADD_NS(inst->wakeupTime,100 + dc_clock_adjust);
>
>Does your inst->cycleNs include the clock adjustement?
>
>I have found the best debug tool for DC is an oscilloscope and a board
>that you probe the SOF and sync0 .  If you see SOF wander with respect
>to sync0 it won't work correctly.  If they don't you are good.
>
>On Wed, 2016-03-02 at 14:47 +, Thomas Bitsky Jr wrote:
>> Thanks for the response, Richard.
>> 
>> CyclicTest on the hardware:
>> user@core:~$ sudo cyclictest --smp -p95 -m
>> # /dev/cpu_dma_latency set to 0us
>> policy: fifo: loadavg: 0.01 0.04 0.04 1/162 1547
>> 
>> T: 0 ( 1543) P:95 I:1000 C:  26093 Min:  6 Act:   30 Avg:   25 Max:  
>> 56
>> T: 1 ( 1544) P:95 I:1500 C:  17386 Min:  6 Act:   14 Avg:   23 Max:  
>> 66
>> T: 2 ( 1545) P:95 I:2000 C:  13036 Min:  5 Act:   28 Avg:   29 Max:  
>> 57
>> T: 3 ( 1546) P:95 I:2500 C:  10424 Min:  6 Act:   30 Avg:   29 Max:  
>> 58
>> 
>> 
>> 
>> So, the max latency is definitely in the ballpark of what I was saying last 
>> night.
>> 
>> Setting cylictest with a break value fails almost instantly.
>> 
>> root@core:/sys/kernel/debug/tracing# cyclictest --smp -p95 -f -b 200
>> # /dev/cpu_dma_latency set to 0us
>> INFO: debugfs mountpoint: /sys/kernel/debug/tracing/
>> policy: fifo: loadavg: 0.00 0.01 0.05 3/165 1703
>> 
>> T: 0 ( 1700) P:95 I:1000 C:  0 Min:100 Act:0 Avg:0 Max:  
>>  0
>> T: 1 ( 1701) P:95 I:1500 C:  0 Min:100 Act:0 Avg:0 Max:  
>>  0
>> T: 2 (0) P:95 I:2000 C:  0 Min:100 Act:0 Avg:0 Max:  
>>  0
>> T: 3 ( 1703) P:95 I:2500 C:  0 Min:100 Act:0 Avg:0 Max:  
>>  0
>> # Thread Ids: 01700 01701 01702 01703
>> # Break thread: 1702
>> # Break value: 217
>> 
>> 
>> 
>> [snip]
>> First things first: when you are running RT proggies, you need a stable 
>> clock source and a preemptable kernel!
>> [/snip]
>> 
>> 
>> I am using RT Preempt:
>> # uname -a
>> Linux core 3.12.50-rt68 #1 SMP PREEMPT RT Mon Nov 23 18:17:14 CST 2015 i686 
>> i686 i686 GNU/Linux
>> 
>> 
>> Thanks!
>> Thomas C. Bitsky Jr. | Lead Developer
>> ADC | automateddesign.com 
>> P: 630-783-1150 F: 630-783-1159 M: 630-632-6679
>> 
>> Follow ADC news and media:
>> Facebook  | Twitter 
>>  | YouTube 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> On 3/2/16, 5:30 AM, "etherlab-users on behalf of Richard Hacker" 
>>  wrote:
>> 
>> >Hi John,
>> >
>> >Skipped frames are either a timing or a hardware problem (faulty 
>> >EtherCAT slaves, cabling, noise). Some slaves don't care, DC slaves puke!
>> >
>> >First things first: when you are running RT proggies, you need a stable 
>> >clock source and a preemptable kernel! Even more so when you're running 
>> >at 250us cycle time. Your jitter of 50us is roughly 20% of your cycle 
>> >time, definitely not a neglegible fraction! So tackle that first.
>> >
>> >Download cyclictest and test that first:
>> >https://rt.wiki.kernel.org/index.php/Cyclictest
>> >
>> >Once your cyclic test is stable (~10us max jitter), you may start 
>> >looking for delays in your code.
>> >
>> >Instrumate your loop and find out what is causing delays. Comment out 
>> >large sections of code to start from a known good state (maybe doing 
>> >EtherCAT IO exclusively and NOTHING else, no GUI, signal processing, 
>> >etc) and reinsert code slowly to find the error.
>> >
>> >Do you have ANY system calls in your RT cycle, like sockets(), read() or 
>> >write()? Do remember that system calls (including those to the master) 
>> >may cause page faults that introduce delays. Only a very small subset of 
>> >known and documented system calls can be used for RT. Notably read() and 
>> >write() to pipes, shared memory (once set up of coarse), semaphores are OK.
>> >
>> >BTW: Needless to say that we have paid particular attention that the 
>> >master does not cause any page faults or other delays once started in 
>> 

Re: [etherlab-users] Distributed Clocks

[Thomas Bitsky Jr]

I’m not very familiar with FTRACE. Is this log file telling me that the mutex 
and spin lock calls are the problem? My understanding is that the “+” indicates 
calls that took exceedingly long.

cyclicte-15722... 64066us+: _mutex_unlock <-rb_simple_write
cyclicte-15722... 64159us+: rt_mutex_unlock <-__do_page_fault
cyclicte-1572211. 64170us+: sub_preempt_count <-migrate_enable
cyclicte-15722... 64190us+: __copy_to_user_ll <-SyS_rt_sigaction
cyclicte-15722... 64476us+: put_pid <-do_fork
  -0   1dN..2.. 64566us+: update_stats_wait_end <-set_next_entity
cyclicte-15733.11 64689us+: rt_spin_lock
migratio-19  1... 65201us+: rt_mutex_lock <-cpu_stopper_thread

Thomas C. Bitsky Jr. | Lead Developer
ADC | automateddesign.com 
P: 630-783-1150 F: 630-783-1159 M: 630-632-6679

Follow ADC news and media:
Facebook  | Twitter 
 | YouTube 










On 3/2/16, 8:47 AM, "Thomas Bitsky Jr"  wrote:

>Thanks for the response, Richard.
>
>CyclicTest on the hardware:
>user@core:~$ sudo cyclictest --smp -p95 -m
># /dev/cpu_dma_latency set to 0us
>policy: fifo: loadavg: 0.01 0.04 0.04 1/162 1547
>
>T: 0 ( 1543) P:95 I:1000 C:  26093 Min:  6 Act:   30 Avg:   25 Max:  56
>T: 1 ( 1544) P:95 I:1500 C:  17386 Min:  6 Act:   14 Avg:   23 Max:  66
>T: 2 ( 1545) P:95 I:2000 C:  13036 Min:  5 Act:   28 Avg:   29 Max:  57
>T: 3 ( 1546) P:95 I:2500 C:  10424 Min:  6 Act:   30 Avg:   29 Max:  58
>
>
>
>So, the max latency is definitely in the ballpark of what I was saying last 
>night.
>
>Setting cylictest with a break value fails almost instantly.
>
>root@core:/sys/kernel/debug/tracing# cyclictest --smp -p95 -f -b 200
># /dev/cpu_dma_latency set to 0us
>INFO: debugfs mountpoint: /sys/kernel/debug/tracing/
>policy: fifo: loadavg: 0.00 0.01 0.05 3/165 1703
>
>T: 0 ( 1700) P:95 I:1000 C:  0 Min:100 Act:0 Avg:0 Max:   0
>T: 1 ( 1701) P:95 I:1500 C:  0 Min:100 Act:0 Avg:0 Max:   0
>T: 2 (0) P:95 I:2000 C:  0 Min:100 Act:0 Avg:0 Max:   0
>T: 3 ( 1703) P:95 I:2500 C:  0 Min:100 Act:0 Avg:0 Max:   0
># Thread Ids: 01700 01701 01702 01703
># Break thread: 1702
># Break value: 217
>
>
>
>[snip]
>First things first: when you are running RT proggies, you need a stable 
>clock source and a preemptable kernel!
>[/snip]
>
>
>I am using RT Preempt:
># uname -a
>Linux core 3.12.50-rt68 #1 SMP PREEMPT RT Mon Nov 23 18:17:14 CST 2015 i686 
>i686 i686 GNU/Linux
>
>
>Thanks!
>Thomas C. Bitsky Jr. | Lead Developer
>ADC | automateddesign.com 
>P: 630-783-1150 F: 630-783-1159 M: 630-632-6679
>
>Follow ADC news and media:
>Facebook  | Twitter 
> | YouTube 
>
>
>
>
>
>
>
>
>
>On 3/2/16, 5:30 AM, "etherlab-users on behalf of Richard Hacker" 
> wrote:
>
>>Hi John,
>>
>>Skipped frames are either a timing or a hardware problem (faulty 
>>EtherCAT slaves, cabling, noise). Some slaves don't care, DC slaves puke!
>>
>>First things first: when you are running RT proggies, you need a stable 
>>clock source and a preemptable kernel! Even more so when you're running 
>>at 250us cycle time. Your jitter of 50us is roughly 20% of your cycle 
>>time, definitely not a neglegible fraction! So tackle that first.
>>
>>Download cyclictest and test that first:
>>https://rt.wiki.kernel.org/index.php/Cyclictest
>>
>>Once your cyclic test is stable (~10us max jitter), you may start 
>>looking for delays in your code.
>>
>>Instrumate your loop and find out what is causing delays. Comment out 
>>large sections of code to start from a known good state (maybe doing 
>>EtherCAT IO exclusively and NOTHING else, no GUI, signal processing, 
>>etc) and reinsert code slowly to find the error.
>>
>>Do you have ANY system calls in your RT cycle, like sockets(), read() or 
>>write()? Do remember that system calls (including those to the master) 
>>may cause page faults that introduce delays. Only a very small subset of 
>>known and documented system calls can be used for RT. Notably read() and 
>>write() to pipes, shared memory (once set up of coarse), semaphores are OK.
>>
>>BTW: Needless to say that we have paid particular attention that the 
>>master does not cause any page faults or other delays once started in 
>>cyclic mode ;)
>>
>>Good luck!
>>Richard
>>
>>On 02.03.2016 05:51, Thomas Bitsky Jr wrote:
>>>
>>> I’ve been continuing to work on this, but have had limited success.
>>> While a distributed clock is technically running, it’s caused a few
>>> other problems that I’ve been unable to correct. I think my main problem
>>> all stems 

Re: [etherlab-users] Distributed Clocks

[Richard Hacker]

Hi John,

Skipped frames are either a timing or a hardware problem (faulty 
EtherCAT slaves, cabling, noise). Some slaves don't care, DC slaves puke!


First things first: when you are running RT proggies, you need a stable 
clock source and a preemptable kernel! Even more so when you're running 
at 250us cycle time. Your jitter of 50us is roughly 20% of your cycle 
time, definitely not a neglegible fraction! So tackle that first.


Download cyclictest and test that first:
https://rt.wiki.kernel.org/index.php/Cyclictest

Once your cyclic test is stable (~10us max jitter), you may start 
looking for delays in your code.


Instrumate your loop and find out what is causing delays. Comment out 
large sections of code to start from a known good state (maybe doing 
EtherCAT IO exclusively and NOTHING else, no GUI, signal processing, 
etc) and reinsert code slowly to find the error.


Do you have ANY system calls in your RT cycle, like sockets(), read() or 
write()? Do remember that system calls (including those to the master) 
may cause page faults that introduce delays. Only a very small subset of 
known and documented system calls can be used for RT. Notably read() and 
write() to pipes, shared memory (once set up of coarse), semaphores are OK.


BTW: Needless to say that we have paid particular attention that the 
master does not cause any page faults or other delays once started in 
cyclic mode ;)


Good luck!
Richard

On 02.03.2016 05:51, Thomas Bitsky Jr wrote:


I’ve been continuing to work on this, but have had limited success.
While a distributed clock is technically running, it’s caused a few
other problems that I’ve been unable to correct. I think my main problem
all stems from this:

[36524.681778] EtherCAT 0: Domain 0: Working counter changed to 10/10.
[36524.681787] EtherCAT 0: Domain 2: Working counter changed to 1/1.
[36524.681792] EtherCAT 0: Domain 1: Working counter changed to 1/1.
[36525.858760] EtherCAT 0: Domain 0: Working counter changed to 0/10.
[36525.858810] EtherCAT 0: Domain 2: Working counter changed to 0/1.
[36525.858827] EtherCAT 0: Domain 1: Working counter changed to 0/1.
[36526.203067] EtherCAT WARNING: Datagram f185d88c (domain0-0-main) was
SKIPPED 2 times.
[36526.203099] EtherCAT WARNING: Datagram f185d90c (domain2-28-main) was
SKIPPED 2 times.
[36526.203104] EtherCAT WARNING: Datagram f185d28c (domain1-22-main) was
SKIPPED 2 times.
[36526.743379] EtherCAT WARNING 0: 12 datagrams UNMATCHED!
[36526.863556] EtherCAT 0: Domain 0: 5 working counter changes - now 10/10.
[36526.863566] EtherCAT 0: Domain 2: 5 working counter changes - now 1/1.
[36526.863572] EtherCAT 0: Domain 1: 5 working counter changes - now 1/1.

… and on and on and on…

The AKD servo drive I’m using will run fine, no warnings, then suddenly
drop to an F125 fault and shut off. The is a frame synchronization
error. Basically, it’s saying that the sync0 frame isn’t received at a
consistent enough rate, so it faults out.

My scan rate is 250 microseconds, and I admit there is jitter. It varies
from as little as +/- 50 microseconds, though I’m not sure why. The
"ethercat master" command reports a steady 4000 frames/sec, but the
scoping part of my project records a timestamp, and I am seeing the +/-
50 microseconds.

My timing function is straight out of the EtherCAT master examples and
is also similar to methods I’ve seen in other cyclic task projects. The
whole cyclic task is below. Can anyone see what idiotic thing I must be
doing to get unmatched datagrams?

#define TIMESPEC_ADD_NS(TS, NS)\
(TS).tv_nsec += (NS);\
while ( (TS).tv_nsec >= NANOS_PER_SEC ){\
(TS).tv_nsec -= NANOS_PER_SEC;\
(TS).tv_sec++; }

#define TIMESPEC2NSEPOCH2000(T) \
((uint64_t) (((T).tv_sec - 946684800ULL) * 10ULL) + (T).tv_nsec)

#define TON struct timespec
#define TON_ENDTIME(MS)\
time_add_ns((MS) * NANOS_PER_MILLISEC)


static TON clockSyncTon_;


int
TON_ISDONE( struct timespec ts )
{
struct timespec now;
clock_gettime(CLOCK_MONOTONIC, );
if ( now.tv_sec > ts.tv_sec )
return 1;
else if ( now.tv_sec == ts.tv_sec
&& now.tv_nsec >= ts.tv_nsec )
return 1;
else
return 0;
}


static bool
wait_period(RtaiMain* inst)
{

int rc;
bool done = false;
while ( !done && inst->doScan && runAll_ )
{
rc = clock_nanosleep(CLOCK_MONOTONIC,
TIMER_ABSTIME,
>wakeupTime,
NULL );


if ( rc == EFAULT )
{
return false;
}
else if ( rc == EINTR )
{
continue;
}
else if ( rc == EINVAL )
{
return false;
}
else
{
done = 1;
}
}
TIMESPEC_ADD_NS(inst->wakeupTime, inst->cycleNs);
return true;

}


static void
cyclic_task(RtaiMain* inst)
{

clock_gettime(CLOCK_MONOTONIC ,&(inst->wakeupTime));
/* start after one second */
inst->wakeupTime.tv_sec++;
wait_period(inst);
while (runAll_ && inst->doScan)
{
//
// Trigger Fieldbus RX here.
//
//
ecrt_master_receive(master_);

  // record the time we received the data so other parts of the program
  // have an accurate time reading
  globalTickTimeNs = ton_get_ns();

  ecrt_domain_process(lrwDomainMgr_.domain);
  

Re: [etherlab-users] distributed clocks query

[Graeme Foot]

Hi,

I have never been able to get the EL7031 and EL7041 stepper modules to work in 
DC sync mode.  I’ve also never really had enough time to really get in and 
figure out why.  If you do and succeed then it would be appreciated if you 
could post your results.  You could perhaps try to get it working under a 
TwinCAT system then compare the differences.


Some more notes on DC:
- During your app setup, the first call to ecrt_master_application_time() 
defines the “time when the clocks start off for the first time”.  Your realtime 
cycle must then be relative to this time once you go realtime.
- There are two parts to DC.  The first is the synchronization of the slaves 
clocks to the reference slaves clock (and the master clock).  The second is 
choosing which slaves use the dc sync events by calling ecrt_slave_config_dc() 
on every slave you wish to be sync’ed, but this can only be called on slaves 
that support DC to this level.


Regards,
Graeme.

From: etherlab-users [mailto:etherlab-users-boun...@etherlab.org] On Behalf Of 
Paul Mulligan
Sent: Wednesday, 2 March 2016 5:27 a.m.
To: Richard Hacker; etherlab-users@etherlab.org
Subject: Re: [etherlab-users] distributed clocks query

Thanks.

It seems the however that the EL7031 stepper motor module does not work in DC 
mode even tho the AssignActivate word is listed in the device description xml 
file as 0x300.

On Tuesday, 1 March 2016, 17:04:39, Richard Hacker 
<h...@igh.de<mailto:h...@igh.de>> wrote:

DC is quite an intricate aspect of EtherCAT.

In DC enabled slaves, there are two clocks that generate events. You
specify the time at which these clocks run as well as the time when the
clocks start off for the first time. What the slave controller does at
these events is implementation detail and must be documented by the
manufacturer of the slave.

There is documentation available at
http://download.beckhoff.com/download/document/io/ethercat-terminals/ethercatsystem_en.chm
Check out Distributed Clocks -> Slave synchronization -> DC Modes.

In the master, it is configured using
void ecrt_slave_config_dc(
ec_slave_config_t *sc, /**< Slave configuration.*/
uint16_t assign_activate, /**< AssignActivate word.*/
uint32_t sync0_cycle, /**< SYNC0 cycle time [ns].*/
int32_t sync0_shift, /**< SYNC0 shift time [ns].*/
uint32_t sync1_cycle, /**< SYNC1 cycle time [ns].*/
int32_t sync1_shift /**< SYNC1 shift time [ns].*/
);

You will be stetting sync0_cycle to your cycle time in nanoseconds and
play around with sync0_shift somewhere half of your cycle time. The
other values will be zero in your case (in fact, sync1_shift is not even
used in the master!)

To check whether your slave is synchronizing, run:

watch -n0 "ethercat reg_read -pX -tsm32 0x92c"

where X is the position of your slave. This value should be quite low,
in the order of a few hundred. This is a correction value in nanoseconds
of the slave's clock.

On 01.03.2016 12:44, Paul Mulligan wrote:
> Hi,
>
> I'm still unsure as to what values are given for the last four
> parameters of this function. I believe the sync1_cycle and sync1_shift
> can be ignored but how is the sync0_shift value determined? I believe
> the frame should reach all the slaves before this time but also the new
> frame should not be received before this time. In the examples given, a
> scan time of 1ms is used but the sync0_shift value is 4.4 ms. How can
> this be correct?
>
> ecrt_slave_config_dc(modules[index].sc, 0x0300, scanTime, ??, ??, ??, ??);
>
>
> On Monday, 29 February 2016, 15:57:30, Richard Hacker 
> <h...@igh.de<mailto:h...@igh.de>> wrote:
>
>
> I only configure DC on the slaves that are actually required to be
> synchronized.
>
> Otherwise it seems correct what you are doing...
>
> Am 2016-02-29 um 14:45 schrieb Paul Mulligan:
>  > Hi,
>  >
>  > Just a question or two about distributed clocks.
>  >
>  > I have a system with an EL1008 digital input module, EL3001 analogue
>  > input module, two EL7031 stepper motor driver modules and two EL2008
>  > digital output modules in that order. I am using the EK1100 bus coupler
>  > terminal as the first module.
>  >
>  > Do I need to call ecrt_slave_config_dc() for all of these modules before
>  > activating the master, or just the first module on the bus? My
>  > understanding from reading about distributed clocks is that the first
>  > module on the bus with DC capability should be used as the reference
> clock.
>  >
>  >  From looking at the example "dc_user" supplied in the master download,
>  > it calls ecrt_slave_config_dc() only for the IDS_COUNTER module.
>  >
>  > In the cyclic_task(), the functions ecrt_master_application_time(),
>  > ecrt_master_sync_referen

Re: [etherlab-users] distributed clocks query

[Paul Mulligan]

Thanks.
It seems the however that the EL7031 stepper motor module does not work in DC 
mode even tho the AssignActivate word is listed in the device description xml 
file as 0x300.  

On Tuesday, 1 March 2016, 17:04:39, Richard Hacker  wrote:
 

 DC is quite an intricate aspect of EtherCAT.

In DC enabled slaves, there are two clocks that generate events. You 
specify the time at which these clocks run as well as the time when the 
clocks start off for the first time. What the slave controller does at 
these events is implementation detail and must be documented by the 
manufacturer of the slave.

There is documentation available at
http://download.beckhoff.com/download/document/io/ethercat-terminals/ethercatsystem_en.chm
Check out Distributed Clocks -> Slave synchronization -> DC Modes.

In the master, it is configured using
void ecrt_slave_config_dc(
    ec_slave_config_t *sc, /**< Slave configuration.*/
    uint16_t assign_activate, /**< AssignActivate word.*/
    uint32_t sync0_cycle, /**< SYNC0 cycle time [ns].*/
    int32_t sync0_shift, /**< SYNC0 shift time [ns].*/
    uint32_t sync1_cycle, /**< SYNC1 cycle time [ns].*/
    int32_t sync1_shift /**< SYNC1 shift time [ns].*/
        );

You will be stetting sync0_cycle to your cycle time in nanoseconds and 
play around with sync0_shift somewhere half of your cycle time. The 
other values will be zero in your case (in fact, sync1_shift is not even 
used in the master!)

To check whether your slave is synchronizing, run:

watch -n0 "ethercat reg_read -pX -tsm32 0x92c"

where X is the position of your slave. This value should be quite low, 
in the order of a few hundred. This is a correction value in nanoseconds 
of the slave's clock.

On 01.03.2016 12:44, Paul Mulligan wrote:
> Hi,
>
> I'm still unsure as to what values are given for the last four
> parameters of this function. I believe the sync1_cycle and sync1_shift
> can be ignored but how is the sync0_shift value determined? I believe
> the frame should reach all the slaves before this time but also the new
> frame should not be received before this time. In the examples given, a
> scan time of 1ms is used but the sync0_shift value is 4.4 ms. How can
> this be correct?
>
> ecrt_slave_config_dc(modules[index].sc, 0x0300, scanTime, ??, ??, ??, ??);
>
>
> On Monday, 29 February 2016, 15:57:30, Richard Hacker  wrote:
>
>
> I only configure DC on the slaves that are actually required to be
> synchronized.
>
> Otherwise it seems correct what you are doing...
>
> Am 2016-02-29 um 14:45 schrieb Paul Mulligan:
>  > Hi,
>  >
>  > Just a question or two about distributed clocks.
>  >
>  > I have a system with an EL1008 digital input module, EL3001 analogue
>  > input module, two EL7031 stepper motor driver modules and two EL2008
>  > digital output modules in that order. I am using the EK1100 bus coupler
>  > terminal as the first module.
>  >
>  > Do I need to call ecrt_slave_config_dc() for all of these modules before
>  > activating the master, or just the first module on the bus? My
>  > understanding from reading about distributed clocks is that the first
>  > module on the bus with DC capability should be used as the reference
> clock.
>  >
>  >  From looking at the example "dc_user" supplied in the master download,
>  > it calls ecrt_slave_config_dc() only for the IDS_COUNTER module.
>  >
>  > In the cyclic_task(), the functions ecrt_master_application_time(),
>  > ecrt_master_sync_reference_clock(), ecrt_master_sync_slave_clocks() are
>  > then called in that order. I notice ecrt_master_reference_clock_time()
>  > is not used at all here.
>  >
>  > Is this all that is required to control the distributed clocks ? Thank
>  > you in advance.
>  >
>  >
>  > ___
>  > etherlab-users mailing list
>  > etherlab-users@etherlab.org 
>  > http://lists.etherlab.org/mailman/listinfo/etherlab-users
>  >
>
>


  ___
etherlab-users mailing list
etherlab-users@etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-users

Re: [etherlab-users] distributed clocks query

[Richard Hacker]

DC is quite an intricate aspect of EtherCAT.

In DC enabled slaves, there are two clocks that generate events. You 
specify the time at which these clocks run as well as the time when the 
clocks start off for the first time. What the slave controller does at 
these events is implementation detail and must be documented by the 
manufacturer of the slave.


There is documentation available at
http://download.beckhoff.com/download/document/io/ethercat-terminals/ethercatsystem_en.chm
Check out Distributed Clocks -> Slave synchronization -> DC Modes.

In the master, it is configured using
void ecrt_slave_config_dc(
ec_slave_config_t *sc, /**< Slave configuration.*/
uint16_t assign_activate, /**< AssignActivate word.*/
uint32_t sync0_cycle, /**< SYNC0 cycle time [ns].*/
int32_t sync0_shift, /**< SYNC0 shift time [ns].*/
uint32_t sync1_cycle, /**< SYNC1 cycle time [ns].*/
int32_t sync1_shift /**< SYNC1 shift time [ns].*/
);

You will be stetting sync0_cycle to your cycle time in nanoseconds and 
play around with sync0_shift somewhere half of your cycle time. The 
other values will be zero in your case (in fact, sync1_shift is not even 
used in the master!)


To check whether your slave is synchronizing, run:

watch -n0 "ethercat reg_read -pX -tsm32 0x92c"

where X is the position of your slave. This value should be quite low, 
in the order of a few hundred. This is a correction value in nanoseconds 
of the slave's clock.


On 01.03.2016 12:44, Paul Mulligan wrote:

Hi,

I'm still unsure as to what values are given for the last four
parameters of this function. I believe the sync1_cycle and sync1_shift
can be ignored but how is the sync0_shift value determined? I believe
the frame should reach all the slaves before this time but also the new
frame should not be received before this time. In the examples given, a
scan time of 1ms is used but the sync0_shift value is 4.4 ms. How can
this be correct?

ecrt_slave_config_dc(modules[index].sc, 0x0300, scanTime, ??, ??, ??, ??);


On Monday, 29 February 2016, 15:57:30, Richard Hacker  wrote:


I only configure DC on the slaves that are actually required to be
synchronized.

Otherwise it seems correct what you are doing...

Am 2016-02-29 um 14:45 schrieb Paul Mulligan:
 > Hi,
 >
 > Just a question or two about distributed clocks.
 >
 > I have a system with an EL1008 digital input module, EL3001 analogue
 > input module, two EL7031 stepper motor driver modules and two EL2008
 > digital output modules in that order. I am using the EK1100 bus coupler
 > terminal as the first module.
 >
 > Do I need to call ecrt_slave_config_dc() for all of these modules before
 > activating the master, or just the first module on the bus? My
 > understanding from reading about distributed clocks is that the first
 > module on the bus with DC capability should be used as the reference
clock.
 >
 >  From looking at the example "dc_user" supplied in the master download,
 > it calls ecrt_slave_config_dc() only for the IDS_COUNTER module.
 >
 > In the cyclic_task(), the functions ecrt_master_application_time(),
 > ecrt_master_sync_reference_clock(), ecrt_master_sync_slave_clocks() are
 > then called in that order. I notice ecrt_master_reference_clock_time()
 > is not used at all here.
 >
 > Is this all that is required to control the distributed clocks ? Thank
 > you in advance.
 >
 >
 > ___
 > etherlab-users mailing list
 > etherlab-users@etherlab.org 
 > http://lists.etherlab.org/mailman/listinfo/etherlab-users
 >



___
etherlab-users mailing list
etherlab-users@etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-users

Re: [etherlab-users] distributed clocks query

[Richard Hacker]

I only configure DC on the slaves that are actually required to be 
synchronized.


Otherwise it seems correct what you are doing...

Am 2016-02-29 um 14:45 schrieb Paul Mulligan:

Hi,

Just a question or two about distributed clocks.

I have a system with an EL1008 digital input module, EL3001 analogue
input module, two EL7031 stepper motor driver modules and two EL2008
digital output modules in that order. I am using the EK1100 bus coupler
terminal as the first module.

Do I need to call ecrt_slave_config_dc() for all of these modules before
activating the master, or just the first module on the bus? My
understanding from reading about distributed clocks is that the first
module on the bus with DC capability should be used as the reference clock.

 From looking at the example "dc_user" supplied in the master download,
it calls ecrt_slave_config_dc() only for the IDS_COUNTER module.

In the cyclic_task(), the functions ecrt_master_application_time(),
ecrt_master_sync_reference_clock(), ecrt_master_sync_slave_clocks() are
then called in that order. I notice ecrt_master_reference_clock_time()
is not used at all here.

Is this all that is required to control the distributed clocks ? Thank
you in advance.


___
etherlab-users mailing list
etherlab-users@etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-users


___
etherlab-users mailing list
etherlab-users@etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-users

Re: [etherlab-users] Distributed Clocks

[Richard Hacker]

I do not have the patience to look into your code in detail to find out 
what you're doing wrong, but here is an excerpt of my code I use in 
etherlab Simulink Coder (ecrt_support.c):


void
ecs_send()
{
//...

list_for_each(master, _data.master_list, list) {
list_for_each(domain, >domain_list, list) {
// ...

ecrt_domain_queue(domain->handle);
}

if (/* send frames */) {
struct timespec tp;

clock_gettime(CLOCK_MONOTONIC, );
ecrt_master_application_time(master->handle,
EC_TIMESPEC2NANO(tp));

if (/* sync_clocks */)) {
ecrt_master_sync_reference_clock(master->handle);
//...
}

ecrt_master_sync_slave_clocks(master->handle);
ecrt_master_send(master->handle);
}
}
}

void
ecs_receive()
{
//...

list_for_each(master, _data.master_list, list) {

if (/* receive frames */) {
ecrt_master_receive(master->handle);
}

list_for_each(domain, >domain_list, list) {

ecrt_domain_process(domain->handle);
ecrt_domain_state(domain->handle, >state);

}
}
}

void
calculate()
{
while (forever) {
ecs_receive();
// do calculations
ecs_send();
sleep();
}
}

Some of this is pseudo code, but I trust in your programming 
capabilities to be able to adapt this to your code ;)


It is very important to call clock_gettime() and 
ecrt_master_application_time() as close to ecrt_master_send() as possible.


As you can see, it is not necessary to call 
ecrt_master_sync_reference_clock() with every ecrt_master_send(). In 
fact, ~100Hz of syncing should be enough, provided that your call to 
clock_gettime() is accurate. On the other hand, if clock_gettime() is 
inaccurate, you're in trouble when using DC anyway!


- Richard

Am 2016-02-22 um 04:23 schrieb Thomas Bitsky Jr:

Ok, after following all your advice, plus the clock adjustment from
Mattieu, I’m pretty close. I enable the DC on the drive, get everything
into op, the domains all come up and it would appear that the clocks are
syncing. Except for one thing:

[10248.633619] EtherCAT 0: Domain 0: Working counter changed to 1/17.
[10249.535776] EtherCAT WARNING 0: 3989 datagrams UNMATCHED!
[10249.539861] EtherCAT WARNING: Datagram f3bf2d0c (domain0-0-main) was
SKIPPED 1 time.
[10249.539887] EtherCAT WARNING: Datagram f3bf2a8c (domain2-120-main)
was SKIPPED 1 time.
[10249.539918] EtherCAT WARNING: Datagram f3bf2e8c (domain1-114-main)
was SKIPPED 1 time.
[10249.635934] EtherCAT 0: Domain 0: 9 working counter changes - now 17/17.
[10249.919163] EtherCAT 0: Domain 2: Working counter changed to 1/1.
[10250.018245] EtherCAT 0: Domain 1: Working counter changed to 1/1.
[10250.040259] EtherCAT 0: Slave states on main device: OP.
[10250.536654] EtherCAT WARNING 0: 4001 datagrams UNMATCHED!
[10251.537469] EtherCAT WARNING 0: 4000 datagrams UNMATCHED!
[10252.538282] EtherCAT WARNING 0: 4000 datagrams UNMATCHED!
[10252.538296] EtherCAT WARNING 0: 4000 datagrams UNMATCHED!
[10253.539097] EtherCAT WARNING 0: 4000 datagrams UNMATCHED!
[10254.539909] EtherCAT WARNING 0: 4000 datagrams UNMATCHED!
[10255.540723] EtherCAT WARNING 0: 4000 datagrams UNMATCHED!
[10256.541537] EtherCAT WARNING 0: 4000 datagrams UNMATCHED!<— This
continues indefinitely


Everything else seems okay:

user@core:~/dist$ ethercat slaves
0  0:0  OP  +  EK1100 EtherCAT-Koppler (2A E-Bus)
1  0:1  OP  +  EL1104 4K. Dig. Eingang 24V, 3ms, Sensorversorgung
2  0:2  OP  +  EL1104 4K. Dig. Eingang 24V, 3ms, Sensorversorgung
3  0:3  OP  +  EL5101 1K. Inc. Encoder 5V
4  0:4  OP  +  EL7041 1K. Schrittmotor-Endstufe (50V, 5A)
5  0:5  OP  +  EL7041 1K. Schrittmotor-Endstufe (50V, 5A)
6  0:6  OP  +  EL2004 4K. Dig. Ausgang 24V, 0.5A
7  0:7  OP  +  EL2004 4K. Dig. Ausgang 24V, 0.5A
8  0:8  OP  +  EL4001 1K. Ana. Ausgang 0-10V, 12bit
9  1:0  OP  +  0x0539:0x0221
user@core:~/dist$ ethercat master
Master0
   Phase: Operation
   Active: yes
   Slaves: 10
   Ethernet devices:
 Main: 00:30:18:c8:d4:92 (attached)
   Link: UP
   Tx frames:   5204542
   Tx bytes:733318966
   Rx frames:   5204539
   Rx bytes:733318636
   Tx errors:   0
   Tx frame rate [1/s]:   4000   3944   2090
   Tx rate [KByte/s]:823.7  811.9  412.9
   Rx frame rate [1/s]:   4000   3944   2090
   Rx rate [KByte/s]:823.7  811.9  412.9
 Common:
   Tx frames:   5204542
   Tx bytes:733318966
   Rx frames:   5204539
   Rx bytes:733318636
   Lost frames: 3
   Tx frame rate [1/s]:   4000   3944   2090
   Tx rate [KByte/s]:823.7  811.9  412.9
   Rx frame rate [1/s]:   4000   3944   2090
   Rx rate [KByte/s]:823.7  811.9  412.9
   Loss rate [1/s]:  0  0  0
   Frame loss [%]: 0.00.00.0
   Distributed clocks:
 

Re: [etherlab-users] Distributed Clocks

[Thomas Bitsky Jr]

No, this was my TIMESPEC2NS:

#define TIMESPEC2NS(T) ((uint64_t) (T).tv_sec * NANOS_PER_SEC + (T).tv_nsec)


So, I’ll update it to:
#define TIMESPEC2NSEPOCH2000(T) ((uint64_t) (((T).tv_sec - 946684800ULL) * 
10ULL) + (T).tv_nsec)


I hadn’t realized I needed to adjust for EtherCAT time EPOCH at all. Thanks!

Thanks!
Thomas C. Bitsky Jr. | Lead Developer

ADC | automateddesign.com 
P: 630-783-1150 F: 630-783-1159 M: 630-632-6679

Follow ADC news and media:
Facebook  | Twitter 
 | YouTube 









On 2/19/16, 7:25 PM, "Matthieu Bec"  wrote:

>Hello Thomas,
>
>Just a wild guess, are you adjusting for EtherCAT time EPOCH? I have something 
>like:
>
>static inline uint64_t TIMESPEC2NSEPOCH2000(struct timespec ts)
>{
>  return (ts.tv_sec - 946684800ULL) * 10ULL + ts.tv_nsec;
>}
>
>Regards,
>Matthieu
>
>
>
>
>
>
>
>
>On 2/19/16, 4:08 PM, "etherlab-users on behalf of Thomas Bitsky Jr" 
> 
>wrote:
>
>Hello.
>
>
>I’ve been using the EtherCAT master for years to great success, but I’m stuck 
>on a problem I can’t figure out that I think several people here are doing 
>successfully. I can’t implement distributed clocks in my application.
>
>
>I am having the same problem on two systems I have up and running:
>
>
>
>
>
>
>SYSTEM ONE:
>
>
>EtherLAB Master 1.52, E1000E Driver, Scan Rate 4Khz, Ubuntu Server 14.04LTS, 
>RT-PREEMPT 3.12.50-rt68
>
>
>alias=0, position=0, device=EK1100 
>alias=0, position=1, device=EL1104 
>alias=0, position=2, device=EL2004
>alias=0, position=3, device=EL9510
>alias=0, position=4, device=EL3356
>alias=0, position=5, device=Kollmorgen AKD
>alias=0, position=6, device=MTS LVDT
>
>
>
>
>
>SYSTEM TWO:
>
>
>EtherLAB Master 1.52, E1000E Driver, Scan Rate 8Khz, Ubuntu Server 14.04LTS, 
>RT-PREEMPT 3.12.50-rt68
>
>
>alias=0, position=0, device=EK1100
>alias=0, position=1, device=EL3001
>alias=0, position=2, device=EL1104
>alias=0, position=3, device=EL1104
>alias=0, position=4, device=EL1104
>alias=0, position=5, device=EL2004
>alias=0, position=6, device=EL2004
>alias=0, position=7, device=EL9505
>alias=0, position=8, device=EL7041
>alias=0, position=9, device=EL7041
>alias=0, position=10, device=EL7041
>alias=0, position=11, device=EL7041
>alias=0, position=12, device=EL7041
>alias=0, position=13, device=EL7041
>alias=0, position=14, device=EK1110
>alias=1, position=0, device=SIGMA5-05
>alias=2, position=0, device=Yaskawa SIGMA5-05
>alias=3, position=0, device=Yaskawa SIGMA5-05
>
>
>
>
>
>Both of the system are fully operational. However, for various reasons, I need 
>to implement distributed clocks on these systems. I’ve never been able to get 
>this to work.
>
>
>
>
>What follows is the code I used for both systems to try this. I read through 
>examples on the mailing list, plus the examples, but I’m not seeing what I’m 
>doing wrong. The result is always the same: all the fieldbus cards go into 
>operation, but the servo
> drives won’t because of “bad configuration.” Take out the distributed clock 
> code, and they work fine. I’m getting away with it for now, but I do need 
> better clock resolution.
>
>
>The systems have an LRW domain, then a separate read domain and write domain 
>for the servo drive(s) for a total of three domains (LRW, read, write). The 
>yaskawa drives necessitate this. The scan rate is usually 4Khz, but I have 
>tried it at both 1Khz and
> 8Khz and gotten the same results. Everything about the implementation is 
> fairly straight-forward. There’s just something fundamental about the DC 
> configuration that I’m not understanding.
>
>
>Almost all the code below is taken right from the examples or the message 
>boards (thanks, everybody!). If anyone could tell me what I’m going wrong or 
>offer any insights, it’s greatly appreciated. For brevity, I tried to narrow 
>it down to relevant parts,
> but please let me know any additional information or code I can provide.
>
>
>Thank you in advance,
>Tom
>
>
>
>
>**
>
>
>// EtherCAT distributed clock variables
>
>
>#define DC_FILTER_CNT  1024
>#define SYNC_MASTER_TO_REF1
>
>
>static uint64_t dc_start_time_ns = 0LL;
>static uint64_t dc_time_ns = 0;
>static uint8_t  dc_started = 0;
>static int32_t  dc_diff_ns = 0;
>static int32_t  prev_dc_diff_ns = 0;
>static int64_t  dc_diff_total_ns = 0LL;
>static int64_t  dc_delta_total_ns = 0LL;
>static int  dc_filter_idx = 0;
>static int64_t  dc_adjust_ns;
>static int64_t  system_time_base = 0LL;
>static uint64_t wakeup_time = 0LL;
>static uint64_t overruns = 0LL;
>
>
>
>
>
>
>/** Get the time in ns for the current cpu, adjusted by system_time_base.
> *
> * \attention Rather than calling rt_get_time_ns() directly, all application
> * time calls should use this 

Re: [etherlab-users] distributed clocks

[Andreas Stewering-Bone]

Hello,

yes it is possible, but you are using an old version.

There will be a new release in the next month I think.
The new release will implement a better DC Handling and documentation.

We made tests on Beckhoff AX5xxx Servocontrollers and it works fine.

Greatings

Andreas



Am Friday 28 May 2010 10:39:01 schrieb dkarolewski:
 Hi all,
 we want use distributed clocks in an realtime application.
 Our system configuration is:

 etherlab-1.3-pre-r589
 ethercat 1.5
 matlab2007a
 ubuntu 9.10

 I found in the simulink block generic slave the item Distributed
 clock, but this entry have no connection to the S-function ec_slave2.
 Is the use of distributed clocks with etherlab possible?
 Does anyone have experience with it?
 Is there any documentation on this subject?


 Thanks in advance.


___
etherlab-users mailing list
etherlab-users@etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-users

Re: [etherlab-users] Distributed Clocks

[Jörg Eßmann]

Hi Florian,

it would be great, having DC for my application. Therefore, I can bear a
part of the expenses. Please put me on the list!

Kind regards

Joerg Essmann



-Ursprüngliche Nachricht-
Von: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] Im Auftrag von Florian Pose
Gesendet: Montag, 20. Oktober 2008 10:35
An: EtherLab Users List
Betreff: [etherlab-users] Distributed Clocks

Hi all,

get more and more request concerning a DC implementation. If I will do
the design/implementation, this will take about a week of work for me.
IgH will not need this feature in the near future, but there are several
companies/institutes, that would want to finance the implementation.

An idea is to collect all companies/institutes wanting to participate
and share the implementation costs among all. So if you have an interest
in this feature, post to the list, or write me directly. I'll manage a
list and announce the number of participants.

What is your opinion regarding this?

-- 
Best regards,
Florian Pose

http://etherlab.org
___
etherlab-users mailing list
etherlab-users@etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-users

___
etherlab-users mailing list
etherlab-users@etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-users