We also are fighting with this type of problem on a customer laser cutting machine. Occasionally we see errors like this: [122501.934306] EtherCAT 0: Domain 0: Working counter changed to 0/9. [122501.934346] EtherCAT 0: Domain 1: Working counter changed to 0/9. [122502.320449] EtherCAT WARNING 0: 5 datagrams TIMED OUT! [122502.935224] EtherCAT 0: Domain 0: Working counter changed to 9/9. [122502.935265] EtherCAT 0: Domain 1: Working counter changed to 9/9.
This was the reason I modified the ethercat command line tool for extended diagnostics regarding several ESC error registers. Attached you will find a patch which might help you. After applying and building the ethercat command line tool it will provide a new command "diag". * Shortly after your ethercat master has been started successfully call: ethercat diag -r This will reset all slaves ESC error registers including Lost Link Counter Register and RX Error Counter Register. * If you detect a an error UNMATCHED and TIMEOUT (sometimes after hours or days) call: ethercat diag If you are lucky you will find one ore more ESC errors displayed on your console. For better understanding the displayed errors you should to picture picture http://www.automation.com/images/article/ethercat/Figure14.jpg (part of http://www.automation.com/automation-news/article/diagnostics-with-ethercat-part-4). Would be happy about any kind of feedback. @Henry: which type of drives do you use? Regards, Ralf On Mon Jul 04 2016 05:19:58 GMT+0200 (CEST), Graeme Foot <graeme.f...@touchcut.com> wrote: > The only time we've had issues like that has been due to either a dodgy > network cable or an RJ45 plug getting a bit grubby. First thing I usually do > is unplug/replug all the plugs a few time to clean up the connections. If it > persists then I start looking for bad cables. > > Another option is that there is an occasional noisy process causing noise on > one of the links. > > Once or twice (only on non-ethercat machines so far) we've had cables that > were in drag chains wearing out, where it showed a problem when at a specific > position of the drag chain. > > You could track down if it's a problem with a link between two particular > slaves by checking each slaves Link Lost Counter and CRC Bad Counter values. > - Lost Link Counter Register (0x0310:0x0313) > - RX Error Counter Register (0x0300:0x0307) > > This link describes some of the diagnostics: > http://www.automation.com/automation-news/article/diagnostics-with-ethercat-part-4 > > I think you can set the above registers to zero after the fieldbus is up and > running, then you can check them if a problem occurs. > > > Haven't actually done it yet myself, so would be interested to see if it > helps you. > > > Regards, > Graeme. > > > > > -----Original Message----- > From: etherlab-users [mailto:etherlab-users-boun...@etherlab.org] On Behalf > Of Henry Bausley > Sent: Saturday, 2 July 2016 5:56 a.m. > To: etherlab-users@etherlab.org > Subject: [etherlab-users] Intermittent Large number of datagrams UNMATCHED > > > > We have a etherlab 1.5.2 kernel mode application running in xenomai > 2.4.6 on Ubuntu 14.04.1 Desktop that will get on rare occasions a large > number of datagrams UNMATCHED. It occurs at random times and relatively > rarely but when it occurs it can result in disaster as we are running a large > number of servos in torque mode. > > For example we can run continuously for 5 days 24hours continuously then get > a message like something below. > > [591785.735172] EtherCAT WARNING 0: 616 datagrams UNMATCHED! > I am struggling as to where to look. Is this something in our app or a known > bug in the stack? > > > > > > Outbound scan for Spam or Virus by Barracuda at Delta Tau > > _______________________________________________ > etherlab-users mailing list > etherlab-users@etherlab.org > http://lists.etherlab.org/mailman/listinfo/etherlab-users > _______________________________________________ > etherlab-users mailing list > etherlab-users@etherlab.org > http://lists.etherlab.org/mailman/listinfo/etherlab-users
>From a6eff643e4ac450284d9be72202246e9c8fc8951 Mon Sep 17 00:00:00 2001 From: Ralf Roesch <ralf.roe...@rw-gmbh.de> Date: Sat, 14 May 2016 12:32:16 +0200 Subject: [PATCH] [ethercat tool] added "diag" command to display various ESC error registers The ethercat diag command can be seleced for specific slaves. The command can be used with two options --reset / -r Reset command. Resets all error registers in ESC. Should be used short after starting EtherCAT. --verbose / -v Verbose output error ESC registers of selected slave(s). Can be used at any time to check if ESC errors have been detected. Signed-off-by: Ralf Roesch <ralf.roe...@rw-gmbh.de> --- tool/Command.cpp | 7 + tool/Command.h | 11 ++ tool/CommandDiag.cpp | 392 +++++++++++++++++++++++++++++++++++++++++++++++++++ tool/CommandDiag.h | 76 ++++++++++ tool/Makefile.am | 1 + tool/main.cpp | 12 +- 6 files changed, 498 insertions(+), 1 deletion(-) create mode 100644 tool/CommandDiag.cpp create mode 100644 tool/CommandDiag.h diff --git a/tool/Command.cpp b/tool/Command.cpp index dccc9d6..eee04cc 100644 --- a/tool/Command.cpp +++ b/tool/Command.cpp @@ -220,6 +220,13 @@ void Command::setForce(bool f) /*****************************************************************************/ +void Command::setReset(bool r) +{ + reset = r; +}; + +/*****************************************************************************/ + void Command::setOutputFile(const string &f) { outputFile = f; diff --git a/tool/Command.h b/tool/Command.h index 1f28723..4350b34 100644 --- a/tool/Command.h +++ b/tool/Command.h @@ -113,6 +113,9 @@ class Command void setForce(bool); bool getForce() const; + void setReset(bool); + bool getReset() const; + void setOutputFile(const string &); const string &getOutputFile() const; @@ -158,6 +161,7 @@ class Command string dataType; bool emergency; bool force; + bool reset; string outputFile; string skin; @@ -208,6 +212,13 @@ inline bool Command::getForce() const /****************************************************************************/ +inline bool Command::getReset() const +{ + return reset; +} + +/****************************************************************************/ + inline const string &Command::getOutputFile() const { return outputFile; diff --git a/tool/CommandDiag.cpp b/tool/CommandDiag.cpp new file mode 100644 index 0000000..6b4a6a6 --- /dev/null +++ b/tool/CommandDiag.cpp @@ -0,0 +1,392 @@ +/***************************************************************************** + * + * $Id$ + * + * Copyright (C) 2016 Ralf Roesch, Roesch & Walter Industrie-Elektronik GmbH + * Copyright (C) 2006-2012 Florian Pose, Ingenieurgemeinschaft IgH + * + * This file is part of the IgH EtherCAT Master. + * + * The IgH EtherCAT Master is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License version 2, as + * published by the Free Software Foundation. + * + * The IgH EtherCAT Master is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General + * Public License for more details. + * + * You should have received a copy of the GNU General Public License along + * with the IgH EtherCAT Master; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + * + * --- + * + * The license mentioned above concerns the source code only. Using the + * EtherCAT technology and brand is only permitted in compliance with the + * industrial property and similar rights of Beckhoff Automation GmbH. + * + ****************************************************************************/ +#include <stdio.h> +#include <iostream> +#include <iomanip> +#include <string.h> +using namespace std; + +#include "CommandDiag.h" +#include "MasterDevice.h" + +/*****************************************************************************/ + +CommandDiag::CommandDiag(): + Command("diag", "Output slave(s) ESC error registers.") +{ +} + +/*****************************************************************************/ + +string CommandDiag::helpString(const string &binaryBaseName) const +{ + stringstream str; + + str << binaryBaseName << " " << getName() + << " [OPTIONS]" << endl + << endl + << getBriefDescription() << endl + << endl + << "Command-specific options:" << endl + << " --alias -a <alias>" << endl + << " --position -p <pos> Slave selection." << endl + << " --reset -r Reset command. Resets all error registers in ESC." << endl + << " --verbose -v Verbose output error ESC registers of selected slave(s)." << endl + << endl; + + return str.str(); +} + +/****************************************************************************/ + +void CommandDiag::execute(const StringVector &args) +{ + MasterIndexList masterIndices; + SlaveList slaves; + bool doIndent; + + if (args.size() > 1) { + stringstream err; + err << "'" << getName() << "' takes max one argument!"; + throwInvalidUsageException(err); + } + + masterIndices = getMasterIndices(); + doIndent = masterIndices.size() > 1; + MasterIndexList::const_iterator mi; + for (mi = masterIndices.begin(); + mi != masterIndices.end(); mi++) { + MasterDevice m(*mi); + m.open(MasterDevice::ReadWrite); + slaves = selectedSlaves(m); + + CheckallSlaves(m, slaves, doIndent); + } +} + +uint32_t CommandDiag::EscRegRead(MasterDevice &m, uint16_t slave_position, uint16_t address, const string &sdataType) +{ + ec_ioctl_slave_reg_t io; + uint32_t data; + stringstream err; + const DataType *dataType; + + if (!(dataType = findDataType(sdataType))) { + err << "Invalid data type '" << sdataType << "'!"; + throwInvalidUsageException(err); + } + + if (!dataType->byteSize) { + err << "The size argument is mandatory, if no datatype is " << endl + << "specified, or the datatype does not imply a size!"; + throwInvalidUsageException(err); + } + + if (dataType->byteSize > 4) { + err << "The size argument (" << dataType->byteSize << ") is invalid (> 4)!"; + throwInvalidUsageException(err); + } + + data = 0; + io.slave_position = slave_position; + io.address = address; + io.size = dataType->byteSize; + io.data = (uint8_t *) &data; + io.emergency = false; + + + try { + m.readReg(&io); + } catch (MasterDeviceException &e) { + printf("EscRegRead slave = %i, address = %x, size = %i\n", io.slave_position, io.address, io.size); + throw e; + } + + return data; +} + +int CommandDiag::EscRegWrite(MasterDevice &m, uint16_t slave_position, uint16_t address, const string &sdataType, uint32_t data) +{ + ec_ioctl_slave_reg_t io; + stringstream err; + const DataType *dataType; + + if (!(dataType = findDataType(sdataType))) { + err << "Invalid data type '" << sdataType << "'!"; + throwInvalidUsageException(err); + } + + if (!dataType->byteSize) { + err << "The size argument is mandatory, if no datatype is " << endl + << "specified, or the datatype does not imply a size!"; + throwInvalidUsageException(err); + } + + if (dataType->byteSize > 4) { + err << "The size argument (" << dataType->byteSize << ") is invalid (> 4)!"; + throwInvalidUsageException(err); + } + + io.slave_position = slave_position; + io.address = address; + io.size = dataType->byteSize; + io.data = (uint8_t *) &data; + io.emergency = false; + + try { + m.writeReg(&io); + } catch (MasterDeviceException &e) { + printf("EscRegWrite slave = %i, address = %x, size = %i, data = 0x%x\n", io.slave_position, io.address, io.size, data); + throw e; + } + + return 0; +} + +/****************************************************************************/ + +void CommandDiag::CheckallSlaves( + MasterDevice &m, + const SlaveList &slaves, + bool doIndent + ) +{ + ec_ioctl_master_t master; + unsigned int i, lastDevice; + ec_ioctl_slave_t slave; + uint16_t lastAlias, aliasIndex; + Info info; + typedef list<Info> InfoList; + InfoList infoList; + InfoList::const_iterator iter; + stringstream str; + unsigned int maxPosWidth = 0, maxAliasWidth = 0, + maxRelPosWidth = 0, maxStateWidth = 0, + maxESCerrorsWidth = 0; + string indent(doIndent ? " " : ""); + + m.getMaster(&master); + + lastAlias = 0; + aliasIndex = 0; + for (i = 0; i < master.slave_count; i++) { + m.getSlave(&slave, i); + + if (slave.alias) { + lastAlias = slave.alias; + aliasIndex = 0; + } + + if (slaveInList(slave, slaves)) { + int slave_position = i; + int llc_reset = 0; + str << dec << i; + info.pos = str.str(); + str.clear(); + str.str(""); + + str << lastAlias; + info.alias = str.str(); + str.str(""); + + str << aliasIndex; + info.relPos = str.str(); + str.str(""); + + info.state = alStateString(slave.al_state); + info.flag = (slave.error_flag ? 'E' : '+'); + info.device = slave.device_index; + + if (strlen(slave.name)) { + info.name = slave.name; + } else { + str << "0x" << hex << setfill('0') + << setw(8) << slave.vendor_id << ":0x" + << setw(8) << slave.product_code; + info.name = str.str(); + str.str(""); + } + + info.ESC_DL_Status = EscRegRead(m, slave_position, 0x110, "uint16"); + + info.ESCerrors = 0; + + for (int i = 0; i < EC_MAX_PORTS; i++) { + int check_port; + /* check port only if: loop is open, and Communication established */ + check_port = (((info.ESC_DL_Status >> (8 + i * 2)) & 0x3) == 2) ? 0x1 : 0x0; + /* some error registers are only availble when MII or EBUS port is present */ + check_port = check_port + (slave.ports[i].desc == EC_PORT_MII) ? 1 : 0; + check_port = check_port + (slave.ports[i].desc == EC_PORT_EBUS) ? 1 : 0; + info.Invalid_Frame_Counter[i] = (check_port > 0) ? EscRegRead(m, slave_position, 0x300 + (i * 2), "uint8") : 0; + info.RX_Error_Counter[i] = (check_port > 0) ? EscRegRead(m, slave_position, 0x301 + (i * 2), "uint8") : 0; + info.Forwarded_RX_Error_Counter[i] = (check_port > 1) ? EscRegRead(m, slave_position, 0x308 + i, "uint8") : 0; + info.ECAT_Processing_Unit_Error_Counter = (check_port > 1) && (i == 0) ? EscRegRead(m, slave_position, 0x30C, "uint8") : 0; + info.Lost_Link_Counter[i] = (check_port > 1) ? EscRegRead(m, slave_position, 0x310 + i, "uint8") : 0; + + info.ESCerrors = info.Invalid_Frame_Counter[i] ? info.ESCerrors | 0x01 : info.ESCerrors; + info.ESCerrors = info.RX_Error_Counter[i] ? info.ESCerrors | 0x02 : info.ESCerrors; + info.ESCerrors = info.Forwarded_RX_Error_Counter[i] ? info.ESCerrors | 0x04 : info.ESCerrors; + info.ESCerrors = info.ECAT_Processing_Unit_Error_Counter ? info.ESCerrors | 0x08 : info.ESCerrors; + info.ESCerrors = info.Lost_Link_Counter[i] ? info.ESCerrors | 0x10 : info.ESCerrors; + llc_reset = (i == 0) && (check_port > 1) ? 1 : llc_reset; + } + + if (info.ESCerrors) { + info.sESCerrors = ""; + if (info.ESCerrors & 0x01) { + info.sESCerrors += "E_IFC "; + } + if (info.ESCerrors & 0x02) { + info.sESCerrors += "E_REC "; + } + if (info.ESCerrors & 0x04) { + info.sESCerrors += "E_FREC "; + } + if (info.ESCerrors & 0x08) { + info.sESCerrors += "E_PUEC "; + } + if (info.ESCerrors & 0x10) { + info.sESCerrors += "E_LLC "; + } + } else { + info.sESCerrors = "ESC no errors. "; + } + + infoList.push_back(info); + + if (info.pos.length() > maxPosWidth) + maxPosWidth = info.pos.length(); + if (info.alias.length() > maxAliasWidth) + maxAliasWidth = info.alias.length(); + if (info.relPos.length() > maxRelPosWidth) + maxRelPosWidth = info.relPos.length(); + if (info.state.length() > maxStateWidth) + maxStateWidth = info.state.length(); + if (info.sESCerrors.length() > maxESCerrorsWidth) + maxESCerrorsWidth = info.sESCerrors.length(); + + if (getReset()) { + /* clear all Invalid Frame Counters, RX Error Counters and Forwarded RX Error Counters */ + EscRegWrite(m, slave_position, 0x300, "uint8", 0x0); + if (llc_reset) { + /* clear all Lost Link Counters */ + EscRegWrite(m, slave_position, 0x310, "uint8", 0x0); + /* clear ECAT Processing Unit Error Counter */ + EscRegWrite(m, slave_position, 0x30C, "uint8", 0x0); + } + } + + } + + aliasIndex++; + } + + if (infoList.size() && doIndent) { + cout << "Master" << dec << m.getIndex() << endl; + } + + lastDevice = EC_DEVICE_MAIN; + for (iter = infoList.begin(); iter != infoList.end(); iter++) { + if (iter->device != lastDevice) { + lastDevice = iter->device; + cout << "xxx LINK FAILURE xxx" << endl; + } + cout << indent << setfill(' ') << right + << setw(maxPosWidth) << iter->pos << " " + << setw(maxAliasWidth) << iter->alias + << ":" << left + << setw(maxRelPosWidth) << iter->relPos << " " + << setw(maxStateWidth) << iter->state << " " + << iter->flag << " " + << setw(maxESCerrorsWidth) << iter->sESCerrors << " " + << iter->name << endl; + if (getVerbosity() == Verbose) { + string indent(" "); + if (iter->ESCerrors & 0x01) { + cout << indent << "Invalid Frame Counter -"; + for (int i = 0; i < EC_MAX_PORTS; i++) { + if (iter->Invalid_Frame_Counter[i]) { + cout << dec << " P[" << i << "]: " << iter->Invalid_Frame_Counter[i]; + } + } + cout << endl; + } + if (iter->ESCerrors & 0x02) { + cout << indent << "RX Error counter -"; + for (int i = 0; i < EC_MAX_PORTS; i++) { + if (iter->RX_Error_Counter[i]) { + cout << dec << " P[" << i << "]: " << iter->RX_Error_Counter[i]; + } + } + cout << endl; + } + if (iter->ESCerrors & 0x04) { + cout << indent << "Forwarded RX Error Counter -"; + for (int i = 0; i < EC_MAX_PORTS; i++) { + if (iter->Forwarded_RX_Error_Counter[i]) { + cout << dec << " P[" << i << "]: " << iter->Forwarded_RX_Error_Counter[i]; + } + } + cout << endl; + } + if (iter->ESCerrors & 0x08) { + cout << indent << "ECAT Processing Unit Error Counter - "; + cout << dec << " << iter->ECAT_Processing_Unit_Error_Counter" << endl; + } + if (iter->ESCerrors & 0x10) { + cout << indent << "Lost Link Counter -"; + for (int i = 0; i < EC_MAX_PORTS; i++) { + if (iter->Lost_Link_Counter[i]) { + cout << dec << " P[" << i << "]: " << iter->Lost_Link_Counter[i]; + } + } + cout << endl; + } + } + } +} + +bool CommandDiag::slaveInList( + const ec_ioctl_slave_t &slave, + const SlaveList &slaves + ) +{ + SlaveList::const_iterator si; + + for (si = slaves.begin(); si != slaves.end(); si++) { + if (si->position == slave.position) { + return true; + } + } + + return false; +} + diff --git a/tool/CommandDiag.h b/tool/CommandDiag.h new file mode 100644 index 0000000..bdf57e1 --- /dev/null +++ b/tool/CommandDiag.h @@ -0,0 +1,76 @@ +/***************************************************************************** + * + * $Id$ + * + * Copyright (C) 2016 Ralf Roesch, Roesch & Walter Industrie-Elektronik GmbH + * Copyright (C) 2006-2009 Florian Pose, Ingenieurgemeinschaft IgH + * + * This file is part of the IgH EtherCAT Master. + * + * The IgH EtherCAT Master is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License version 2, as + * published by the Free Software Foundation. + * + * The IgH EtherCAT Master is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General + * Public License for more details. + * + * You should have received a copy of the GNU General Public License along + * with the IgH EtherCAT Master; if not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + * + * --- + * + * The license mentioned above concerns the source code only. Using the + * EtherCAT technology and brand is only permitted in compliance with the + * industrial property and similar rights of Beckhoff Automation GmbH. + * + ****************************************************************************/ + +#ifndef __COMMANDDIAG_H__ +#define __COMMANDDIAG_H__ + +#include "Command.h" +#include "DataTypeHandler.h" + +/****************************************************************************/ + +class CommandDiag: + public Command, + public DataTypeHandler +{ + public: + CommandDiag(); + + string helpString(const string &) const; + void execute(const StringVector &); + + protected: + struct Info { + string pos; + string alias; + string relPos; + string state; + string flag; + string name; + unsigned int device; + unsigned ESCerrors; + string sESCerrors; + unsigned int ESC_DL_Status; + unsigned int Invalid_Frame_Counter[EC_MAX_PORTS]; + unsigned int RX_Error_Counter[EC_MAX_PORTS]; + unsigned int Forwarded_RX_Error_Counter[EC_MAX_PORTS]; + unsigned int ECAT_Processing_Unit_Error_Counter; + unsigned int Lost_Link_Counter[EC_MAX_PORTS]; + }; + void CheckallSlaves(MasterDevice &, const SlaveList &, bool); + static bool slaveInList( const ec_ioctl_slave_t &, const SlaveList &); + uint32_t EscRegRead(MasterDevice &, uint16_t, uint16_t, const std::string &); + int EscRegWrite(MasterDevice &, uint16_t, uint16_t, const std::string &, uint32_t data); +}; + +/****************************************************************************/ + +#endif + diff --git a/tool/Makefile.am b/tool/Makefile.am index 9300bfb..b9ab803 100644 --- a/tool/Makefile.am +++ b/tool/Makefile.am @@ -43,6 +43,7 @@ ethercat_SOURCES = \ CommandConfig.cpp \ CommandData.cpp \ CommandDebug.cpp \ + CommandDiag.cpp \ CommandDomains.cpp \ CommandDownload.cpp \ CommandFoeRead.cpp \ diff --git a/tool/main.cpp b/tool/main.cpp index 9723e5c..b5c12e4 100644 --- a/tool/main.cpp +++ b/tool/main.cpp @@ -40,6 +40,7 @@ using namespace std; #include "CommandCStruct.h" #include "CommandData.h" #include "CommandDebug.h" +#include "CommandDiag.h" #include "CommandDomains.h" #include "CommandDownload.h" #ifdef EC_EOE @@ -86,6 +87,7 @@ Command::Verbosity verbosity = Command::Normal; bool force = false; bool emergency = false; bool helpRequested = false; +bool reset = false; string outputFile; string skin; @@ -122,6 +124,7 @@ string usage() << endl << " --force -f Force a command." << endl << " --quiet -q Output less information." << endl + << " --reset -r reset command." << endl << " --verbose -v Output more information." << endl << " --help -h Show this help." << endl << endl @@ -154,13 +157,14 @@ void getOptions(int argc, char **argv) {"emergency", no_argument, NULL, 'e'}, {"force", no_argument, NULL, 'f'}, {"quiet", no_argument, NULL, 'q'}, + {"reset", no_argument, NULL, 'r'}, {"verbose", no_argument, NULL, 'v'}, {"help", no_argument, NULL, 'h'}, {} }; do { - c = getopt_long(argc, argv, "m:a:p:d:t:o:s:efqvh", longOptions, NULL); + c = getopt_long(argc, argv, "m:a:p:d:t:o:s:efqrvh", longOptions, NULL); switch (c) { case 'm': @@ -203,6 +207,10 @@ void getOptions(int argc, char **argv) verbosity = Command::Quiet; break; + case 'r': + reset = true; + break; + case 'v': verbosity = Command::Verbose; break; @@ -281,6 +289,7 @@ int main(int argc, char **argv) commandList.push_back(new CommandCStruct()); commandList.push_back(new CommandData()); commandList.push_back(new CommandDebug()); + commandList.push_back(new CommandDiag()); commandList.push_back(new CommandDomains()); commandList.push_back(new CommandDownload()); #ifdef EC_EOE @@ -325,6 +334,7 @@ int main(int argc, char **argv) cmd->setSkin(skin); cmd->setEmergency(emergency); cmd->setForce(force); + cmd->setReset(reset); cmd->execute(commandArgs); } catch (InvalidUsageException &e) { cerr << e.what() << endl << endl; -- 2.7.4
_______________________________________________ etherlab-users mailing list etherlab-users@etherlab.org http://lists.etherlab.org/mailman/listinfo/etherlab-users