What’s on IP address 10.0.11.210? It’s sending Ignite something that it doesn’t
understand. Maybe it’s not another copy of Ignite? Could it be a firewall
setting truncating the message? Or perhaps the remote node has a different
configuration, for example mixing up communication and discovery ports?
Regards,
Stephen
> On 24 Sep 2019, at 13:00, Marco Bernagozzi <marco.bernago...@gmail.com> wrote:
>
> Hi.
> I get this error sometimes, it seems to be quite random. Any idea what this
> might be caused by?
> Since every time it's thrown a hundred times or so, I had to temporarily
> suppress all the errors from that class. Is there a way to fix this or at
> least to make it be thrown just once?
>
> My settings:
>
> AwsConfiguration awsConfiguration = AwsConfigurationSingleton.getInstance();
> BasicAWSCredentials creds = new BasicAWSCredentials(
> awsConfiguration.getAwsAccessKey(),
> awsConfiguration.getAwsSecretKey()
> );
> TcpDiscoveryS3IpFinder ipFinder = new TcpDiscoveryS3IpFinder();
> ipFinder.setAwsCredentials(creds);
> ipFinder.setBucketEndpoint("s3.eu-west-1.amazonaws.com
> <http://s3.eu-west-1.amazonaws.com/>");
> ipFinder.setBucketName(awsConfiguration.getIgniteBucket());
>
> TcpDiscoverySpi spi = new TcpDiscoverySpi();
> spi.setIpFinder(ipFinder);
> IgniteConfiguration cfg = new IgniteConfiguration();
> DataStorageConfiguration storageCfg = new DataStorageConfiguration();
> storageCfg.getDefaultDataRegionConfiguration().setPersistenceEnabled(false);
> cfg.setDataStorageConfiguration(storageCfg);
> IgniteLogger log = new Slf4jLogger();
> cfg.setGridLogger(log);
> TcpCommunicationSpi commSpi = new TcpCommunicationSpi();
> commSpi.setMessageQueueLimit(10000);
> cfg.setCommunicationSpi(commSpi);
> cfg.setDiscoverySpi(spi);
> cfg.setBinaryConfiguration(new BinaryConfiguration());
> cfg.getBinaryConfiguration().setCompactFooter(false);
> cfg.setFailureDetectionTimeout(60 * 1000);
>
> The log:
>
> INFO [2019-09-24 10:04:21,402] org.apache.ignite.internal.IgniteKernal:
> >>> __________ ________________
> >>> / _/ ___/ |/ / _/_ __/ __/
> >>> _/ // (7 7 // / / / / _/
> >>> /___/\___/_/|_/___/ /_/ /___/
> >>>
> >>> ver. 2.7.5#20190603-sha1:be4f2a15
> >>> 2018 Copyright(C) Apache Software Foundation
> >>>
> >>> Ignite documentation: http://ignite.apache.org <http://ignite.apache.org/>
> INFO [2019-09-24 10:04:21,402] org.apache.ignite.internal.IgniteKernal:
> Config URL: n/a
> INFO [2019-09-24 10:04:21,413] org.apache.ignite.internal.IgniteKernal:
> IgniteConfiguration [igniteInstanceName=null, pubPoolSize=36, svcPoolSize=36,
> callbackPoolSize=36, stripedPoolSize=36, sysPoolSize=36, mgmtPoolSize=4,
> igfsPoolSize=36, dataStreamerPoolSize=36, utilityCachePoolSize=36,
> utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=36,
> igniteHome=/opt/ignite/apache-ignite-2.7.5-bin,
> igniteWorkDir=/opt/ignite/apache-ignite-2.7.5-bin/work,
> mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@47c81abf,
> nodeId=09156c05-f869-4826-814e-1f47fcefeeb4, marsh=BinaryMarshaller [],
> marshLocJobs=false, daemon=false, p2pEnabled=false, netTimeout=5000,
> sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000,
> metricsUpdateFreq=2000, metricsExpTime=9223372036854775807,
> discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0,
> marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000,
> forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null],
> segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true,
> allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi
> [connectGate=null,
> connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@104d9de4,
> enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null,
> locHost=null, locPort=47100, locPortRange=100, shmemPort=-1, directBuf=true,
> directSndBuf=false, idleConnTimeout=600000, connTimeout=5000,
> maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768,
> msgQueueLimit=10000, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null,
> usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true,
> filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0,
> sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1,
> selectorsCnt=18, selectorSpins=0, addrRslvr=null,
> ctxInitLatch=java.util.concurrent.CountDownLatch@2b43b6cc[Count = 1],
> stopping=false],
> evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@3221402f,
> colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [],
> indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@415fad70,
> addrRslvr=null,
> encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@7947ad4c,
> clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration
> [txSerEnabled=false, dfltIsolation=REPEATABLE_READ,
> dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0,
> txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0,
> pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null,
> useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000,
> deployMode=SHARED, p2pMissedCacheSize=100, locHost=null,
> timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=60000,
> sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=30000,
> metricsLogFreq=60000, hadoopCfg=null, connectorCfg=ConnectorConfiguration
> [jettyPath=null, host=null, port=11211, noDelay=true, directBuf=false,
> sndBufSize=32768, rcvBufSize=32768, idleQryCurTimeout=600000,
> idleQryCurCheckFreq=60000, sndQueueLimit=0, selectorCnt=4, idleTimeout=7000,
> sslEnabled=false, sslClientAuth=false, sslCtxFactory=null, sslFactory=null,
> portRange=100, threadPoolSize=36, msgInterceptor=null], odbcCfg=null,
> warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000,
> cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null,
> sslCtxFactory=null, platformCfg=null, binaryCfg=BinaryConfiguration
> [idMapper=null, nameMapper=null, serializer=null, compactFooter=false],
> memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration
> [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0,
> concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default,
> maxSize=14753283276, initSize=268435456, swapPath=null,
> pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100,
> metricsEnabled=false, metricsSubIntervalCount=5,
> metricsRateTimeInterval=60000, persistenceEnabled=false,
> checkpointPageBufSize=0], dataRegions=null, storagePath=null,
> checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4,
> checkpointWriteOrder=SEQUENTIAL, walHistSize=20,
> maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864,
> walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false,
> walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000,
> walFsyncDelay=1000, walRecordIterBuffSize=67108864,
> alwaysWriteFullPages=false,
> fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@349479c0,
> metricsSubIntervalCnt=5, metricsRateTimeInterval=60000,
> walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false,
> walCompactionEnabled=false, walCompactionLevel=1,
> checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true,
> longQryWarnTimeout=3000, sqlConnCfg=null,
> cliConnCfg=ClientConnectorConfiguration [host=null, port=10800,
> portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true,
> maxOpenCursorsPerConn=128, threadPoolSize=36, idleTimeout=0,
> jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false,
> useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null],
> mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false,
> failureHnd=null, commFailureRslvr=null]
> INFO [2019-09-24 10:04:21,413] org.apache.ignite.internal.IgniteKernal:
> Daemon mode: off
> INFO [2019-09-24 10:04:21,413] org.apache.ignite.internal.IgniteKernal: OS:
> Linux 4.4.0-1020-aws amd64
> INFO [2019-09-24 10:04:21,413] org.apache.ignite.internal.IgniteKernal: OS
> user: root
> INFO [2019-09-24 10:04:21,415] org.apache.ignite.internal.IgniteKernal: PID: 1
> INFO [2019-09-24 10:04:21,416] org.apache.ignite.internal.IgniteKernal:
> Language runtime: Java Platform API Specification ver. 11
> INFO [2019-09-24 10:04:21,416] org.apache.ignite.internal.IgniteKernal: VM
> information: OpenJDK Runtime Environment 11.0.3+1-Debian-1bpo91 Oracle
> Corporation OpenJDK 64-Bit Server VM 11.0.3+1-Debian-1bpo91
> INFO [2019-09-24 10:04:21,416] org.apache.ignite.internal.IgniteKernal: VM
> total memory: 17.0GB
> INFO [2019-09-24 10:04:21,416] org.apache.ignite.internal.IgniteKernal:
> Remote Management [restart: off, REST: on, JMX (remote: off)]
> INFO [2019-09-24 10:04:21,416] org.apache.ignite.internal.IgniteKernal:
> Logger: Slf4jLogger [impl=Logger[org.apache.ignite.internal.IgniteKernal],
> quiet=false]
> INFO [2019-09-24 10:04:21,416] org.apache.ignite.internal.IgniteKernal:
> IGNITE_HOME=/opt/ignite/apache-ignite-2.7.5-bin
> INFO [2019-09-24 10:04:21,416] org.apache.ignite.internal.IgniteKernal: VM
> arguments: [-XX:+AlwaysPreTouch, -XX:+UseG1GC, -XX:+DisableExplicitGC,
> -XX:+ScavengeBeforeFullGC,
> --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED,
> --add-exports=java.base/sun.nio.ch <http://sun.nio.ch/>=ALL-UNNAMED,
> --add-exports=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED,
> --add-exports=jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED,
> --add-exports=java.base/sun.reflect.generics.reflectiveObjects=ALL-UNNAMED,
> --illegal-access=permit, -Djdk.tls.client.protocols=TLSv1.2,
> -Djava.net.preferIPv4Stack=true, -DIGNITE_QUIET=false]
> INFO [2019-09-24 10:04:21,416] org.apache.ignite.internal.IgniteKernal:
> System cache's DataRegion size is configured to 40 MB. Use
> DataStorageConfiguration.systemRegionInitialSize property to change the
> setting.
> INFO [2019-09-24 10:04:21,420] org.apache.ignite.internal.IgniteKernal:
> Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']]
> INFO [2019-09-24 10:04:21,422] org.apache.ignite.internal.IgniteKernal: 3-rd
> party licenses can be found at:
> /opt/ignite/apache-ignite-2.7.5-bin/libs/licenses
> INFO [2019-09-24 10:04:21,422] org.apache.ignite.internal.IgniteKernal: Local
> node user attribute [algotworker=true]
> INFO [2019-09-24 10:04:21,460]
> org.apache.ignite.internal.processors.plugin.IgnitePluginProcessor:
> Configured plugins:
> INFO [2019-09-24 10:04:21,460]
> org.apache.ignite.internal.processors.plugin.IgnitePluginProcessor: ^-- None
> INFO [2019-09-24 10:04:21,460]
> org.apache.ignite.internal.processors.plugin.IgnitePluginProcessor:
> INFO [2019-09-24 10:04:21,460]
> org.apache.ignite.internal.processors.failure.FailureProcessor: Configured
> failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED,
> SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]]
> INFO [2019-09-24 10:04:21,495]
> org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi: Successfully
> bound communication NIO server to TCP port [port=47100,
> locHost=0.0.0.0/0.0.0.0 <http://0.0.0.0/0.0.0.0>, selectorsCnt=18,
> selectorSpins=0, pairedConn=false]
> WARN [2019-09-24 10:04:21,519]
> org.apache.ignite.spi.checkpoint.noop.NoopCheckpointSpi: Checkpoints are
> disabled (to enable configure any GridCheckpointSpi implementation)
> WARN [2019-09-24 10:04:21,533]
> org.apache.ignite.internal.managers.collision.GridCollisionManager: Collision
> resolution is disabled (all jobs will be activated upon arrival).
> INFO [2019-09-24 10:04:21,536] org.apache.ignite.internal.IgniteKernal:
> Security status [authentication=off, tls/ssl=off]
> WARN [2019-09-24 10:04:21,656]
> org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager:
> DataRegionConfiguration.maxWalArchiveSize instead
> DataRegionConfiguration.walHistorySize would be used for removing old archive
> wal files
> INFO [2019-09-24 10:04:21,666]
> org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager:
> Evict partition permits=9
> INFO [2019-09-24 10:04:21,785]
> org.apache.ignite.internal.processors.odbc.ClientListenerProcessor: Client
> connector processor has started on TCP port 10800
> INFO [2019-09-24 10:04:21,811]
> org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestProtocol:
> Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0
> <http://0.0.0.0/0.0.0.0>, port=11211]
> INFO [2019-09-24 10:04:21,817] org.eclipse.jetty.server.Server:
> jetty-9.0.z-SNAPSHOT
> INFO [2019-09-24 10:04:21,819] org.eclipse.jetty.server.ServerConnector:
> Started ServerConnector@47eb8e95{HTTP/1.1}{0.0.0.0:8080
> <http://0.0.0.0:8080/>}
> INFO [2019-09-24 10:04:21,819]
> org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol:
> Command protocol successfully started [name=Jetty REST, host=/0.0.0.0
> <http://0.0.0.0/>, port=8080]
> INFO [2019-09-24 10:04:21,843] org.apache.ignite.internal.IgniteKernal:
> Non-loopback local IPs: 10.0.31.159, 169.254.1.1, 172.17.0.1, 172.18.0.1
> INFO [2019-09-24 10:04:21,843] org.apache.ignite.internal.IgniteKernal:
> Enabled local MACs: 024254D57DDE, 0242BC90FC25, 0632865AE178, 8AB58EDB020C
> INFO [2019-09-24 10:04:21,863]
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi: Connection check
> threshold is calculated: 60000
> INFO [2019-09-24 10:04:21,867]
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi: Successfully bound to
> TCP port [port=47500, localHost=0.0.0.0/0.0.0.0 <http://0.0.0.0/0.0.0.0>,
> locNodeId=09156c05-f869-4826-814e-1f47fcefeeb4]
> WARN [2019-09-24 10:04:21,869]
> org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder:
> Amazon client configuration is not set (will use default).
> INFO [2019-09-24 10:04:22,520]
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi: TCP discovery accepted
> incoming connection [rmtAddr=/10.0.11.210 <http://10.0.11.210/>,
> rmtPort=57994]
> INFO [2019-09-24 10:04:22,524]
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi: TCP discovery spawning a
> new thread for connection [rmtAddr=/10.0.11.210 <http://10.0.11.210/>,
> rmtPort=57994]
> INFO [2019-09-24 10:04:22,524]
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi: Started serving remote
> node connection [rmtAddr=/10.0.11.210:57994 <http://10.0.11.210:57994/>,
> rmtPort=57994]
> WARN [2019-09-24 10:04:22,525]
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi: Failed to read magic
> header (too few bytes received) [rmtAddr=/10.0.11.210:57994
> <http://10.0.11.210:57994/>, locAddr=/10.0.31.159:47500
> <http://10.0.31.159:47500/>]
> INFO [2019-09-24 10:04:22,525]
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi: Finished serving remote
> node connection [rmtAddr=/10.0.11.210:57994 <http://10.0.11.210:57994/>,
> rmtPort=57994
> INFO [2019-09-24 10:04:22,722]
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi: TCP discovery accepted
> incoming connection [rmtAddr=/10.0.11.210 <http://10.0.11.210/>,
> rmtPort=57996]
> INFO [2019-09-24 10:04:22,722]
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi: TCP discovery spawning a
> new thread for connection [rmtAddr=/10.0.11.210 <http://10.0.11.210/>,
> rmtPort=57996]
> INFO [2019-09-24 10:04:22,722]
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi: Started serving remote
> node connection [rmtAddr=/10.0.11.210:57996 <http://10.0.11.210:57996/>,
> rmtPort=57996]
> WARN [2019-09-24 10:04:22,722]
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi: Failed to read magic
> header (too few bytes received) [rmtAddr=/10.0.11.210:57996
> <http://10.0.11.210:57996/>, locAddr=/10.0.31.159:47500
> <http://10.0.31.159:47500/>]
> INFO [2019-09-24 10:04:22,722]
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi: Finished serving remote
> node connection [rmtAddr=/10.0.11.210:57996 <http://10.0.11.210:57996/>,
> rmtPort=57996
> INFO [2019-09-24 10:04:22,923]
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi: TCP discovery accepted
> incoming connection [rmtAddr=/10.0.11.210 <http://10.0.11.210/>,
> rmtPort=57998]
> INFO [2019-09-24 10:04:22,924]
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi: TCP discovery spawning a
> new thread for connection [rmtAddr=/10.0.11.210 <http://10.0.11.210/>,
> rmtPort=57998]
