[ https://issues.apache.org/jira/browse/CASSANDRA-15276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922516#comment-16922516 ]
Alex Efimov commented on CASSANDRA-15276: ----------------------------------------- I can confirm the issue. And it happens on quite a small amount of data (50 Mb in my case.) {code:java} progress: [/172.0.30.173]0:1/2 6 % total: 6% 228.754KiB/s (avg: 228.754KiB/s) progress: [/172.0.30.173]0:1/2 26 % total: 26% 531.535KiB/s (avg: 397.567KiB/s) progress: [/172.0.30.173]0:1/2 46 % total: 46% 426.074KiB/s (avg: 409.262KiB/s) ERROR 13:56:16,649 [Stream #a10affb0-cf1b-11e9-b329-0f37ec112982] Streaming error occurred on session with peer 172.0.30.173 java.net.SocketException: End-of-stream reached at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:71) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:311) ~[apache-cassandra-3.11.4.jar:3.11.4] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212] progress: [/172.0.30.173]0:1/2 66 % total: 66% 462.619KiB/s (avg: 423.893KiB/s) progress: [/172.0.30.173]0:1/2 85 % total: 85% 514.001KiB/s (avg: 441.729KiB/s) progress: [/172.0.30.173]0:2/2 100% total: 100% 451.664KiB/s (avg: 443.107KiB/s) progress: [/172.0.30.173]0:2/2 100% total: 100% 0.000KiB/s (avg: 443.098KiB/s) WARN 13:57:05,525 [Stream #a10affb0-cf1b-11e9-b329-0f37ec112982] Stream failed Streaming to the following hosts failed: [/172.0.30.173] java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:98) at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48) Caused by: org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88) at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215) at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191) at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481) at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:571) at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:324) at java.lang.Thread.run(Thread.java:748) Exception in thread "main" org.apache.cassandra.tools.BulkLoadException: java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:114) at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48) Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:98) ... 1 more Caused by: org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88) at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215) at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191) at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481) at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:571) at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:324) at java.lang.Thread.run(Thread.java:748) {code} > SSTableLoader failes if loading large table > ------------------------------------------- > > Key: CASSANDRA-15276 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15276 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Tools > Reporter: Roman Danko > Priority: Normal > > I'm trying to restore backups created by Opscenter. I restore to fresh > cluster, so it is necessary to use sstableloader. Backups are created from > DSE cluster version 6.7.3, restored to 6.7.4. Restore procedure hangs on > large tables(190GB). > Cluster is tunned and configured to withstand higher load, than normaly. > Maintanance features like nodesync and compactions are throttled to minimum. > All configuration changes are: > * disabled noderepair > * nodesync: rate_in_kb: 64 #originaly 8192 > * compaction_throughput_mb_per_sec: 1 > * concurrent_compactors: 1 > * memtable_allocation_type: offheap_objects > * memtable_space_in_mb: 2048 > * max_heapsize_mb: 32786 > * streaming_keep_alive_period_in_secs: 72000 > * streaming_socket_timeout_in_ms: 86400000 #tested, but dse 6.7.4 won't > start up with this setting > * dse_cassandra_streaming_connections_per_host: 3 #originaly 1 > Logs from system.log > > {code:java} > INFO [STREAM-INIT-/10.132.16.10:38620] 2019-08-12 14:16:51,722 > StreamResultFuture.java:129 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d, > ID#0] Received streaming plan for Bulk Load > INFO [STREAM-IN-/10.132.16.10:38620] 2019-08-12 14:16:51,741 > StreamResultFuture.java:194 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d > ID#0] Prepare completed. Receiving 17 files(191.218GiB), sending 0 > files(0.000KiB) > ERROR [STREAM-IN-/10.132.16.10:38620] 2019-08-12 14:18:00,640 > StreamSession.java:650 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d] > Streaming error occurred on session with peer 10.132.16.10 > java.net.SocketException: End-of-stream reached > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:107) > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:318) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.lang.Thread.run(Thread.java:748) > INFO [STREAM-IN-/10.132.16.10:38620] 2019-08-12 14:18:00,648 > StreamResultFuture.java:208 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d] > Session with /10.132.16.10 is complete > WARN [STREAM-IN-/10.132.16.10:38620] 2019-08-12 14:18:00,650 > StreamResultFuture.java:235 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d] > Stream failed > INFO [STREAM-INIT-/10.132.16.8:44336] 2019-08-12 14:21:48,227 > StreamResultFuture.java:122 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7 > ID#0] Creating new streaming plan for Bulk Load > INFO [STREAM-INIT-/10.132.16.8:44336] 2019-08-12 14:21:48,229 > StreamResultFuture.java:129 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7, > ID#0] Received streaming plan for Bulk Load > INFO [STREAM-INIT-/10.132.16.8:44342] 2019-08-12 14:21:48,230 > StreamResultFuture.java:129 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7, > ID#0] Received streaming plan for Bulk Load > INFO [STREAM-IN-/10.132.16.8:44342] 2019-08-12 14:21:48,243 > StreamResultFuture.java:194 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7 > ID#0] Prepare completed. Receiving 13 files(185.643GiB), sending 0 > files(0.000KiB) > INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 14:24:42,740 > NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 0B (0B/s), 0% was > inconsistent. > INFO [STREAM-INIT-/10.132.16.9:48802] 2019-08-12 14:27:12,562 > StreamResultFuture.java:122 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08 > ID#0] Creating new streaming plan for Bulk Load > INFO [STREAM-INIT-/10.132.16.9:48802] 2019-08-12 14:27:12,564 > StreamResultFuture.java:129 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08, > ID#0] Received streaming plan for Bulk Load > INFO [STREAM-INIT-/10.132.16.9:48806] 2019-08-12 14:27:12,565 > StreamResultFuture.java:129 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08, > ID#0] Received streaming plan for Bulk Load > INFO [STREAM-IN-/10.132.16.9:48806] 2019-08-12 14:27:12,582 > StreamResultFuture.java:194 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08 > ID#0] Prepare completed. Receiving 15 files(188.633GiB), sending 0 > files(0.000KiB) > INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 14:34:42,740 > NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 29MB (50kB/s), 0% > was inconsistent. > INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 14:44:42,740 > NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 98MB (168kB/s), 0% > was inconsistent. > INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 14:54:42,740 > NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 20MB (35kB/s), 0% > was inconsistent. > INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 15:04:42,740 > NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 0B (0B/s), 0% was > inconsistent. > ERROR [STREAM-IN-/10.132.16.8:44342] 2019-08-12 15:14:36,634 > StreamSession.java:650 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7] > Streaming error occurred on session with peer 10.132.16.8 > java.net.SocketException: End-of-stream reached > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:107) > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:318) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.lang.Thread.run(Thread.java:748) > INFO [STREAM-IN-/10.132.16.8:44342] 2019-08-12 15:14:36,640 > StreamResultFuture.java:208 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7] > Session with /10.132.16.8 is complete > WARN [STREAM-IN-/10.132.16.8:44342] 2019-08-12 15:14:36,641 > StreamResultFuture.java:235 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7] > Stream failed > INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 15:14:42,739 > NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 37MB (63kB/s), 0% > was inconsistent. > INFO [GCInspector:1] 2019-08-12 15:21:11,323 GCInspector.java:313 - G1 > Young Generation GC in 201ms. G1 Old Gen: 3516715984 -> 3535248344; G1 > Survivor Space: 394264576 -> 427819008; G1 Eden Space: 20216545280 -> 0 > INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 15:24:42,740 > NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 59MB (100kB/s), 0% > was inconsistent. > INFO [GCInspector:1] 2019-08-12 15:27:41,343 GCInspector.java:313 - G1 > Young Generation GC in 232ms. G1 Old Gen: 3562767320 -> 3585796048; G1 > Survivor Space: 503316480 -> 545259520; G1 Eden Space: 20107493376 -> 0 > INFO [GCInspector:1] 2019-08-12 15:31:55,093 GCInspector.java:313 - G1 > Young Generation GC in 238ms. G1 Old Gen: 3597089752 -> 3617499096; G1 > Survivor Space: 587202560 -> 612368384; G1 Eden Space: 20023607296 -> 0 > INFO [GCInspector:1] 2019-08-12 15:34:05,240 GCInspector.java:313 - G1 > Young Generation GC in 234ms. G1 Old Gen: 3617499096 -> 3632501208; G1 > Survivor Space: 612368384 -> 662700032; G1 Eden Space: 19998441472 -> 0 > INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 15:34:42,739 > NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 24MB (42kB/s), 0% > was inconsistent. > ERROR [STREAM-IN-/10.132.16.9:48806] 2019-08-12 15:35:32,089 > StreamSession.java:650 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08] > Streaming error occurred on session with peer 10.132.16.9 > java.net.SocketException: End-of-stream reached > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:107) > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:318) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.lang.Thread.run(Thread.java:748) > INFO [STREAM-IN-/10.132.16.9:48806] 2019-08-12 15:35:32,094 > StreamResultFuture.java:208 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08] > Session with /10.132.16.9 is complete > WARN [STREAM-IN-/10.132.16.9:48806] 2019-08-12 15:35:32,095 > StreamResultFuture.java:235 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08] > Stream failed > {code} > > I suspected that sstableloader may ignore parameter > streaming_keep_alive_period_in_secs, because streams seems to consistently > fails after short aproximately 1hour interval. > -- This message was sent by Atlassian Jira (v8.3.2#803003) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org