[
https://issues.apache.org/jira/browse/CASSANDRA-15276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Danko updated CASSANDRA-15276:
------------------------------------
Description:
I'm trying to restore backups created by Opscenter. I restore to fresh cluster,
so it is necessary to use sstableloader. Backups are created from DSE cluster
version 6.7.3, restored to 6.7.4. Restore procedure hangs on large
tables(190GB).
Cluster is tunned and configured to withstand higher load, than normaly.
Maintanance features like nodesync and compactions are throttled to minimum.
All configuration changes are:
* disabled noderepair
* nodesync: rate_in_kb: 64 #originaly 8192
* compaction_throughput_mb_per_sec: 1
* concurrent_compactors: 1
* memtable_allocation_type: offheap_objects
* memtable_space_in_mb: 2048
* max_heapsize_mb: 32786
* streaming_keep_alive_period_in_secs: 72000
* streaming_socket_timeout_in_ms: 86400000 #tested, but dse 6.7.4 won't start
up with this setting
* dse_cassandra_streaming_connections_per_host: 3 #originaly 1
Logs from system.log
{code:java}
INFO [STREAM-INIT-/10.132.16.10:38620] 2019-08-12 14:16:51,722
StreamResultFuture.java:129 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d,
ID#0] Received streaming plan for Bulk Load
INFO [STREAM-IN-/10.132.16.10:38620] 2019-08-12 14:16:51,741
StreamResultFuture.java:194 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d
ID#0] Prepare completed. Receiving 17 files(191.218GiB), sending 0
files(0.000KiB)
ERROR [STREAM-IN-/10.132.16.10:38620] 2019-08-12 14:18:00,640
StreamSession.java:650 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d]
Streaming error occurred on session with peer 10.132.16.10
java.net.SocketException: End-of-stream reached
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:107)
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:318)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
INFO [STREAM-IN-/10.132.16.10:38620] 2019-08-12 14:18:00,648
StreamResultFuture.java:208 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d]
Session with /10.132.16.10 is complete
WARN [STREAM-IN-/10.132.16.10:38620] 2019-08-12 14:18:00,650
StreamResultFuture.java:235 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d]
Stream failed
INFO [STREAM-INIT-/10.132.16.8:44336] 2019-08-12 14:21:48,227
StreamResultFuture.java:122 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7
ID#0] Creating new streaming plan for Bulk Load
INFO [STREAM-INIT-/10.132.16.8:44336] 2019-08-12 14:21:48,229
StreamResultFuture.java:129 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7,
ID#0] Received streaming plan for Bulk Load
INFO [STREAM-INIT-/10.132.16.8:44342] 2019-08-12 14:21:48,230
StreamResultFuture.java:129 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7,
ID#0] Received streaming plan for Bulk Load
INFO [STREAM-IN-/10.132.16.8:44342] 2019-08-12 14:21:48,243
StreamResultFuture.java:194 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7
ID#0] Prepare completed. Receiving 13 files(185.643GiB), sending 0
files(0.000KiB)
INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 14:24:42,740
NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 0B (0B/s), 0% was
inconsistent.
INFO [STREAM-INIT-/10.132.16.9:48802] 2019-08-12 14:27:12,562
StreamResultFuture.java:122 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08
ID#0] Creating new streaming plan for Bulk Load
INFO [STREAM-INIT-/10.132.16.9:48802] 2019-08-12 14:27:12,564
StreamResultFuture.java:129 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08,
ID#0] Received streaming plan for Bulk Load
INFO [STREAM-INIT-/10.132.16.9:48806] 2019-08-12 14:27:12,565
StreamResultFuture.java:129 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08,
ID#0] Received streaming plan for Bulk Load
INFO [STREAM-IN-/10.132.16.9:48806] 2019-08-12 14:27:12,582
StreamResultFuture.java:194 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08
ID#0] Prepare completed. Receiving 15 files(188.633GiB), sending 0
files(0.000KiB)
INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 14:34:42,740
NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 29MB (50kB/s), 0%
was inconsistent.
INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 14:44:42,740
NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 98MB (168kB/s), 0%
was inconsistent.
INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 14:54:42,740
NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 20MB (35kB/s), 0%
was inconsistent.
INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 15:04:42,740
NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 0B (0B/s), 0% was
inconsistent.
ERROR [STREAM-IN-/10.132.16.8:44342] 2019-08-12 15:14:36,634
StreamSession.java:650 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7]
Streaming error occurred on session with peer 10.132.16.8
java.net.SocketException: End-of-stream reached
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:107)
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:318)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
INFO [STREAM-IN-/10.132.16.8:44342] 2019-08-12 15:14:36,640
StreamResultFuture.java:208 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7]
Session with /10.132.16.8 is complete
WARN [STREAM-IN-/10.132.16.8:44342] 2019-08-12 15:14:36,641
StreamResultFuture.java:235 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7]
Stream failed
INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 15:14:42,739
NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 37MB (63kB/s), 0%
was inconsistent.
INFO [GCInspector:1] 2019-08-12 15:21:11,323 GCInspector.java:313 - G1 Young
Generation GC in 201ms. G1 Old Gen: 3516715984 -> 3535248344; G1 Survivor
Space: 394264576 -> 427819008; G1 Eden Space: 20216545280 -> 0
INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 15:24:42,740
NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 59MB (100kB/s), 0%
was inconsistent.
INFO [GCInspector:1] 2019-08-12 15:27:41,343 GCInspector.java:313 - G1 Young
Generation GC in 232ms. G1 Old Gen: 3562767320 -> 3585796048; G1 Survivor
Space: 503316480 -> 545259520; G1 Eden Space: 20107493376 -> 0
INFO [GCInspector:1] 2019-08-12 15:31:55,093 GCInspector.java:313 - G1 Young
Generation GC in 238ms. G1 Old Gen: 3597089752 -> 3617499096; G1 Survivor
Space: 587202560 -> 612368384; G1 Eden Space: 20023607296 -> 0
INFO [GCInspector:1] 2019-08-12 15:34:05,240 GCInspector.java:313 - G1 Young
Generation GC in 234ms. G1 Old Gen: 3617499096 -> 3632501208; G1 Survivor
Space: 612368384 -> 662700032; G1 Eden Space: 19998441472 -> 0
INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 15:34:42,739
NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 24MB (42kB/s), 0%
was inconsistent.
ERROR [STREAM-IN-/10.132.16.9:48806] 2019-08-12 15:35:32,089
StreamSession.java:650 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08]
Streaming error occurred on session with peer 10.132.16.9
java.net.SocketException: End-of-stream reached
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:107)
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:318)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
INFO [STREAM-IN-/10.132.16.9:48806] 2019-08-12 15:35:32,094
StreamResultFuture.java:208 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08]
Session with /10.132.16.9 is complete
WARN [STREAM-IN-/10.132.16.9:48806] 2019-08-12 15:35:32,095
StreamResultFuture.java:235 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08]
Stream failed
{code}
I suspected that sstableloader may ignore parameter
streaming_keep_alive_period_in_secs, because streams seems to consistently
fails after short aproximately 1hour interval.
was:
I'm trying to restore backups created by Opscenter. I restore to fresh cluster,
so it is necessary to use sstableloader. Backups are created from DSE cluster
version 6.7.3, restored to 6.7.4. Restore procedure hangs on large
tables(190GB).
Cluster is tunned and configured to withstand higher load, than normaly.
Maintanance features like nodesync and compactions are throttled to minimum.
All configuration changes are:
*
disabled noderepair
*
nodesync: rate_in_kb: 64 #originaly 8192
*
compaction_throughput_mb_per_sec: 1
*
concurrent_compactors: 1
*
memtable_allocation_type: offheap_objects
*
memtable_space_in_mb: 2048
*
max_heapsize_mb: 32786
*
streaming_keep_alive_period_in_secs: 72000
*
streaming_socket_timeout_in_ms: 86400000 #tested, but dse 6.7.4 won't start up
with this setting
*
dse_cassandra_streaming_connections_per_host: 3 #originaly 1
Logs from system.log
{code:java}
INFO [STREAM-INIT-/10.132.16.10:38620] 2019-08-12 14:16:51,722
StreamResultFuture.java:129 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d,
ID#0] Received streaming plan for Bulk Load
INFO [STREAM-IN-/10.132.16.10:38620] 2019-08-12 14:16:51,741
StreamResultFuture.java:194 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d
ID#0] Prepare completed. Receiving 17 files(191.218GiB), sending 0
files(0.000KiB)
ERROR [STREAM-IN-/10.132.16.10:38620] 2019-08-12 14:18:00,640
StreamSession.java:650 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d]
Streaming error occurred on session with peer 10.132.16.10
java.net.SocketException: End-of-stream reached
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:107)
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:318)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
INFO [STREAM-IN-/10.132.16.10:38620] 2019-08-12 14:18:00,648
StreamResultFuture.java:208 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d]
Session with /10.132.16.10 is complete
WARN [STREAM-IN-/10.132.16.10:38620] 2019-08-12 14:18:00,650
StreamResultFuture.java:235 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d]
Stream failed
INFO [STREAM-INIT-/10.132.16.8:44336] 2019-08-12 14:21:48,227
StreamResultFuture.java:122 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7
ID#0] Creating new streaming plan for Bulk Load
INFO [STREAM-INIT-/10.132.16.8:44336] 2019-08-12 14:21:48,229
StreamResultFuture.java:129 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7,
ID#0] Received streaming plan for Bulk Load
INFO [STREAM-INIT-/10.132.16.8:44342] 2019-08-12 14:21:48,230
StreamResultFuture.java:129 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7,
ID#0] Received streaming plan for Bulk Load
INFO [STREAM-IN-/10.132.16.8:44342] 2019-08-12 14:21:48,243
StreamResultFuture.java:194 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7
ID#0] Prepare completed. Receiving 13 files(185.643GiB), sending 0
files(0.000KiB)
INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 14:24:42,740
NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 0B (0B/s), 0% was
inconsistent.
INFO [STREAM-INIT-/10.132.16.9:48802] 2019-08-12 14:27:12,562
StreamResultFuture.java:122 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08
ID#0] Creating new streaming plan for Bulk Load
INFO [STREAM-INIT-/10.132.16.9:48802] 2019-08-12 14:27:12,564
StreamResultFuture.java:129 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08,
ID#0] Received streaming plan for Bulk Load
INFO [STREAM-INIT-/10.132.16.9:48806] 2019-08-12 14:27:12,565
StreamResultFuture.java:129 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08,
ID#0] Received streaming plan for Bulk Load
INFO [STREAM-IN-/10.132.16.9:48806] 2019-08-12 14:27:12,582
StreamResultFuture.java:194 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08
ID#0] Prepare completed. Receiving 15 files(188.633GiB), sending 0
files(0.000KiB)
INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 14:34:42,740
NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 29MB (50kB/s), 0%
was inconsistent.
INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 14:44:42,740
NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 98MB (168kB/s), 0%
was inconsistent.
INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 14:54:42,740
NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 20MB (35kB/s), 0%
was inconsistent.
INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 15:04:42,740
NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 0B (0B/s), 0% was
inconsistent.
ERROR [STREAM-IN-/10.132.16.8:44342] 2019-08-12 15:14:36,634
StreamSession.java:650 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7]
Streaming error occurred on session with peer 10.132.16.8
java.net.SocketException: End-of-stream reached
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:107)
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:318)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
INFO [STREAM-IN-/10.132.16.8:44342] 2019-08-12 15:14:36,640
StreamResultFuture.java:208 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7]
Session with /10.132.16.8 is complete
WARN [STREAM-IN-/10.132.16.8:44342] 2019-08-12 15:14:36,641
StreamResultFuture.java:235 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7]
Stream failed
INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 15:14:42,739
NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 37MB (63kB/s), 0%
was inconsistent.
INFO [GCInspector:1] 2019-08-12 15:21:11,323 GCInspector.java:313 - G1 Young
Generation GC in 201ms. G1 Old Gen: 3516715984 -> 3535248344; G1 Survivor
Space: 394264576 -> 427819008; G1 Eden Space: 20216545280 -> 0
INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 15:24:42,740
NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 59MB (100kB/s), 0%
was inconsistent.
INFO [GCInspector:1] 2019-08-12 15:27:41,343 GCInspector.java:313 - G1 Young
Generation GC in 232ms. G1 Old Gen: 3562767320 -> 3585796048; G1 Survivor
Space: 503316480 -> 545259520; G1 Eden Space: 20107493376 -> 0
INFO [GCInspector:1] 2019-08-12 15:31:55,093 GCInspector.java:313 - G1 Young
Generation GC in 238ms. G1 Old Gen: 3597089752 -> 3617499096; G1 Survivor
Space: 587202560 -> 612368384; G1 Eden Space: 20023607296 -> 0
INFO [GCInspector:1] 2019-08-12 15:34:05,240 GCInspector.java:313 - G1 Young
Generation GC in 234ms. G1 Old Gen: 3617499096 -> 3632501208; G1 Survivor
Space: 612368384 -> 662700032; G1 Eden Space: 19998441472 -> 0
INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 15:34:42,739
NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 24MB (42kB/s), 0%
was inconsistent.
ERROR [STREAM-IN-/10.132.16.9:48806] 2019-08-12 15:35:32,089
StreamSession.java:650 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08]
Streaming error occurred on session with peer 10.132.16.9
java.net.SocketException: End-of-stream reached
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:107)
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:318)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
INFO [STREAM-IN-/10.132.16.9:48806] 2019-08-12 15:35:32,094
StreamResultFuture.java:208 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08]
Session with /10.132.16.9 is complete
WARN [STREAM-IN-/10.132.16.9:48806] 2019-08-12 15:35:32,095
StreamResultFuture.java:235 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08]
Stream failed
{code}
I suspected that sstableloader may ignore parameter
streaming_keep_alive_period_in_secs, because streams seems to consistently
fails after short aproximately 1hour interval.
> SSTableLoader failes if loading large table
> -------------------------------------------
>
> Key: CASSANDRA-15276
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15276
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Tools
> Reporter: Roman Danko
> Priority: Normal
>
> I'm trying to restore backups created by Opscenter. I restore to fresh
> cluster, so it is necessary to use sstableloader. Backups are created from
> DSE cluster version 6.7.3, restored to 6.7.4. Restore procedure hangs on
> large tables(190GB).
> Cluster is tunned and configured to withstand higher load, than normaly.
> Maintanance features like nodesync and compactions are throttled to minimum.
> All configuration changes are:
> * disabled noderepair
> * nodesync: rate_in_kb: 64 #originaly 8192
> * compaction_throughput_mb_per_sec: 1
> * concurrent_compactors: 1
> * memtable_allocation_type: offheap_objects
> * memtable_space_in_mb: 2048
> * max_heapsize_mb: 32786
> * streaming_keep_alive_period_in_secs: 72000
> * streaming_socket_timeout_in_ms: 86400000 #tested, but dse 6.7.4 won't
> start up with this setting
> * dse_cassandra_streaming_connections_per_host: 3 #originaly 1
> Logs from system.log
>
> {code:java}
> INFO [STREAM-INIT-/10.132.16.10:38620] 2019-08-12 14:16:51,722
> StreamResultFuture.java:129 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d,
> ID#0] Received streaming plan for Bulk Load
> INFO [STREAM-IN-/10.132.16.10:38620] 2019-08-12 14:16:51,741
> StreamResultFuture.java:194 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d
> ID#0] Prepare completed. Receiving 17 files(191.218GiB), sending 0
> files(0.000KiB)
> ERROR [STREAM-IN-/10.132.16.10:38620] 2019-08-12 14:18:00,640
> StreamSession.java:650 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d]
> Streaming error occurred on session with peer 10.132.16.10
> java.net.SocketException: End-of-stream reached
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:107)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:318)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> INFO [STREAM-IN-/10.132.16.10:38620] 2019-08-12 14:18:00,648
> StreamResultFuture.java:208 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d]
> Session with /10.132.16.10 is complete
> WARN [STREAM-IN-/10.132.16.10:38620] 2019-08-12 14:18:00,650
> StreamResultFuture.java:235 - [Stream #ccaadb30-bd0b-11e9-b05b-75f1a4f2840d]
> Stream failed
> INFO [STREAM-INIT-/10.132.16.8:44336] 2019-08-12 14:21:48,227
> StreamResultFuture.java:122 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7
> ID#0] Creating new streaming plan for Bulk Load
> INFO [STREAM-INIT-/10.132.16.8:44336] 2019-08-12 14:21:48,229
> StreamResultFuture.java:129 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7,
> ID#0] Received streaming plan for Bulk Load
> INFO [STREAM-INIT-/10.132.16.8:44342] 2019-08-12 14:21:48,230
> StreamResultFuture.java:129 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7,
> ID#0] Received streaming plan for Bulk Load
> INFO [STREAM-IN-/10.132.16.8:44342] 2019-08-12 14:21:48,243
> StreamResultFuture.java:194 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7
> ID#0] Prepare completed. Receiving 13 files(185.643GiB), sending 0
> files(0.000KiB)
> INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 14:24:42,740
> NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 0B (0B/s), 0% was
> inconsistent.
> INFO [STREAM-INIT-/10.132.16.9:48802] 2019-08-12 14:27:12,562
> StreamResultFuture.java:122 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08
> ID#0] Creating new streaming plan for Bulk Load
> INFO [STREAM-INIT-/10.132.16.9:48802] 2019-08-12 14:27:12,564
> StreamResultFuture.java:129 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08,
> ID#0] Received streaming plan for Bulk Load
> INFO [STREAM-INIT-/10.132.16.9:48806] 2019-08-12 14:27:12,565
> StreamResultFuture.java:129 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08,
> ID#0] Received streaming plan for Bulk Load
> INFO [STREAM-IN-/10.132.16.9:48806] 2019-08-12 14:27:12,582
> StreamResultFuture.java:194 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08
> ID#0] Prepare completed. Receiving 15 files(188.633GiB), sending 0
> files(0.000KiB)
> INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 14:34:42,740
> NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 29MB (50kB/s), 0%
> was inconsistent.
> INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 14:44:42,740
> NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 98MB (168kB/s), 0%
> was inconsistent.
> INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 14:54:42,740
> NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 20MB (35kB/s), 0%
> was inconsistent.
> INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 15:04:42,740
> NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 0B (0B/s), 0% was
> inconsistent.
> ERROR [STREAM-IN-/10.132.16.8:44342] 2019-08-12 15:14:36,634
> StreamSession.java:650 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7]
> Streaming error occurred on session with peer 10.132.16.8
> java.net.SocketException: End-of-stream reached
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:107)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:318)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> INFO [STREAM-IN-/10.132.16.8:44342] 2019-08-12 15:14:36,640
> StreamResultFuture.java:208 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7]
> Session with /10.132.16.8 is complete
> WARN [STREAM-IN-/10.132.16.8:44342] 2019-08-12 15:14:36,641
> StreamResultFuture.java:235 - [Stream #7dfbc520-bd0c-11e9-9cf8-d3315f3fffd7]
> Stream failed
> INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 15:14:42,739
> NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 37MB (63kB/s), 0%
> was inconsistent.
> INFO [GCInspector:1] 2019-08-12 15:21:11,323 GCInspector.java:313 - G1
> Young Generation GC in 201ms. G1 Old Gen: 3516715984 -> 3535248344; G1
> Survivor Space: 394264576 -> 427819008; G1 Eden Space: 20216545280 -> 0
> INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 15:24:42,740
> NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 59MB (100kB/s), 0%
> was inconsistent.
> INFO [GCInspector:1] 2019-08-12 15:27:41,343 GCInspector.java:313 - G1
> Young Generation GC in 232ms. G1 Old Gen: 3562767320 -> 3585796048; G1
> Survivor Space: 503316480 -> 545259520; G1 Eden Space: 20107493376 -> 0
> INFO [GCInspector:1] 2019-08-12 15:31:55,093 GCInspector.java:313 - G1
> Young Generation GC in 238ms. G1 Old Gen: 3597089752 -> 3617499096; G1
> Survivor Space: 587202560 -> 612368384; G1 Eden Space: 20023607296 -> 0
> INFO [GCInspector:1] 2019-08-12 15:34:05,240 GCInspector.java:313 - G1
> Young Generation GC in 234ms. G1 Old Gen: 3617499096 -> 3632501208; G1
> Survivor Space: 612368384 -> 662700032; G1 Eden Space: 19998441472 -> 0
> INFO [NodeSyncMaintenanceTasks:1] 2019-08-12 15:34:42,739
> NodeSyncMaintenanceTasks.java:172 - In last 10m: validated 24MB (42kB/s), 0%
> was inconsistent.
> ERROR [STREAM-IN-/10.132.16.9:48806] 2019-08-12 15:35:32,089
> StreamSession.java:650 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08]
> Streaming error occurred on session with peer 10.132.16.9
> java.net.SocketException: End-of-stream reached
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:107)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:318)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> INFO [STREAM-IN-/10.132.16.9:48806] 2019-08-12 15:35:32,094
> StreamResultFuture.java:208 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08]
> Session with /10.132.16.9 is complete
> WARN [STREAM-IN-/10.132.16.9:48806] 2019-08-12 15:35:32,095
> StreamResultFuture.java:235 - [Stream #3f501c30-bd0d-11e9-9641-6d81d6b3fb08]
> Stream failed
> {code}
>
> I suspected that sstableloader may ignore parameter
> streaming_keep_alive_period_in_secs, because streams seems to consistently
> fails after short aproximately 1hour interval.
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
