Grant Henke created KUDU-2830:
---------------------------------
Summary: Improve RPC traces on RPC timeout
Key: KUDU-2830
URL: https://issues.apache.org/jira/browse/KUDU-2830
Project: Kudu
Issue Type: Improvement
Reporter: Grant Henke
When RPCs are retried repeatedly and then eventually a timeout is caused, the
logged RPC trace includes the entire nested RPC call stack. This is often way
to much to read and usually includes a lot of repeated information. Perhaps we
could use various levels of detail and summarization based on the log level to
make things more useful.
Below is a sample RPC trace:
{noformat}
19/01/22 06:01:56 WARN scheduler.TaskSetManager: Lost task 1125.0 in stage 1.0
(TID 1152, vc1308.halxg.cloudera.com, executor 9): java.lang.RuntimeException:
failed to write 516 rows from DataFrame to Kudu; sample errors: Timed out: can
not complete before timeout: Batch{operations=516,
tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>),
ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write,
tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35,
DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master,
[0ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[60ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [60ms] sending RPC to server
09d6bf7a02124145b43f43cb7a667b3d, [97ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [97ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [119ms] querying master, [119ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[120ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [120ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [160ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [160ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [218ms] querying master, [218ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[218ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [219ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [262ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [262ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [479ms] querying master, [479ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[480ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [480ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [516ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [516ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [719ms] querying master, [719ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[720ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [720ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [758ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [758ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [879ms] querying master, [879ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[879ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [879ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [916ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [916ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [1060ms] querying master, [1060ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[1060ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [1060ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.23% of capacity), [1103ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.23% of capacity), [4480ms] querying master, [4480ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[4480ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [4480ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.34% of capacity), [4524ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.34% of capacity), [7020ms] querying master, [7020ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[7020ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [7020ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.42% of capacity), [7061ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.42% of capacity), [9820ms] querying master, [9820ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[9821ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [9821ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.51% of capacity), [9860ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.51% of capacity), [13100ms] sending RPC to server
09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.63% of capacity), [13138ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.63% of capacity), [16920ms] querying master, [16920ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[16920ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [16920ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.71% of capacity), [16957ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.71% of capacity), [18959ms] querying master, [18959ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[18960ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [18960ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.75% of capacity), [19002ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.75% of capacity), [22439ms] querying master, [22439ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[22440ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [22440ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.82% of capacity), [22476ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.82% of capacity), [24520ms] querying master, [24520ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[24520ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [24520ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.87% of capacity), [24562ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.87% of capacity), [25079ms] querying master, [25079ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[25080ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [25080ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.88% of capacity), [25117ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.88% of capacity), [28739ms] querying master, [28740ms] trace
too long, truncated)}Timed out: can not complete before timeout:
Batch{operations=516, tablet="c31055f8d3f74ee3b52687e7aff5b68a"
[0x0000000300000003, <end>), ignoreAllDuplicateRows=false,
rpc=KuduRpc(method=Write, tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35,
DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master,
[0ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[60ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [60ms] sending RPC to server
09d6bf7a02124145b43f43cb7a667b3d, [97ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [97ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [119ms] querying master, [119ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[120ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [120ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [160ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [160ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [218ms] querying master, [218ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[218ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [219ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [262ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [262ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [479ms] querying master, [479ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[480ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [480ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [516ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [516ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [719ms] querying master, [719ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[720ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [720ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [758ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [758ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [879ms] querying master, [879ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[879ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [879ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [916ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [916ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [1060ms] querying master, [1060ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[1060ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [1060ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.23% of capacity), [1103ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.23% of capacity), [4480ms] querying master, [4480ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[4480ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [4480ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.34% of capacity), [4524ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.34% of capacity), [7020ms] querying master, [7020ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[7020ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [7020ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.42% of capacity), [7061ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.42% of capacity), [9820ms] querying master, [9820ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[9821ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [9821ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.51% of capacity), [9860ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.51% of capacity), [13100ms] sending RPC to server
09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.63% of capacity), [13138ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.63% of capacity), [16920ms] querying master, [16920ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[16920ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [16920ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.71% of capacity), [16957ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.71% of capacity), [18959ms] querying master, [18959ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[18960ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [18960ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.75% of capacity), [19002ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.75% of capacity), [22439ms] querying master, [22439ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[22440ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [22440ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.82% of capacity), [22476ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.82% of capacity), [24520ms] querying master, [24520ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[24520ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [24520ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.87% of capacity), [24562ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.87% of capacity), [25079ms] querying master, [25079ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[25080ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [25080ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.88% of capacity), [25117ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.88% of capacity), [28739ms] querying master, [28740ms] trace
too long, truncated)}Timed out: can not complete before timeout:
Batch{operations=516, tablet="c31055f8d3f74ee3b52687e7aff5b68a"
[0x0000000300000003, <end>), ignoreAllDuplicateRows=false,
rpc=KuduRpc(method=Write, tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35,
DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master,
[0ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[60ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [60ms] sending RPC to server
09d6bf7a02124145b43f43cb7a667b3d, [97ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [97ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [119ms] querying master, [119ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[120ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [120ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [160ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [160ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [218ms] querying master, [218ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[218ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [219ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [262ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [262ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [479ms] querying master, [479ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[480ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [480ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [516ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [516ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [719ms] querying master, [719ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[720ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [720ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [758ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [758ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [879ms] querying master, [879ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[879ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [879ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [916ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [916ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [1060ms] querying master, [1060ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[1060ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [1060ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.23% of capacity), [1103ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.23% of capacity), [4480ms] querying master, [4480ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[4480ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [4480ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.34% of capacity), [4524ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.34% of capacity), [7020ms] querying master, [7020ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[7020ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [7020ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.42% of capacity), [7061ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.42% of capacity), [9820ms] querying master, [9820ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[9821ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [9821ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.51% of capacity), [9860ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.51% of capacity), [13100ms] sending RPC to server
09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.63% of capacity), [13138ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.63% of capacity), [16920ms] querying master, [16920ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[16920ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [16920ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.71% of capacity), [16957ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.71% of capacity), [18959ms] querying master, [18959ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[18960ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [18960ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.75% of capacity), [19002ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.75% of capacity), [22439ms] querying master, [22439ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[22440ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [22440ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.82% of capacity), [22476ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.82% of capacity), [24520ms] querying master, [24520ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[24520ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [24520ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.87% of capacity), [24562ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.87% of capacity), [25079ms] querying master, [25079ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[25080ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [25080ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.88% of capacity), [25117ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.88% of capacity), [28739ms] querying master, [28740ms] trace
too long, truncated)}Timed out: can not complete before timeout:
Batch{operations=516, tablet="c31055f8d3f74ee3b52687e7aff5b68a"
[0x0000000300000003, <end>), ignoreAllDuplicateRows=false,
rpc=KuduRpc(method=Write, tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35,
DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master,
[0ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[60ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [60ms] sending RPC to server
09d6bf7a02124145b43f43cb7a667b3d, [97ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [97ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [119ms] querying master, [119ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[120ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [120ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [160ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [160ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [218ms] querying master, [218ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[218ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [219ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [262ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [262ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [479ms] querying master, [479ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[480ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [480ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [516ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [516ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [719ms] querying master, [719ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[720ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [720ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [758ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [758ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [879ms] querying master, [879ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[879ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [879ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [916ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [916ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [1060ms] querying master, [1060ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[1060ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [1060ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.23% of capacity), [1103ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.23% of capacity), [4480ms] querying master, [4480ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[4480ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [4480ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.34% of capacity), [4524ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.34% of capacity), [7020ms] querying master, [7020ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[7020ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [7020ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.42% of capacity), [7061ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.42% of capacity), [9820ms] querying master, [9820ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[9821ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [9821ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.51% of capacity), [9860ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.51% of capacity), [13100ms] sending RPC to server
09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.63% of capacity), [13138ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.63% of capacity), [16920ms] querying master, [16920ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[16920ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [16920ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.71% of capacity), [16957ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.71% of capacity), [18959ms] querying master, [18959ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[18960ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [18960ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.75% of capacity), [19002ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.75% of capacity), [22439ms] querying master, [22439ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[22440ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [22440ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.82% of capacity), [22476ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.82% of capacity), [24520ms] querying master, [24520ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[24520ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [24520ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.87% of capacity), [24562ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.87% of capacity), [25079ms] querying master, [25079ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[25080ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [25080ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.88% of capacity), [25117ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.88% of capacity), [28739ms] querying master, [28740ms] trace
too long, truncated)}Timed out: can not complete before timeout:
Batch{operations=516, tablet="c31055f8d3f74ee3b52687e7aff5b68a"
[0x0000000300000003, <end>), ignoreAllDuplicateRows=false,
rpc=KuduRpc(method=Write, tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35,
DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master,
[0ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[60ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [60ms] sending RPC to server
09d6bf7a02124145b43f43cb7a667b3d, [97ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [97ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [119ms] querying master, [119ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[120ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [120ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [160ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [160ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [218ms] querying master, [218ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[218ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [219ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [262ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [262ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [479ms] querying master, [479ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[480ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [480ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [516ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [516ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [719ms] querying master, [719ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[720ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [720ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [758ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [758ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [879ms] querying master, [879ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[879ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [879ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [916ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.22% of capacity), [916ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.22% of capacity), [1060ms] querying master, [1060ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[1060ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [1060ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.23% of capacity), [1103ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.23% of capacity), [4480ms] querying master, [4480ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[4480ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [4480ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.34% of capacity), [4524ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.34% of capacity), [7020ms] querying master, [7020ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[7020ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [7020ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.42% of capacity), [7061ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.42% of capacity), [9820ms] querying master, [9820ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[9821ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [9821ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.51% of capacity), [9860ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.51% of capacity), [13100ms] sending RPC to server
09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.63% of capacity), [13138ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.63% of capacity), [16920ms] querying master, [16920ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[16920ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [16920ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.71% of capacity), [16957ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.71% of capacity), [18959ms] querying master, [18959ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[18960ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [18960ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.75% of capacity), [19002ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.75% of capacity), [22439ms] querying master, [22439ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[22440ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [22440ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.82% of capacity), [22476ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.82% of capacity), [24520ms] querying master, [24520ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[24520ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [24520ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.87% of capacity), [24562ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.87% of capacity), [25079ms] querying master, [25079ms] Sub rpc:
GetTableLocations sending RPC to server master-vc1320.halxg.cloudera.com:7051,
[25080ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK, [25080ms] sending RPC to
server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms] received from server
09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
unavailable: Soft memory limit exceeded (at 95.88% of capacity), [25117ms]
delaying RPC due to Service unavailable: Service unavailable: Soft memory limit
exceeded (at 95.88% of capacity), [28739ms] querying master, [28740ms] trace
too long, truncated)}
at
org.apache.kudu.spark.kudu.KuduContext$$anonfun$writeRows$1.apply(KuduContext.scala:311)
at
org.apache.kudu.spark.kudu.KuduContext$$anonfun$writeRows$1.apply(KuduContext.scala:299)
at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
