Yonger created HADOOP-14770:
-------------------------------
Summary: S3A http connection in s3a driver not reuse in Spark
application
Key: HADOOP-14770
URL: https://issues.apache.org/jira/browse/HADOOP-14770
Project: Hadoop Common
Issue Type: Bug
Reporter: Yonger
I print out connection stats every 2 s when running Spark application against
s3-compatible storage:
ESTAB 0 0 ::ffff:10.0.2.36:44446
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44454
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44374
::ffff:10.0.2.254:80
ESTAB 159724 0 ::ffff:10.0.2.36:44436
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44448
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44338
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44438
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44414
::ffff:10.0.2.254:80
ESTAB 0 480 ::ffff:10.0.2.36:44450
::ffff:10.0.2.254:80 timer:(on,170ms,0)
ESTAB 0 0 ::ffff:10.0.2.36:44442
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44390
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44326
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44452
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44394
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44444
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44456
::ffff:10.0.2.254:80
======================
ESTAB 0 0 ::ffff:10.0.2.36:44508
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44476
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44524
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44374
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44500
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44504
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44512
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44506
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44464
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44518
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44510
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44442
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44526
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44472
::ffff:10.0.2.254:80
ESTAB 0 0 ::ffff:10.0.2.36:44466
::ffff:10.0.2.254:80
the connection in the above of "=" and below were changed all the time. But
this haven't seen in MR application.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
