[ https://issues.apache.org/jira/browse/HADOOP-14770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yonger resolved HADOOP-14770. ----------------------------- Resolution: Duplicate Thanks Steve. when we apply the read input policy with random in our workload after upgrade to Hadoop 2.8.1, it works as my expect, connections not destroyed every time but reused. > S3A http connection in s3a driver not reuse in Spark application > ---------------------------------------------------------------- > > Key: HADOOP-14770 > URL: https://issues.apache.org/jira/browse/HADOOP-14770 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 > Affects Versions: 2.7.3 > Reporter: Yonger > Assignee: Yonger > Priority: Minor > > I print out connection stats every 2 s when running Spark application against > s3-compatible storage: > {code} > ESTAB 0 0 ::ffff:10.0.2.36:44446 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44454 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44374 > ::ffff:10.0.2.254:80 > ESTAB 159724 0 ::ffff:10.0.2.36:44436 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44448 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44338 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44438 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44414 > ::ffff:10.0.2.254:80 > ESTAB 0 480 ::ffff:10.0.2.36:44450 > ::ffff:10.0.2.254:80 timer:(on,170ms,0) > ESTAB 0 0 ::ffff:10.0.2.36:44442 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44390 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44326 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44452 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44394 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44444 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44456 > ::ffff:10.0.2.254:80 > ====================== > ESTAB 0 0 ::ffff:10.0.2.36:44508 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44476 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44524 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44374 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44500 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44504 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44512 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44506 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44464 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44518 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44510 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44442 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44526 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44472 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44466 > ::ffff:10.0.2.254:80 > {code} > the connection in the above of "=" and below were changed all the time. But > this haven't seen in MR application. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org