[ https://issues.apache.org/jira/browse/HADOOP-14770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-14770: ------------------------------------ Description: I print out connection stats every 2 s when running Spark application against s3-compatible storage: {code} ESTAB 0 0 ::ffff:10.0.2.36:44446 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44454 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44374 ::ffff:10.0.2.254:80 ESTAB 159724 0 ::ffff:10.0.2.36:44436 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44448 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44338 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44438 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44414 ::ffff:10.0.2.254:80 ESTAB 0 480 ::ffff:10.0.2.36:44450 ::ffff:10.0.2.254:80 timer:(on,170ms,0) ESTAB 0 0 ::ffff:10.0.2.36:44442 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44390 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44326 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44452 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44394 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44444 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44456 ::ffff:10.0.2.254:80 ====================== ESTAB 0 0 ::ffff:10.0.2.36:44508 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44476 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44524 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44374 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44500 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44504 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44512 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44506 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44464 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44518 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44510 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44442 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44526 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44472 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44466 ::ffff:10.0.2.254:80 {code} the connection in the above of "=" and below were changed all the time. But this haven't seen in MR application. was: I print out connection stats every 2 s when running Spark application against s3-compatible storage: ESTAB 0 0 ::ffff:10.0.2.36:44446 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44454 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44374 ::ffff:10.0.2.254:80 ESTAB 159724 0 ::ffff:10.0.2.36:44436 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44448 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44338 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44438 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44414 ::ffff:10.0.2.254:80 ESTAB 0 480 ::ffff:10.0.2.36:44450 ::ffff:10.0.2.254:80 timer:(on,170ms,0) ESTAB 0 0 ::ffff:10.0.2.36:44442 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44390 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44326 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44452 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44394 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44444 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44456 ::ffff:10.0.2.254:80 ====================== ESTAB 0 0 ::ffff:10.0.2.36:44508 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44476 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44524 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44374 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44500 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44504 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44512 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44506 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44464 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44518 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44510 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44442 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44526 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44472 ::ffff:10.0.2.254:80 ESTAB 0 0 ::ffff:10.0.2.36:44466 ::ffff:10.0.2.254:80 the connection in the above of "=" and below were changed all the time. But this haven't seen in MR application. > S3A http connection in s3a driver not reuse in Spark application > ---------------------------------------------------------------- > > Key: HADOOP-14770 > URL: https://issues.apache.org/jira/browse/HADOOP-14770 > Project: Hadoop Common > Issue Type: Bug > Reporter: Yonger > Assignee: Yonger > > I print out connection stats every 2 s when running Spark application against > s3-compatible storage: > {code} > ESTAB 0 0 ::ffff:10.0.2.36:44446 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44454 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44374 > ::ffff:10.0.2.254:80 > ESTAB 159724 0 ::ffff:10.0.2.36:44436 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44448 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44338 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44438 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44414 > ::ffff:10.0.2.254:80 > ESTAB 0 480 ::ffff:10.0.2.36:44450 > ::ffff:10.0.2.254:80 timer:(on,170ms,0) > ESTAB 0 0 ::ffff:10.0.2.36:44442 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44390 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44326 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44452 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44394 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44444 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44456 > ::ffff:10.0.2.254:80 > ====================== > ESTAB 0 0 ::ffff:10.0.2.36:44508 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44476 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44524 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44374 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44500 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44504 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44512 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44506 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44464 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44518 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44510 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44442 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44526 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44472 > ::ffff:10.0.2.254:80 > ESTAB 0 0 ::ffff:10.0.2.36:44466 > ::ffff:10.0.2.254:80 > {code} > the connection in the above of "=" and below were changed all the time. But > this haven't seen in MR application. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org