[
https://issues.apache.org/jira/browse/HADOOP-14770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174816#comment-16174816
]
Steve Loughran commented on HADOOP-14770:
-----------------------------------------
Good to hear. You should find IO numbers much better too
> S3A http connection in s3a driver not reuse in Spark application
> ----------------------------------------------------------------
>
> Key: HADOOP-14770
> URL: https://issues.apache.org/jira/browse/HADOOP-14770
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.7.3
> Reporter: Yonger
> Assignee: Yonger
> Priority: Minor
>
> I print out connection stats every 2 s when running Spark application against
> s3-compatible storage:
> {code}
> ESTAB 0 0 ::ffff:10.0.2.36:44446
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44454
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44374
> ::ffff:10.0.2.254:80
> ESTAB 159724 0 ::ffff:10.0.2.36:44436
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44448
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44338
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44438
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44414
> ::ffff:10.0.2.254:80
> ESTAB 0 480 ::ffff:10.0.2.36:44450
> ::ffff:10.0.2.254:80 timer:(on,170ms,0)
> ESTAB 0 0 ::ffff:10.0.2.36:44442
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44390
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44326
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44452
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44394
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44444
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44456
> ::ffff:10.0.2.254:80
> ======================
> ESTAB 0 0 ::ffff:10.0.2.36:44508
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44476
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44524
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44374
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44500
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44504
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44512
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44506
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44464
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44518
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44510
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44442
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44526
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44472
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44466
> ::ffff:10.0.2.254:80
> {code}
> the connection in the above of "=" and below were changed all the time. But
> this haven't seen in MR application.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
