Hi Qin,
Sure, we can change the headline of Section 3.4.4 to plural, and remove the
example of TCP retransmissions.
Would the text below address your comment? ([...] is unchanged to my last
e-mail).
Thanks
Michael
*****************************************
[...]
3.4.2. Measurement of the Impact
[...]
o Application performance: The objective of ALTO is improve
application performance. ALTO can be used by very different types
applications, with different communication characteristics and
requirements. For instance, if ALTO guidance achieves traffic
localization, one would expect that applications achieve a higher
throughput and/or smaller delays to retrieve data. If
application-specific performance characteristics (e.g., video or
audio quality) can be monitored, such metrics related to user
experience could also help to analyze the benefit of an ALTO
deployment. If available, selected statistics from the TCP/IP
stack in hosts could be leveraged, too.
[...]
3.4.4. Monitoring Infrastructures
[...]
> -----Original Message-----
> From: Qin Wu [mailto:[email protected]]
> Sent: Tuesday, March 11, 2014 11:28 AM
> To: Scharf, Michael (Michael)
> Cc: IETF ALTO
> Subject: RE: Review of draft-ietf-alto-deployments-08#appendix-A
>
> Hi, Michael:
> Sorry for late reply.
> As I raised earlier in the ALTO session, it is hard to believe
> We have to use the single monitoring infrastructure or measurement tool
> to measure not only the impact of ALTO but also system performance.
> They could be two different monitoring infrastructures, one is used to
> evaluate
> How alto server affects inter domain traffic and influence the
> application performance while
> The other one is used to measure how powerful the alto server is.
> Sometimes even though the ALTO server is very powerful, however we
> misuse the alto sever, the alto server
> May not help improve traffic optimization and user experience.
> Sometimes measuring the alto sever performance may be equivalent to
> evaluate how alto server influence
> inter-domain traffic if no misusing happen.
>
> I would suggest to change the title of section 3.4.4
> as follows:
> s/ Monitoring Infrastructure/ Monitoring Infrastructures
>
> Also I am not sure "the number of retransmitted TCP segments" can be
> used to measure application performance.
>
> In the 2nd bullet of section 3.4.2, it says:
> "
> For instance, if ALTO guidance achieves traffic
> localization, one would expect that applications achieve a higher
> throughput and/or smaller delays to retrieve data. Application-
> specific performance characteristics (e.g., video or audio
> quality) can be useful as well.
> "
> It seems two sentences are disconnected. You'd better fix this as well.
>
>
> Regards!
> -Qin
> -----Original Message-----
> From: Scharf, Michael (Michael) [mailto:michael.scharf@alcatel-
> lucent.com]
> Sent: Wednesday, February 26, 2014 4:53 AM
> To: Qin Wu
> Cc: IETF ALTO
> Subject: RE: Review of draft-ietf-alto-deployments-08#appendix-A
>
> Hi,
>
> In order to sort out this discussion regarding terminology, I have now
> ...
>
> 1. Reverted the main headline of the section to what it was in -08 and
> before ("Monitoring ALTO"),
>
> 2. Aligned the wording with draft-ietf-alto-protocol-26 Section 16.1.4
> and Section 16.2.5, under the assumption that the content and wording
> therein is WG consensus
>
> 3. Added the desired sentence on third parties and trust issues to
> Section 3.4.1
>
> 4. Separated the discussion of "impact" and "performance" into two
> sections, as suggested in this thread
>
> Below is a copy of the updated, pre-10 text.
>
> Please let me know if I missed something.
>
> Thanks
>
> Michael
>
>
> *** draft-ietf-alto-deployments-010pre ***
>
>
> 3.4. Monitoring ALTO
>
> 3.4.1. Impact and Observation on Network Operation
>
> ALTO presents a new opportunity for managing network traffic by
> providing additional information to clients. In particular, the
> deployment of an ALTO Server may shift network traffic patterns, and
> the potential impact to network operation can be large. An ISP
> providing ALTO may want to assess the benefits of ALTO as part of
> the
> management and operations (cf. [I-D.ietf-alto-protocol]). For
> instance, the ISP might be interested in understanding whether the
> provided ALTO maps are effective, and in order to decide whether an
> adjustment of the ALTO configuration would be useful. Such insight
> can be obtained from a monitoring infrastructure. An NSP offering
> ALTO could consider the impact on (or integration with) traffic
> engineering and the deployment of a monitoring service to observe
> the
> effects of ALTO operations. The measurement of impacts can be
> challenging because ALTO-enabled applications may not provide
> related
> information back to the ALTO Service Provider.
>
> To construct an effective monitoring infrastructure, the ALTO
> Service
> Provider should decide how to monitor the performance of ALTO and
> identify and deploy data sources to collect data to compute the
> performance metrics. In certain trusted deployment environments, it
> may be possible to collect information directly from ALTO clients.
> It may also be possible to vary or selectively disable ALTO guidance
> for a portion of ALTO clients either by time, geographical region,
> or
> some other criteria to compare the network traffic characteristics
> with and without ALTO. Monitoring an ALTO service could also be
> realized by third parties. In this case, insight into ALTO data may
> require a trust relationship between the monitoring system operator
> and the network service provider offering an ALTO service.
>
> The required monitoring depends on the network infrastructure and
> the
> use of ALTO, and an exhaustive description is outside the scope of
> this document.
>
> 3.4.2. Measurement of the Impact
>
> ALTO realizes an interface between the network and applications.
> This implies that an effective monitoring infrastructure may have to
> deal with both network and application performance metrics. This
> document does not comprehensively list all performance metrics that
> could be relevant, nor does it formally specify metrics.
>
> The impact of ALTO can be classified regarding a number of different
> criteria:
>
> o Total amount and distribution of traffic: ALTO enables ISPs to
> influence and localize traffic of applications that use the ALTO
> service. An ISP may therefore be interested in analyzing the
> impact on the traffic, i.e., whether network traffic patterns are
> shifted. For instance, if ALTO shall be used to reduce the
> inter-
> domain P2P traffic, it makes sense to evaluate the total amount
> of
> inter-domain traffic of an ISP. Then, one possibility is to
> study
> how the introduction of ALTO reduces the total inter-domain
> traffic (inbound and/our outbound). If the ISPs intention is to
> localize the traffic inside his network, the network-internal
> traffic distribution will be of interest. Effectiveness of
> localization can be quantified in different ways, e.g., by the
> load on core routers and backbone links, or by considering more
> advanced effects, such as the average number of hops that traffic
> traverses inside a domain.
>
> o Application performance: The objective of ALTO is improve
> application performance. ALTO can be used by very different
> types
> applications, with different communication characteristics and
> requirements. For instance, if ALTO guidance achieves traffic
> localization, one would expect that applications achieve a higher
> throughput and/or smaller delays to retrieve data. Application-
> specific performance characteristics (e.g., video or audio
> quality) can be useful as well. In addition, selected statistics
> from the TCP/IP stack in hosts can be useful, e.g., the number of
> retransmitted TCP segments.
>
> Of potential interest can also be the share of applications or
> customers that actually use an offered ALTO service, i.e., the
> adoption of the service.
>
> Monitoring statistics can be aggregated, averaged, and normalized in
> different ways. This document does not mandate specific ways how to
> calculate metrics.
>
> 3.4.3. System and Service Performance
>
> A number of interesting parameters can be measured at the ALTO
> server. [I-D.ietf-alto-protocol] suggests certain ALTO-specific
> metrics to be monitored:
>
> o Requests and responses for each service listed in a Information
> Directory (total counts and size in bytes).
>
> o CPU and memory utilization
>
> o ALTO map updates
>
> o Number of PIDs
>
> o ALTO map sizes (in-memory size, encoded size, number of entries)
>
> This data characterizes the workload, the system performance as well
> as the map data. Obviously, such data will depend on the
> implementation and the actual deployment of the ALTO service.
>
> 3.4.4. Monitoring Infrastructure
>
> Understanding the impact of ALTO may require interaction between
> different systems, operating at different layers. Some information
> discussed in the preceding sections is only visible to an ISP, while
> application-level performance can hardly be measured inside the
> network. It is possible that not all information of potential
> interest can directly be measured, either because no corresponding
> monitoring infrastructure or measurement method exists, or because
> it
> is not easily accessible.
>
> One way to quantify the benefit of deploying ALTO is to measure
> before and after enabling the ALTO service. In addition to passive
> monitoring, some data could also be obtained by active measurements,
> but due to the resulting overhead, the latter should be used with
> care. Yet, in all monitoring activities an ALTO service provider
> has
> to take into account that ALTO Clients are not bound to ALTO Server
> guidance as ALTO is only one source of information, and any
> measurement result may thus be biased.
>
> Potential sources for monitoring the use of ALTO include:
>
> o Network Operations, Administration, and Maintenance (OAM)
> systems:
> Many ISPs deploy OAM systems to monitor the network traffic,
> which
> may have insight into traffic volumes, network topology, and
> bandwidth information inside the management area. Data can be
> obtained by SNMP, Netflow, IP Flow Information Export (IPFIX),
> syslog, etc.
>
> o Applications/clients: Relevant data could be obtained by
> instrumentation of applications.
>
> o ALTO server: If available, log files or other statistics data
> could be analyzed.
>
> o Other application entities: In several use cases, there are other
> application entities that could provide data as well. For
> instance, there may be centralized log servers that collect data.
>
> In many ALTO use cases some data sources are located within an ISP
> while some other data is gathered at application level. Correlation
> of data would require a collaboration agreement between the ISP and
> an application owner, including agreements of data interchange
> formats, methods of delivery, etc. In practice, such a
> collaboration
> may not be possible in all use cases of ALTO, because the monitoring
> data can be sensitive, and because the interacting entities may have
> different priorities. Details of how to build an over-arching
> monitoring system for evaluating the benefits of ALTO are outside
> the
> scope of this memo.
_______________________________________________
alto mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/alto
