Hi, Michael: Sorry for late reply. As I raised earlier in the ALTO session, it is hard to believe We have to use the single monitoring infrastructure or measurement tool to measure not only the impact of ALTO but also system performance. They could be two different monitoring infrastructures, one is used to evaluate How alto server affects inter domain traffic and influence the application performance while The other one is used to measure how powerful the alto server is. Sometimes even though the ALTO server is very powerful, however we misuse the alto sever, the alto server May not help improve traffic optimization and user experience. Sometimes measuring the alto sever performance may be equivalent to evaluate how alto server influence inter-domain traffic if no misusing happen.
I would suggest to change the title of section 3.4.4 as follows: s/ Monitoring Infrastructure/ Monitoring Infrastructures Also I am not sure "the number of retransmitted TCP segments" can be used to measure application performance. In the 2nd bullet of section 3.4.2, it says: " For instance, if ALTO guidance achieves traffic localization, one would expect that applications achieve a higher throughput and/or smaller delays to retrieve data. Application- specific performance characteristics (e.g., video or audio quality) can be useful as well. " It seems two sentences are disconnected. You'd better fix this as well. Regards! -Qin -----Original Message----- From: Scharf, Michael (Michael) [mailto:[email protected]] Sent: Wednesday, February 26, 2014 4:53 AM To: Qin Wu Cc: IETF ALTO Subject: RE: Review of draft-ietf-alto-deployments-08#appendix-A Hi, In order to sort out this discussion regarding terminology, I have now ... 1. Reverted the main headline of the section to what it was in -08 and before ("Monitoring ALTO"), 2. Aligned the wording with draft-ietf-alto-protocol-26 Section 16.1.4 and Section 16.2.5, under the assumption that the content and wording therein is WG consensus 3. Added the desired sentence on third parties and trust issues to Section 3.4.1 4. Separated the discussion of "impact" and "performance" into two sections, as suggested in this thread Below is a copy of the updated, pre-10 text. Please let me know if I missed something. Thanks Michael *** draft-ietf-alto-deployments-010pre *** 3.4. Monitoring ALTO 3.4.1. Impact and Observation on Network Operation ALTO presents a new opportunity for managing network traffic by providing additional information to clients. In particular, the deployment of an ALTO Server may shift network traffic patterns, and the potential impact to network operation can be large. An ISP providing ALTO may want to assess the benefits of ALTO as part of the management and operations (cf. [I-D.ietf-alto-protocol]). For instance, the ISP might be interested in understanding whether the provided ALTO maps are effective, and in order to decide whether an adjustment of the ALTO configuration would be useful. Such insight can be obtained from a monitoring infrastructure. An NSP offering ALTO could consider the impact on (or integration with) traffic engineering and the deployment of a monitoring service to observe the effects of ALTO operations. The measurement of impacts can be challenging because ALTO-enabled applications may not provide related information back to the ALTO Service Provider. To construct an effective monitoring infrastructure, the ALTO Service Provider should decide how to monitor the performance of ALTO and identify and deploy data sources to collect data to compute the performance metrics. In certain trusted deployment environments, it may be possible to collect information directly from ALTO clients. It may also be possible to vary or selectively disable ALTO guidance for a portion of ALTO clients either by time, geographical region, or some other criteria to compare the network traffic characteristics with and without ALTO. Monitoring an ALTO service could also be realized by third parties. In this case, insight into ALTO data may require a trust relationship between the monitoring system operator and the network service provider offering an ALTO service. The required monitoring depends on the network infrastructure and the use of ALTO, and an exhaustive description is outside the scope of this document. 3.4.2. Measurement of the Impact ALTO realizes an interface between the network and applications. This implies that an effective monitoring infrastructure may have to deal with both network and application performance metrics. This document does not comprehensively list all performance metrics that could be relevant, nor does it formally specify metrics. The impact of ALTO can be classified regarding a number of different criteria: o Total amount and distribution of traffic: ALTO enables ISPs to influence and localize traffic of applications that use the ALTO service. An ISP may therefore be interested in analyzing the impact on the traffic, i.e., whether network traffic patterns are shifted. For instance, if ALTO shall be used to reduce the inter- domain P2P traffic, it makes sense to evaluate the total amount of inter-domain traffic of an ISP. Then, one possibility is to study how the introduction of ALTO reduces the total inter-domain traffic (inbound and/our outbound). If the ISPs intention is to localize the traffic inside his network, the network-internal traffic distribution will be of interest. Effectiveness of localization can be quantified in different ways, e.g., by the load on core routers and backbone links, or by considering more advanced effects, such as the average number of hops that traffic traverses inside a domain. o Application performance: The objective of ALTO is improve application performance. ALTO can be used by very different types applications, with different communication characteristics and requirements. For instance, if ALTO guidance achieves traffic localization, one would expect that applications achieve a higher throughput and/or smaller delays to retrieve data. Application- specific performance characteristics (e.g., video or audio quality) can be useful as well. In addition, selected statistics from the TCP/IP stack in hosts can be useful, e.g., the number of retransmitted TCP segments. Of potential interest can also be the share of applications or customers that actually use an offered ALTO service, i.e., the adoption of the service. Monitoring statistics can be aggregated, averaged, and normalized in different ways. This document does not mandate specific ways how to calculate metrics. 3.4.3. System and Service Performance A number of interesting parameters can be measured at the ALTO server. [I-D.ietf-alto-protocol] suggests certain ALTO-specific metrics to be monitored: o Requests and responses for each service listed in a Information Directory (total counts and size in bytes). o CPU and memory utilization o ALTO map updates o Number of PIDs o ALTO map sizes (in-memory size, encoded size, number of entries) This data characterizes the workload, the system performance as well as the map data. Obviously, such data will depend on the implementation and the actual deployment of the ALTO service. 3.4.4. Monitoring Infrastructure Understanding the impact of ALTO may require interaction between different systems, operating at different layers. Some information discussed in the preceding sections is only visible to an ISP, while application-level performance can hardly be measured inside the network. It is possible that not all information of potential interest can directly be measured, either because no corresponding monitoring infrastructure or measurement method exists, or because it is not easily accessible. One way to quantify the benefit of deploying ALTO is to measure before and after enabling the ALTO service. In addition to passive monitoring, some data could also be obtained by active measurements, but due to the resulting overhead, the latter should be used with care. Yet, in all monitoring activities an ALTO service provider has to take into account that ALTO Clients are not bound to ALTO Server guidance as ALTO is only one source of information, and any measurement result may thus be biased. Potential sources for monitoring the use of ALTO include: o Network Operations, Administration, and Maintenance (OAM) systems: Many ISPs deploy OAM systems to monitor the network traffic, which may have insight into traffic volumes, network topology, and bandwidth information inside the management area. Data can be obtained by SNMP, Netflow, IP Flow Information Export (IPFIX), syslog, etc. o Applications/clients: Relevant data could be obtained by instrumentation of applications. o ALTO server: If available, log files or other statistics data could be analyzed. o Other application entities: In several use cases, there are other application entities that could provide data as well. For instance, there may be centralized log servers that collect data. In many ALTO use cases some data sources are located within an ISP while some other data is gathered at application level. Correlation of data would require a collaboration agreement between the ISP and an application owner, including agreements of data interchange formats, methods of delivery, etc. In practice, such a collaboration may not be possible in all use cases of ALTO, because the monitoring data can be sensitive, and because the interacting entities may have different priorities. Details of how to build an over-arching monitoring system for evaluating the benefits of ALTO are outside the scope of this memo. _______________________________________________ alto mailing list [email protected] https://www.ietf.org/mailman/listinfo/alto
