That's great to hear Richard.
And great thanks to the team.
http: we should see that a modern version of http gets into components of
XrootD....
Best regards
Harvey
On 7/24/2022 6:54 PM, Y. Richard Yang wrote:
Hi Mihai, Steve, Mario, all,
This is an update on the IETF hackathon which just finished today
(July 24) in Philadelphia, PA. Mahdi presented the results to the
whole hackathon audience at slightly over 2 pm on behalf of the team.
The initial integration of ALTO and FTS and the zero-order algorithm
are fully functional (at least in the concurrent setting).
This excellent progress is the result of hard work that takes a
village; Jensen stayed up 24 hours non-stop until 2 pm to finish up
the FTS production code integration framework, Mahdi was sleeping only
4 hours per day in consecutive days to implement the full zero-order
gradient algorithm, Kai and his team made great progress on
implementing openalto with both standard features (path vector) and
next step features (FCS), and the team worked under the great
organization of Jordi. We plan to do some more demos and socials
tomorrow (Monday) at the IETF Happy Hour to try to get more IETF
feedback. The working slides are at:
https://docs.google.com/presentation/d/1CcCMvZ_SeWLRAN32Qz8RFcHrF8spAu_6/edit?usp=sharing&ouid=110882839032059047983&rtpof=true&sd=true
<https://docs.google.com/presentation/d/1CcCMvZ_SeWLRAN32Qz8RFcHrF8spAu_6/edit?usp=sharing&ouid=110882839032059047983&rtpof=true&sd=true>
As soon as the IETF this week is over, we will schedule an overall
group meeting to go over the technical details, and plan for the next
steps. I consider HTTP a great example of a hugely successful
collaboration between the networking community (IETF) and the data
sciences (CERN). Let's use this project to promote collaboration
between these two communities again.
Cheers,
Richard for the team
On Thu, Jul 21, 2022 at 4:08 AM Mihai Patrascoiu
<[email protected]> wrote:
Hello Richard,
I'll summarize my answers from the other thread.
Unfortunately, due to other arrangements, we won't be able to
attend the hackathon this weekend.
However, we are very excited about the proposed objectives. We
already established a way for the FTS Optimizer zero-order
algorithm to make it's way into production. Once the feature is
ready, with the help of Mario, we can organize a large-scale test
and compare the results we get in production with the simulated
results.
On further ALTO and FTS integration, one aspect I see particularly
promising is:
1. Use ALTO information to assess the network capability of a
given RSE
2. Set a maximum limit for how much network percentage FTS can
occupy. In a feedback loop via ALTO, FTS would find whether it may
increase or should decrease traffic involving that RSE
This maps directly to what we see in production: FTS can hit some
sites too hard.
Any extra knowledge FTS gets on this aspect would allow it to
improve.
Best Regards,
Mihai for the FTS team
On 20/07/2022 03:58 Y. Richard Yang <[email protected]> wrote:
Hi all,
During the weekly ALTO meeting today, we discussed the coming
hackathon and Qin suggested that we send an update to the mailing
list on the FTS+ALTO project, and here is the update:
- In short, if you want to work on ALTO integration
with production, open projects, this can be a wonderful project
to work on: it may help both ALTO (in terms of its crucial
deployment mandate) and the largest data-intensive science
projects at CERN.
Specifically:
- For those who have not tracked the hackathon, as Jordi said, it
is a continuation of the 113 Hackathon. In particular, the 113
Hackathon focused on the integration with Rucio, which is a
wonderfully designed tool used widely in CERN and some
other projects for data movement of a large amount of data (PB ->
EB). In 113, the hackathon modified the manual workflow, which is
the workflow to select the source to download a dataset when a
client issues a download command. The other workflow of Rucio is
the automatic workflow, which selects the sources and
destinations to realize user-specified replication rules. In our
original plan, 114 would focus on the automatic workflow.
However, the team realized that Rucio is built on top of FTS,
which is the main tool used at CERN to schedule which transfer is
scheduled at what time, where Rucio is at a higher layer
providing the transfers for FTS to schedule.
- The 114 hackathon includes 2 objectives: (1) introducing
resource control to FTS, and (2) evaluating an alternative design
of FTS core: the FTS Optimizer. Hence, the work includes 3
components, integrating the FTS production code and framework:
(1/basic) allow FTS to specify resource control goal and use ALTO
to map FTS control state (called links in FTS, where a link is a
pair consisting of a source node to a destination node, where a
node is called RSE) to network state;
(2/basic) implement a full zero-order algorithm, to achieve fully
efficient, zero-order gradient control as FTS Optimizer;
(3/stretch) realize a composition framework to compose end-to-end
resource performance function, including both zero-order and
first-order gradient, covering the bottleneck structure.
The project is exciting both in terms of its technical content
and also in terms of its potential impacts. The hackathon project
will work under the guidance of the Rucio project lead (Dr. Mario
Lassnig) and the FTS project lead (Mihai Patrascoiu and Steven
Murray); all cc'd in this email. This can be a wonderful
opportunity for IETF, the networked systems community and the
data-intensive sciences communities to work together.
Please feel free to reach out to us (Jordi, Mahdi and me) if you
want to join the hackathon or later.
Thanks,
Richard on behalf of the IETF 114 ALTO+FTS Hackathon Team
On Tue, Jul 12, 2022 at 1:14 AM Jordi Ros Giralt
<[email protected]> wrote:
Hi all,
I have uploaded to the IETF Hackathon Wiki the project
description for ALTO. The proposed project is a natural
evolution from the work done during the 113 Hackathon. I want
to also thank and credit Jensen for providing the initial
plan for the Hackathon.
https://trac.ietf.org/trac/ietf/meeting/wiki/114hackathon
You will also see that in addition to the ALTO
hackathon, Ziyang will be leading a second project under the
name "The SDN-based MPTCP-aware and MPQUIC-aware Transmission
Control Model using ALTO". Many thanks Ziyang for leading
this very interesting project too.
Jordi
On behalf of ALTO Hackathon group
_______________________________________________
alto mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/alto
_______________________________________________
alto mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/alto
