Re: [hpx-users] Contributing to HPX

[Hartmut Kaiser]

> On Wed, Oct 24, 2018 at 10:14 PM Biddiscombe, John A.
>  wrote:
> Ahmed
>
> I'm a little bit frightened by the idea of you working on hwloc+hpx - the
> reason is the same as the one that caused problems in gsoc.
>
> Something I have in mind, which would be a good fit with clear direction
> and specification would be this here:
> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0796r1.pdf

I'd second this suggestion. This would also align with the hwloc idea 
mentioned before.

Regards Hartmut
---
http://stellar.cct.lsu.edu
https://github.com/STEllAR-GROUP/hpx


>
>
> * we don't have a well defined plan on what we want to do with hwloc yet.
> I would like to clean it up and make it better, but only because it is
> missing a couple of functions that I would like - but I don't really know
> hwloc that well either. (same as I didn't really know fflib)
>
> If you are working on a a problem and the tools you have don't quite solve
> the problem, then you can tweak the tools to work - for this reason I am
> looking at hwloc and asking myself - how do I find out which socket on the
> node is 'closest' to the gpu? Should I create a thread pool for GPU
> managment on socket 0, or socket 1 (or 2,3.4 ...). I will be working on a
> machine with 6 gpu's and 2 processors. it is probably safe to assume that
> gpu's 0,1,2 are closest to socket 0, and 3,4,5 to socket 1 - but maybe
> it's 0,2,4 and 1,3,5 - I'd like to be able to query hwloc and get this
> info and build it into our resource partitioned in such a way that the
> programmer can just say "which gpus should I use from this socket" or vice
> versa.
> So I know what I want - but I don't really have a plan yet, and usually,
> one knocks up a bit of code, get's something that half works and is good
> enough, then tries to make it fit into the existing framework and realize
> that something else needs to be tweaked to make it fit nicely with the
> rest of HPX. This requires a deal of understanding of the hpx internals
> and how it all fits together.
>
> So in summary, its better for you to find a project that already interests
> you (part of your coursework?) and use HPX for that project - this gives
> you time to learn how hpx works and use it a bit - then extend it
> gradually as your knowledge grows. hwloc/topology is quite a low level
> part of hpx and does require a bit of help. If you were here in the office
> next door to me, it'd be no problem - cos you could get help any time, but
> working on your own is going to be tough.
>
> I'm not saying do't do it (I did suggest it after all), but it will be
> hard to get going.
>
> I will try to think of some other less low level things we could use help
> with. (Ideas Thomas?)
>
> JB
>
> 
> From: mailto:hpx-users-boun...@stellar.cct.lsu.edu [mailto:hpx-users-
> boun...@stellar.cct.lsu.edu] on behalf of Ahmed Ali
> [mailto:ahmed.al...@eng-st.cu.edu.eg]
> Sent: 24 October 2018 14:12
> To: mailto:hartmut.kai...@gmail.com
> Cc: mailto:hpx-users@stellar.cct.lsu.edu
> Subject: Re: [hpx-users] Contributing to HPX
> John, Hartmut,
>
> Well I can go with that hwloc part. Where should I start? I think
> I  should start by reading about hwloc from the open-mpi document. Is
> there any document about the HPX interface with hwloc?
>
> Best,
> Ahmed Samir
>
> On Tue, Oct 23, 2018 at 1:46 PM Hartmut Kaiser
>  wrote:
>
> John, Ahmed,
>
> The hwloc integration sounds like a good idea to me. But anything else
> would be fine as well.,,
>
> Regards Hartmut
> ---
> http://stellar.cct.lsu.edu
> https://github.com/STEllAR-GROUP/hpx
>
> From: mailto:hpx-users-boun...@stellar.cct.lsu.edu  boun...@stellar.cct.lsu.edu> On Behalf Of Biddiscombe, John A.
> Sent: Tuesday, October 23, 2018 3:43 AM
> To: mailto:hpx-users@stellar.cct.lsu.edu
> Subject: Re: [hpx-users] Contributing to HPX
>
> Ahmed
>
> Good to hear from you again.
>
> >I was a participant in GSoC'18 in HPX but I've failed the program.
>
> I still feel bad about that.
>
> >However, I liked the project and the contribution to HPX. I wanted to
> continue contributing to HPX so I want someone to guide me.
> Nice.
>
> >What part should I work on now? and where should I start?
> Well, there is the obvious option of continuing the parcelport work, but I
> suspect you want to do something else since we didn't help you enough
> first time around. I'd certainly like to carry on with that and get it up
> and running. It's on my list, but I have a full plate already
> unfortunately, so it has to wait if I'm doing it myself.
>
> There should still be a list of 'projects' that were compiled for GSoC
> that you could look through and if something looks interesting, have a go
> at it.
>
> If you have a project that you are already working on for your studies or
> hobbies - why not try involving that somehow. Experience tells me that if
> someone 

Re: [hpx-users] Contributing to HPX

[Thomas Heller]

On Wed, Oct 24, 2018 at 10:14 PM Biddiscombe, John A. 
wrote:

> Ahmed
>
> I'm a little bit frightened by the idea of you working on hwloc+hpx - the
> reason is the same as the one that caused problems in gsoc.
>

Something I have in mind, which would be a good fit with clear direction
and specification would be this here:
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0796r1.pdf


>
> * we don't have a well defined plan on what we want to do with hwloc yet.
> I would like to clean it up and make it better, but only because it is
> missing a couple of functions that I would like - but I don't really know
> hwloc that well either. (same as I didn't really know fflib)
>
> If you are working on a a problem and the tools you have don't quite solve
> the problem, then you can tweak the tools to work - for this reason I am
> looking at hwloc and asking myself - how do I find out which socket on the
> node is 'closest' to the gpu? Should I create a thread pool for GPU
> managment on socket 0, or socket 1 (or 2,3.4 ...). I will be working on a
> machine with 6 gpu's and 2 processors. it is probably safe to assume that
> gpu's 0,1,2 are closest to socket 0, and 3,4,5 to socket 1 - but maybe it's
> 0,2,4 and 1,3,5 - I'd like to be able to query hwloc and get this info and
> build it into our resource partitioned in such a way that the programmer
> can just say "which gpus should I use from this socket" or vice versa.
> So I know what I want - but I don't really have a plan yet, and usually,
> one knocks up a bit of code, get's something that half works and is good
> enough, then tries to make it fit into the existing framework and realize
> that something else needs to be tweaked to make it fit nicely with the rest
> of HPX. This requires a deal of understanding of the hpx internals and how
> it all fits together.
>
> So in summary, its better for you to find a project that already interests
> you (part of your coursework?) and use HPX for that project - this gives
> you time to learn how hpx works and use it a bit - then extend it gradually
> as your knowledge grows. hwloc/topology is quite a low level part of hpx
> and does require a bit of help. If you were here in the office next door to
> me, it'd be no problem - cos you could get help any time, but working on
> your own is going to be tough.
>
> I'm not saying do't do it (I did suggest it after all), but it will be
> hard to get going.
>
> I will try to think of some other less low level things we could use help
> with. (Ideas Thomas?)
>
> JB
>
> --
> *From:* hpx-users-boun...@stellar.cct.lsu.edu [
> hpx-users-boun...@stellar.cct.lsu.edu] on behalf of Ahmed Ali [
> ahmed.al...@eng-st.cu.edu.eg]
> *Sent:* 24 October 2018 14:12
> *To:* hartmut.kai...@gmail.com
> *Cc:* hpx-users@stellar.cct.lsu.edu
> *Subject:* Re: [hpx-users] Contributing to HPX
>
> John, Hartmut,
>
> Well I can go with that hwloc part. Where should I start? I think I
> should start by reading about hwloc from the open-mpi document. Is there
> any document about the HPX interface with hwloc?
>
> Best,
> Ahmed Samir
>
> On Tue, Oct 23, 2018 at 1:46 PM Hartmut Kaiser 
> wrote:
>
>>
>>
>> John, Ahmed,
>>
>>
>>
>> The hwloc integration sounds like a good idea to me. But anything else
>> would be fine as well.,,
>>
>>
>>
>> Regards Hartmut
>>
>> ---
>>
>> http://stellar.cct.lsu.edu
>>
>> https://github.com/STEllAR-GROUP/hpx
>>
>>
>>
>> *From:* hpx-users-boun...@stellar.cct.lsu.edu <
>> hpx-users-boun...@stellar.cct.lsu.edu> *On Behalf Of *Biddiscombe, John
>> A.
>> *Sent:* Tuesday, October 23, 2018 3:43 AM
>> *To:* hpx-users@stellar.cct.lsu.edu
>> *Subject:* Re: [hpx-users] Contributing to HPX
>>
>>
>>
>> Ahmed
>>
>>
>>
>> Good to hear from you again.
>>
>>
>>
>> >I was a participant in GSoC'18 in HPX but I've failed the program.
>>
>>
>>
>> I still feel bad about that.
>>
>>
>>
>> >However, I liked the project and the contribution to HPX. I wanted to
>> continue contributing to HPX so I want someone to guide me.
>>
>> Nice.
>>
>>
>>
>> >What part should I work on now? and where should I start?
>>
>> Well, there is the obvious option of continuing the parcelport work, but
>> I suspect you want to do something else since we didn’t help you enough
>> first time around. I’d certainly like to carry on with that and get it up
>> and running. It’s on my list, but I have a full plate already
>> unfortunately, so it has to wait if I’m doing it myself.
>>
>>
>>
>> There should still be a list of ‘projects’ that were compiled for GSoC
>> that you could look through and if something looks interesting, have a go
>> at it.
>>
>>
>>
>> If you have a project that you are already working on for your studies or
>> hobbies - why not try involving that somehow. Experience tells me that if
>> someone has a problem to solve and they use a library like HPX to solve it,
>> then they get much better results than if they just make up a project and
>> try to implement it. The 

Re: [hpx-users] Contributing to HPX

[Simberg Mikael]

Hi Ahmed,

One thing that you could work on is adding more (CDash) timing outputs to our 
benchmarks. John already started this with 
https://github.com/STEllAR-GROUP/hpx/pull/3421 and 
https://github.com/STEllAR-GROUP/hpx/issues/3422. I know this is not 
necessarily technically very challenging, but it is something we would 
appreciate and it would expose you to parts of HPX that you might not have seen 
before. Maybe you'll find a part of HPX that you'd be interested in working on, 
either unrelated to benchmarking or directly related by improving the 
performance of something in HPX.

If you think this is interesting a good starting point are the two links above. 
After that essentially everything in "tests/performance" could have timing 
output added (at least everything that benchmarks HPX performance). Some unit 
tests could also have timing output added (as John already started doing).

Kind regards,
Mikael


From: hpx-users-boun...@stellar.cct.lsu.edu 
[hpx-users-boun...@stellar.cct.lsu.edu] on behalf of Ahmed Samir 
[asami...@hotmail.com]
Sent: Thursday, October 25, 2018 7:20 PM
To: hpx-users@stellar.cct.lsu.edu
Subject: Re: [hpx-users] Contributing to HPX

Patrick,

Great. Do you recommend a specific issue that I could start with? I took a look 
on the issues and found out that I need someone to direct me to some keys to 
start with any issue. So could you give me some pointers on an issue to get 
started?

Best,
Ahmed Samir

Sent from Mail for Windows 10


From: hpx-users-boun...@stellar.cct.lsu.edu 
 on behalf of Patrick Diehl 

Sent: Thursday, October 25, 2018 2:17:28 PM
To: hpx-users@stellar.cct.lsu.edu
Subject: Re: [hpx-users] Contributing to HPX

Ahmed,

I would recommend to look at the issues for HPX. One first step could be to 
understand one of the issues and start to work on it.

Best,

Patrick

On Thu, Oct 25, 2018, 3:55 AM Ahmed Samir 
mailto:asami...@hotmail.com>> wrote:
John,

Currently I am not working on a project in which I could integrate HPX. I’ll 
think about that.

It would be great if you got any ideas about small parts in HPX in which I 
could go on with, without going deeply in the HPX architecture. If you got any 
ideas please send it to me.

Best,
Ahmed Samir

Sent from Mail for Windows 10


From: 
hpx-users-boun...@stellar.cct.lsu.edu
 
mailto:hpx-users-boun...@stellar.cct.lsu.edu>>
 on behalf of Biddiscombe, John A. mailto:biddi...@cscs.ch>>
Sent: Wednesday, October 24, 2018 10:14:17 PM
To: hpx-users@stellar.cct.lsu.edu; 
hartmut.kai...@gmail.com

Subject: Re: [hpx-users] Contributing to HPX

Ahmed

I'm a little bit frightened by the idea of you working on hwloc+hpx - the 
reason is the same as the one that caused problems in gsoc.

* we don't have a well defined plan on what we want to do with hwloc yet. I 
would like to clean it up and make it better, but only because it is missing a 
couple of functions that I would like - but I don't really know hwloc that well 
either. (same as I didn't really know fflib)

If you are working on a a problem and the tools you have don't quite solve the 
problem, then you can tweak the tools to work - for this reason I am looking at 
hwloc and asking myself - how do I find out which socket on the node is 
'closest' to the gpu? Should I create a thread pool for GPU managment on socket 
0, or socket 1 (or 2,3.4 ...). I will be working on a machine with 6 gpu's and 
2 processors. it is probably safe to assume that gpu's 0,1,2 are closest to 
socket 0, and 3,4,5 to socket 1 - but maybe it's 0,2,4 and 1,3,5 - I'd like to 
be able to query hwloc and get this info and build it into our resource 
partitioned in such a way that the programmer can just say "which gpus should I 
use from this socket" or vice versa.
So I know what I want - but I don't really have a plan yet, and usually, one 
knocks up a bit of code, get's something that half works and is good enough, 
then tries to make it fit into the existing framework and realize that 
something else needs to be tweaked to make it fit nicely with the rest of HPX. 
This requires a deal of understanding of the hpx internals and how it all fits 
together.

So in summary, its better for you to find a project that already interests you 
(part of your coursework?) and use HPX for that project - this gives you time 
to learn how hpx works and use it a bit - then extend it gradually as your 
knowledge grows. hwloc/topology is quite a low level part of hpx and does 
require a bit of help. If you were here in the office next door to me, it'd be 
no problem - cos you could get help any time, but working on your own is going 
to be tough.

I'm not saying do't do it (I did suggest it after all), but 

Re: [hpx-users] Contributing to HPX

[Ahmed Samir]

Patrick,

Great. Do you recommend a specific issue that I could start with? I took a look 
on the issues and found out that I need someone to direct me to some keys to 
start with any issue. So could you give me some pointers on an issue to get 
started?

Best,
Ahmed Samir

Sent from Mail for Windows 10


From: hpx-users-boun...@stellar.cct.lsu.edu 
 on behalf of Patrick Diehl 

Sent: Thursday, October 25, 2018 2:17:28 PM
To: hpx-users@stellar.cct.lsu.edu
Subject: Re: [hpx-users] Contributing to HPX

Ahmed,

I would recommend to look at the issues for HPX. One first step could be to 
understand one of the issues and start to work on it.

Best,

Patrick

On Thu, Oct 25, 2018, 3:55 AM Ahmed Samir 
mailto:asami...@hotmail.com>> wrote:
John,

Currently I am not working on a project in which I could integrate HPX. I’ll 
think about that.

It would be great if you got any ideas about small parts in HPX in which I 
could go on with, without going deeply in the HPX architecture. If you got any 
ideas please send it to me.

Best,
Ahmed Samir

Sent from Mail for Windows 10


From: 
hpx-users-boun...@stellar.cct.lsu.edu
 
mailto:hpx-users-boun...@stellar.cct.lsu.edu>>
 on behalf of Biddiscombe, John A. mailto:biddi...@cscs.ch>>
Sent: Wednesday, October 24, 2018 10:14:17 PM
To: hpx-users@stellar.cct.lsu.edu; 
hartmut.kai...@gmail.com

Subject: Re: [hpx-users] Contributing to HPX

Ahmed

I'm a little bit frightened by the idea of you working on hwloc+hpx - the 
reason is the same as the one that caused problems in gsoc.

* we don't have a well defined plan on what we want to do with hwloc yet. I 
would like to clean it up and make it better, but only because it is missing a 
couple of functions that I would like - but I don't really know hwloc that well 
either. (same as I didn't really know fflib)

If you are working on a a problem and the tools you have don't quite solve the 
problem, then you can tweak the tools to work - for this reason I am looking at 
hwloc and asking myself - how do I find out which socket on the node is 
'closest' to the gpu? Should I create a thread pool for GPU managment on socket 
0, or socket 1 (or 2,3.4 ...). I will be working on a machine with 6 gpu's and 
2 processors. it is probably safe to assume that gpu's 0,1,2 are closest to 
socket 0, and 3,4,5 to socket 1 - but maybe it's 0,2,4 and 1,3,5 - I'd like to 
be able to query hwloc and get this info and build it into our resource 
partitioned in such a way that the programmer can just say "which gpus should I 
use from this socket" or vice versa.
So I know what I want - but I don't really have a plan yet, and usually, one 
knocks up a bit of code, get's something that half works and is good enough, 
then tries to make it fit into the existing framework and realize that 
something else needs to be tweaked to make it fit nicely with the rest of HPX. 
This requires a deal of understanding of the hpx internals and how it all fits 
together.

So in summary, its better for you to find a project that already interests you 
(part of your coursework?) and use HPX for that project - this gives you time 
to learn how hpx works and use it a bit - then extend it gradually as your 
knowledge grows. hwloc/topology is quite a low level part of hpx and does 
require a bit of help. If you were here in the office next door to me, it'd be 
no problem - cos you could get help any time, but working on your own is going 
to be tough.

I'm not saying do't do it (I did suggest it after all), but it will be hard to 
get going.

I will try to think of some other less low level things we could use help with. 
(Ideas Thomas?)

JB


From: 
hpx-users-boun...@stellar.cct.lsu.edu
 
[hpx-users-boun...@stellar.cct.lsu.edu]
 on behalf of Ahmed Ali 
[ahmed.al...@eng-st.cu.edu.eg]
Sent: 24 October 2018 14:12
To: hartmut.kai...@gmail.com
Cc: hpx-users@stellar.cct.lsu.edu
Subject: Re: [hpx-users] Contributing to HPX

John, Hartmut,

Well I can go with that hwloc part. Where should I start? I think I  should 
start by reading about hwloc from the open-mpi document. Is there any document 
about the HPX interface with hwloc?

Best,
Ahmed Samir

On Tue, Oct 23, 2018 at 1:46 PM Hartmut Kaiser 
mailto:hartmut.kai...@gmail.com>> wrote:

John, Ahmed,

The hwloc integration sounds like a good idea to me. But anything else would be 
fine as well.,,

Regards Hartmut
---
http://stellar.cct.lsu.edu
https://github.com/STEllAR-GROUP/hpx

From: 

Re: [hpx-users] Contributing to HPX

[Patrick Diehl]

Ahmed,

I would recommend to look at the issues for HPX. One first step could be to
understand one of the issues and start to work on it.

Best,

Patrick

On Thu, Oct 25, 2018, 3:55 AM Ahmed Samir  wrote:

> John,
>
> Currently I am not working on a project in which I could integrate HPX.
> I’ll think about that.
>
>
>
> It would be great if you got any ideas about small parts in HPX in which I
> could go on with, without going deeply in the HPX architecture. If you got
> any ideas please send it to me.
>
> Best,
> Ahmed Samir
>
>
>
> Sent from Mail  for
> Windows 10
>
>
> --
> *From:* hpx-users-boun...@stellar.cct.lsu.edu <
> hpx-users-boun...@stellar.cct.lsu.edu> on behalf of Biddiscombe, John A. <
> biddi...@cscs.ch>
> *Sent:* Wednesday, October 24, 2018 10:14:17 PM
> *To:* hpx-users@stellar.cct.lsu.edu; hartmut.kai...@gmail.com
>
> *Subject:* Re: [hpx-users] Contributing to HPX
>
> Ahmed
>
> I'm a little bit frightened by the idea of you working on hwloc+hpx - the
> reason is the same as the one that caused problems in gsoc.
>
> * we don't have a well defined plan on what we want to do with hwloc yet.
> I would like to clean it up and make it better, but only because it is
> missing a couple of functions that I would like - but I don't really know
> hwloc that well either. (same as I didn't really know fflib)
>
> If you are working on a a problem and the tools you have don't quite solve
> the problem, then you can tweak the tools to work - for this reason I am
> looking at hwloc and asking myself - how do I find out which socket on the
> node is 'closest' to the gpu? Should I create a thread pool for GPU
> managment on socket 0, or socket 1 (or 2,3.4 ...). I will be working on a
> machine with 6 gpu's and 2 processors. it is probably safe to assume that
> gpu's 0,1,2 are closest to socket 0, and 3,4,5 to socket 1 - but maybe it's
> 0,2,4 and 1,3,5 - I'd like to be able to query hwloc and get this info and
> build it into our resource partitioned in such a way that the programmer
> can just say "which gpus should I use from this socket" or vice versa.
> So I know what I want - but I don't really have a plan yet, and usually,
> one knocks up a bit of code, get's something that half works and is good
> enough, then tries to make it fit into the existing framework and realize
> that something else needs to be tweaked to make it fit nicely with the rest
> of HPX. This requires a deal of understanding of the hpx internals and how
> it all fits together.
>
> So in summary, its better for you to find a project that already interests
> you (part of your coursework?) and use HPX for that project - this gives
> you time to learn how hpx works and use it a bit - then extend it gradually
> as your knowledge grows. hwloc/topology is quite a low level part of hpx
> and does require a bit of help. If you were here in the office next door to
> me, it'd be no problem - cos you could get help any time, but working on
> your own is going to be tough.
>
> I'm not saying do't do it (I did suggest it after all), but it will be
> hard to get going.
>
> I will try to think of some other less low level things we could use help
> with. (Ideas Thomas?)
>
> JB
>
> --
> *From:* hpx-users-boun...@stellar.cct.lsu.edu [
> hpx-users-boun...@stellar.cct.lsu.edu] on behalf of Ahmed Ali [
> ahmed.al...@eng-st.cu.edu.eg]
> *Sent:* 24 October 2018 14:12
> *To:* hartmut.kai...@gmail.com
> *Cc:* hpx-users@stellar.cct.lsu.edu
> *Subject:* Re: [hpx-users] Contributing to HPX
>
> John, Hartmut,
>
> Well I can go with that hwloc part. Where should I start? I think I
> should start by reading about hwloc from the open-mpi document. Is there
> any document about the HPX interface with hwloc?
>
> Best,
> Ahmed Samir
>
> On Tue, Oct 23, 2018 at 1:46 PM Hartmut Kaiser 
> wrote:
>
>>
>>
>> John, Ahmed,
>>
>>
>>
>> The hwloc integration sounds like a good idea to me. But anything else
>> would be fine as well.,,
>>
>>
>>
>> Regards Hartmut
>>
>> ---
>>
>> http://stellar.cct.lsu.edu
>>
>> https://github.com/STEllAR-GROUP/hpx
>>
>>
>>
>> *From:* hpx-users-boun...@stellar.cct.lsu.edu <
>> hpx-users-boun...@stellar.cct.lsu.edu> *On Behalf Of *Biddiscombe, John
>> A.
>> *Sent:* Tuesday, October 23, 2018 3:43 AM
>> *To:* hpx-users@stellar.cct.lsu.edu
>> *Subject:* Re: [hpx-users] Contributing to HPX
>>
>>
>>
>> Ahmed
>>
>>
>>
>> Good to hear from you again.
>>
>>
>>
>> >I was a participant in GSoC'18 in HPX but I've failed the program.
>>
>>
>>
>> I still feel bad about that.
>>
>>
>>
>> >However, I liked the project and the contribution to HPX. I wanted to
>> continue contributing to HPX so I want someone to guide me.
>>
>> Nice.
>>
>>
>>
>> >What part should I work on now? and where should I start?
>>
>> Well, there is the obvious option of continuing the parcelport work, but
>> I suspect you want to do something else since we didn’t help you