Dear Josh, Thanks again for your reply ! I understand the mapping part ! Concerning the VCL model as you might mentioned the terms could be different comparing to a typical IaaS. Most probably I was bit confusing the images provisioning. If I understood it correctly, once reservation process starts, a new image will be deployed in the corresponding computer or host [just streaming the disk images files to build a complete environment within specific resources CPU and RAM]
Therefore, that will be the Physical/Virtual deployer [Hypervisor] role to assign those resources ? Another interesting point which highlights the point if many users can make more than one reservation for the same image ? does it affect the image loading time [queue of requests to access/use shared resource] ? besides, if the image is already deployed and running in a specific computer [can be physical/virtual], can many users access the same computer at the same time ? Thanks for your clarification ! Best regards, Phd Team 2014-11-05 22:34 GMT+01:00 Josh Thompson <[email protected]>: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Khder, > > Mapping is just a way for the scheduler part of the frontend to know what > computers a given image can be loaded on.This could be because of an > architectural restriction or it could simply be because a researcher bought > her own hardware and only her images should be run on them. Mapping > doesn't > actually have to do with where the image files reside. I don't think the > backend code ever even looks at the mapping information. > > Remember that VCL predates the terms cloud computing, IaaS, PaaS, etc. So, > the terms used for VCL might be somewhat different from more recent > systems. > > In VCL, each "image" or "environment" consists of a set of disk image files > and the associated information about the image that is in the database. > Initial "base images" must be created somewhat manually. Then, further > images > are derived from those base images. I don't know that VCL really has > flavors > because each image will have its own CPU and RAM specifications. > > Does that answer your questions? > > Josh > > On Tuesday, November 04, 2014 11:00:01 PM Khder Omar wrote: > > Dear Josh, > > > > I was wondering during the creation of new image, it does not require to > be > > stored in the computer node itself, it can be stored in an image library > > like a SAN or NAS storage then we mapped a given image to a certain > > computer or computer group ! the fact to 'map' an image or image group > to a > > computer or computer group, does it mean 'streaming' the image 'locally' > to > > the computer node ? or only during the reservation ? > > I was bit confused between the terms used in VCL and in a typical IaaS > > environment: the image is claimed as a template of a virtual machine, in > > VCL, I have noticed that the image is being created from a base OS and > > resources information are assigned in the database such as RAM, CPU, > arch, > > id etc... > > Then I might assume [correct me if I understood it wrong] the Virtual > > Machine will be initiated [considering an Hypervisor KVM or ESXi ] while > > processing the reservation. Therefore, we can describe such image in a > > typical IaaS as a virtual machine flavor ready to be deployed and run on > > Host or Computer [within an hypervisor supported]. > > > > Thanks! > > > > PhD team > > > > 2014-10-30 20:07 GMT+01:00 Khder Omar <[email protected]>: > > > Dear Josh, > > > > > > That is true ! Thanks again for the clarification ! I will start > doing > > > some experiments and see what it gives from performance perspective ! > In > > > the other hand, I will keep updating for any proposal of ideas can be > > > generated then we can refine the scheduler code and retest again ! > > > > > > Many thanks for sharing such valuable information! > > > > > > We will keep updated ! > > > > > > Best Regards > > > PhD Team > > > > > > 2014-10-30 19:56 GMT+01:00 Josh Thompson <[email protected]>: > > >> -----BEGIN PGP SIGNED MESSAGE----- > > >> Hash: SHA1 > > >> > > >> Khder, > > >> > > >> The scheduler only allocates a computer randomly if > > >> SCHEDULER_ALLOCATE_RANDOM_COMPUTER is set to 1 in conf.php, which it > is > > >> not by > > >> default. > > >> > > >> Yes, you could create pools that each have a weight or ranking > assigned. > > >> Each > > >> computer would be assigned to a pool. Then, when the initial set of > > >> computers > > >> is generated, they could be ranked by the pool weight/ranks. It would > > >> then be > > >> up to an administer or external application to manage the weight/rank > of > > >> each > > >> pool. > > >> > > >> Josh > > >> > > >> On Thursday, October 30, 2014 7:48:50 PM Khder Omar wrote: > > >> > Dear Josh, > > >> > > > >> > I would like to thank you for the precious hints concerning the > > >> > > >> scheduling > > >> > > >> > nature within VCL. I can see that the set of computers is filtered > in > > >> > advance and then the scheduler will pick up RANDOMLY one of them. To > > >> > > >> check > > >> > > >> > the possibilities to extend the function considering a large scale > > >> > environment where we can introduce the 'region' or 'pool' term in > the > > >> > > >> code. > > >> > > >> > With multiple management nodes, do you think that using the 'weight' > > >> > for > > >> > the workload per pool and then per computer therefore will rank it > will > > >> > > >> be > > >> > > >> > more precise on how to balance the load across the hole > infrastructure > > >> > ? > > >> > > > >> > Best Regards, > > >> > PhD team > > >> > > > >> > 2014-10-30 19:28 GMT+01:00 Josh Thompson <[email protected]>: > > >> > > -----BEGIN PGP SIGNED MESSAGE----- > > >> > > Hash: SHA1 > > >> > > > > >> > > Khder, > > >> > > > > >> > > Sorry for the late response. > > >> > > > > >> > > You can certainly experiment with extending the code to include > more > > >> > > advanced > > >> > > scheduling functions. The existing ranking of computers is > something > > >> > > >> like > > >> > > >> > > this: > > >> > > > > >> > > 1) generate a set of all computers meeting the minimum > requirements > > >> > > >> of the > > >> > > >> > > image, that the user has access to, and that the image is mapped > to; > > >> > > >> order > > >> > > >> > > that list by node specs from low to high > > >> > > 2) remove from that set any computers that are already assigned > > >> > > reservations > > >> > > 3) if the image is virtual, remove any VMs for which the host > doesn't > > >> > > >> have > > >> > > >> > > enough memory to load the VM (without overbooking) > > >> > > 4) assign the new reservation the first computer from that set for > > >> > > >> which > > >> > > >> > > an > > >> > > active management node can be found > > >> > > > > >> > > There's actually 3 sets of computers being tracked - those already > > >> > > >> loaded > > >> > > >> > > with > > >> > > the image being requested, those set aside for a block allocation > of > > >> > > >> which > > >> > > >> > > the > > >> > > user is a member, and then a set of both of those plus any > available > > >> > > computers > > >> > > that are currently loaded with another image. > > >> > > > > >> > > There is a configuration option > (SCHEDULER_ALLOCATE_RANDOM_COMPUTER) > > >> > > >> in > > >> > > >> > > conf.php that allows the sets of computers to be randomized. The > > >> > > motivation > > >> > > behind this option is for sites that have homogeneous virtual > > >> > > >> machines and > > >> > > >> > > hosts. Randomizing the assigned computer should help spread the > load > > >> > > equally > > >> > > among all existing hosts. > > >> > > > > >> > > I hope that helps. > > >> > > > > >> > > Josh > > >> > > > > >> > > On Tuesday, October 21, 2014 9:01:17 PM Khder Omar wrote: > > >> > > > Hi Josh, > > >> > > > > > >> > > > Thanks for your reaction. I have checked the code source and I > > >> > > >> think the > > >> > > >> > > > only function which might fulfill my question is function > > >> > > > > >> > > allocComputer(...) > > >> > > > > >> > > > Eventually, the function a piece test schedule code > > >> > > > > > >> > > > if(SCHEDULER_ALLOCATE_RANDOM_COMPUTER) { > > >> > > > > > >> > > > shuffle($blockids); > > >> > > > shuffle($currentids); > > >> > > > shuffle($computerids); > > >> > > > > > >> > > > } > > >> > > > > > >> > > > > > >> > > > I might assume if it was correctly understood that VCL scheduler > > >> > > >> process > > >> > > >> > > > determines in first place a computer to be assigned to a > management > > >> > > >> node > > >> > > >> > > > from a given array or table in order otherwise it can be > randomly > > >> > > > chosen! > > >> > > > Then, we can assume that the scheduling decision is still using > a > > >> > > >> basic > > >> > > >> > > > order as FCFS or a chance/Random order! It that assumed be > correct > > >> > > >> ? if > > >> > > >> > > so, > > >> > > > > >> > > > is there a way to extend the code by including more advanced > > >> > > >> scheduling > > >> > > >> > > > functions ? > > >> > > > > > >> > > > Best regards, > > >> > > > PhD Team > > >> > > > > > >> > > > 2014-10-21 16:11 GMT+01:00 Josh Thompson < > [email protected]>: > > >> > > > > -----BEGIN PGP SIGNED MESSAGE----- > > >> > > > > Hash: SHA1 > > >> > > > > > > >> > > > > Khder, > > >> > > > > > > >> > > > > The scheduling of reservations is actually done in the web > code. > > >> > > >> Have > > >> > > >> > > a > > >> > > > > >> > > > > look > > >> > > > > at the isAvailable function in the web/.ht-inc/utils.php file. > > >> > > > > > > >> > > > > > https://svn.apache.org/repos/asf/vcl/trunk/web/.ht-inc/utils.php > > >> > > > > > > >> > > > > Josh > > >> > > > > > > >> > > > > On Tuesday, October 21, 2014 4:39:05 PM Khder Omar wrote: > > >> > > > > > Dear all, > > >> > > > > > > > >> > > > > > We were wondering what kind of scheduler algorithms VCL > might > > >> > > >> use ? > > >> > > >> > > Any > > >> > > > > >> > > > > > hints about the scheduler source code will be appreciated. > The > > >> > > >> idea > > >> > > >> > > is > > >> > > > > >> > > > > > actually to check how VCL will perform while changing the > > >> > > >> scheduler > > >> > > >> > > > > > algorithm [in management node]. > > >> > > > > > > > >> > > > > > > > >> > > > > > Thanks in advance > > >> > > > > > > > >> > > > > > Best regards, > > >> > > > > > Phd team > > >> > > > > > > >> > > > > - -- > > >> > > > > - ------------------------------- > > >> > > > > Josh Thompson > > >> > > > > VCL Developer > > >> > > > > North Carolina State University > > >> > > > > > > >> > > > > my GPG/PGP key can be found at pgp.mit.edu > > >> > > > > > > >> > > > > All electronic mail messages in connection with State business > > >> > > >> which > > >> > > >> > > > > are sent to or received by this account are subject to the NC > > >> > > >> Public > > >> > > >> > > > > Records Law and may be disclosed to third parties. > > >> > > > > -----BEGIN PGP SIGNATURE----- > > >> > > > > Version: GnuPG v2.0.22 (GNU/Linux) > > >> > > > > > > >> > > > > > iEYEARECAAYFAlRGd60ACgkQV/LQcNdtPQPMjQCdF/28fx+VlhmZV0WEMobcv+7p > > >> > > > > OAEAn1mct5Iz5bWrLnnX/yQl13wMVwRg > > >> > > > > =RMTU > > >> > > > > -----END PGP SIGNATURE----- > > >> > > > > >> > > - -- > > >> > > - ------------------------------- > > >> > > Josh Thompson > > >> > > VCL Developer > > >> > > North Carolina State University > > >> > > > > >> > > my GPG/PGP key can be found at pgp.mit.edu > > >> > > > > >> > > All electronic mail messages in connection with State business > which > > >> > > are sent to or received by this account are subject to the NC > Public > > >> > > Records Law and may be disclosed to third parties. > > >> > > -----BEGIN PGP SIGNATURE----- > > >> > > Version: GnuPG v2.0.22 (GNU/Linux) > > >> > > > > >> > > iEYEARECAAYFAlRSgzoACgkQV/LQcNdtPQMRBgCfaBOvBvUNnfdr5BLR7cnr+vMx > > >> > > 5uQAniwpvKVjVGdW//rvNBYCKu485Siy > > >> > > =xWum > > >> > > -----END PGP SIGNATURE----- > > >> > > >> - -- > > >> - ------------------------------- > > >> Josh Thompson > > >> VCL Developer > > >> North Carolina State University > > >> > > >> my GPG/PGP key can be found at pgp.mit.edu > > >> > > >> All electronic mail messages in connection with State business which > > >> are sent to or received by this account are subject to the NC Public > > >> Records Law and may be disclosed to third parties. > > >> -----BEGIN PGP SIGNATURE----- > > >> Version: GnuPG v2.0.22 (GNU/Linux) > > >> > > >> iEYEARECAAYFAlRSieYACgkQV/LQcNdtPQOjlACfS97i9Hk0Tmspy+6vfgOcENht > > >> UtYAn008A0w5VBq+eg5iGg1GRlff3zDB > > >> =tCAJ > > >> -----END PGP SIGNATURE----- > - -- > - ------------------------------- > Josh Thompson > VCL Developer > North Carolina State University > > my GPG/PGP key can be found at pgp.mit.edu > > All electronic mail messages in connection with State business which > are sent to or received by this account are subject to the NC Public > Records Law and may be disclosed to third parties. > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.22 (GNU/Linux) > > iEYEARECAAYFAlRal+cACgkQV/LQcNdtPQOAGACeL2Sanhs7ShZaCyKcSo5N2BYk > wLEAn08YbcyiCmR4rY9u4kZJocfxY1Ju > =pxdL > -----END PGP SIGNATURE----- > >
