Re: [Gluster-devel] Assistance needed for testing Gluster in the CentOS CI infrastructure

[Vipul Nayyar]

Hey Niels,
I'd like to volunteer for this to work on it in my spare time. Can't promise 
dedicated bandwidth for this, but it'll surely get me back on track about 
learning GlusterFS. I've helped out Justin in some infra related tasks some 
time back. My present day job is as DevOps, so this is surely a great learning 
opportunity. 
Please do point me to related resources where I can start learning about the 
CentOS architecture. I already have some knowledge about Jenkins and how to 
setup jobs on it.
RegardsVipul Nayyar 
 


On Monday, 30 November 2015 8:26 PM, Niels de Vos  wrote:
 

 Hi!

We're looking for interested users and developers that can spend some
time on setting up tests for Gluster in the CentOS CI infrastructure.
Gluster is part of the CentOS Storage SIG, and that enables us to use
the CentOS CI testing systems.

I would like to have a few volunteers that are willing to create the
first Jenkins jobs for Gluster on the CentOS CI. This is not a common
Jenkins environment like we have on build.gluster.org. On ci.centos.org
the Jenkins slaves function as a management node which reserve
additional (non Jenkins slaves) for running tests. It will require some
dedicated time to understand the available infrastructure and get
something useful out of it. There are several CentOS admins/developers
that are willing to explain and help with the testing, but we need some
people that can take care of it from a Gluster perspective.

Some links that would be of interest for this work:
 - http://wiki.centos.org/QaWiki/CI
 - https://wiki.centos.org/SpecialInterestGroup/Storage
 - 
https://github.com/gluster/glusterfs/blob/master/doc/developer-guide/Using-Gluster-Test-Framework.md

Anyone that is interested and can regularly spend a little time on this
is most welcome to sign up. Please contact me and I'll try to get a
little group on its way.

Thanks,
Niels

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

  ___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-infra] Gerrit login not working

[Vipul Nayyar]

Hey there,
Just checking on the status for this. Still can't login with github onto 
gerrit. Added my details to the pad.
RegardsVipul Nayyar 
 


 On Thursday, 13 August 2015 7:06 PM, M S Vishwanath Bhat 
msvb...@gmail.com wrote:
   

 Vipul, Please add your name in this pad 
https://public.pad.fsfe.org/p/gluster-gerrit-migration

Vijay, Neils or someone will help you out soon.

Cheers
MS

On 13 August 2015 at 18:55, Vipul Nayyar nayyar_vi...@yahoo.com wrote:

Hey everyone,
I'm having trouble cloning the glusterfs repo from gerrit. Apparently I need to 
update my ssh keys. But I can't login to the gerrit system like in the past, 
with yahoo or normal email authentication. The github authentication link at 
the top right after authenticating with github credentials redirects to a page 
with simply 'Forbidden' written on it. 
I'm not sure, if I missed a change in the mailing lists about another method to 
officially clone and contribute to the repo. But kindly help me out.
Regards
Vipul Nayyar 

___
Gluster-infra mailing list
gluster-in...@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-infra





  ___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Gerrit login not working

[Vipul Nayyar]

Hey everyone,
I'm having trouble cloning the glusterfs repo from gerrit. Apparently I need to 
update my ssh keys. But I can't login to the gerrit system like in the past, 
with yahoo or normal email authentication. The github authentication link at 
the top right after authenticating with github credentials redirects to a page 
with simply 'Forbidden' written on it. 
I'm not sure, if I missed a change in the mailing lists about another method to 
officially clone and contribute to the repo. But kindly help me out.
Regards
Vipul Nayyar 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Anyone willing to help out with GlusterFS infrastructure?

[Vipul Nayyar]

Hi Justin,

Any room for someone who's looking to contribute and learn, starting with some 
basic tasks? :-D

Regards

Vipul Nayyar 



On Friday, 3 October 2014 5:04 AM, Justin Clift jus...@gluster.org wrote:
 


As a thought, we have a bunch of VM's and similar online for GlusterFS,
running regression tests, our git repos, the main website, and so on.

It's interesting stuff sometimes, and other times a bit of a time kill... ;)

Is anyone out there with a bit of time occasionally to help out, keeping
this stuff running, helping keep the website updated, and so on?

Note - asking for ongoing volunteers, NOT looking for people wanting a
paid job. :)

Regards and best wishes,

Justin Clift

--
GlusterFS - http://www.gluster.org

An open source, distributed file system scaling to several
petabytes, and handling thousands of clients.

My personal twitter: twitter.com/realjustinclift

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Feedback and future plans for glusterfsiostat

[Vipul Nayyar]

Hi Justin,

Regarding the server.py script, yes that graph is live and it is refreshed 
periodically. According to the model right now, the server script at the 
backend polls the meta directory of mounted volumes repeatedly, calculates the 
speed from the difference of values encountered and stores it. Another thread 
on the server flushes this data to the webpage as and when the server is 
accessed through a browser. The current interval of requests done from server 
to mounted volume and from browser to server is 1 sec.

On the other note, I'm not sure that io-stats stores stats related to a 
specific directory or inode as of know, IIRC. As Krishnan said, we'll have to 
look into other methods if we intend to do that.

Regards
Vipul Nayyar 



On Thursday, 4 September 2014 3:17 AM, Justin Clift jus...@gluster.org wrote:
 


On 03/09/2014, at 3:48 PM, Vipul Nayyar wrote:
snip
 3) server.py which is a web server written with very basic python libraries. 
 I've attached some screenshots of the server in action while visualizing live 
 data from gluster mounts in the form of web based graphs.
 
 Justin, since this web server implementation is a bit similar to your tool 
 glusterflow, your feedback regarding this is really important and valuable.

Thanks.  Server.py is the bit that caught my attention too. :)

I like the screenshots.  Does server.py update/refresh in real-time?

Also, one thing that's missing in the screenshots is the time info
on the X axis.  eg how far apart are each of the tick marks on the
x axis?  Minor oversight, but important. ;)


 Some key points for those who want to test this thing.
 
 • Apply this patch http://review.gluster.org/#/c/8244/ , build and install.
 • Please grab the latest source code from 
 https://forge.gluster.org/glusterfsiostat
 • Profiling needs to be turned on for the volumes regarding which you want 
 I/O data.
 • Run the cmd line tool by `python stat.py` Giving --help tag would list the 
 other options.
 • Start the server.py script in the same way and point your browser at 
 'localhost:8080'
 • The gluster mounts would be visible as separate tabs on the left in 
 browser. Click on a tab to see the related graphs.
 • Since these graphs show live I/O activity, you need to run a read/write 
 operation to see the graphs in action.
 Please do contact me regarding any suggestions or thoughts about this.

This is is very cool.  As a thought, since I don't know the code
at all, is could it do stuff for parts of a volume?

For example in the server.py GUI a person could give a directory
path inside a volume, and it would show the IO operations stats
for just that path?


+ Justin

--
GlusterFS - http://www.gluster.org

An open source, distributed file system scaling to several
petabytes, and handling thousands of clients.

My personal twitter: twitter.com/realjustinclift___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Academic project in Distributed systems

[Vipul Nayyar]

Hi Justin,

I'd like to work preferably in C or Python. BTW apart from Java, language is 
not a barrier for me if the problem being encountered is challenging. 

Also I won't say I'm very good at maths, but I'm quite comfortable with 
pursuing any topic that might be relevant to the project.
Please do refer me to any relevant topics that I should look at.

Regards
Vipul Nayyar 



On Thursday, 7 August 2014 9:06 PM, Justin Clift jus...@gluster.org wrote:
 


- Original Message -
 Hello,
 
 I'm Vipul. I'm currently in my final year of computer engineering and would
 like some guidance on choosing an academic project to spend 2-3 months
 working on it. I've got a little experience in some internal Glusterfs
 components. But overall, I'm interested in contributing to the field of
 Distributed systems, OS or Cloud. Although an opportunity to contribute back
 to an Open source project including Gluster or some another would be great,
 but a research oriented project in this domain would be really exciting to
 work on.

As a first thought before recommending stuff, which programming languages do
you have a good understanding of (and like to use)?

That will help figure things out.  Also, how good are your math skills?  If
they're very strong, then some of the clustering algorithm stuff might be
the thing to look at. :)



 On another note, I apologize if this email perceives you to be a misuse of
 the mailing list, but I'd really be grateful, if I could get any pointers
 regarding this.

You're fine.  It's not abuse of the mailing list at all. :)

Regards and best wishes,

Justin Clift

-- 
GlusterFS - http://www.gluster.org

An open source, distributed file system scaling to several
petabytes, and handling thousands of clients.

My personal twitter: twitter.com/realjustinclift___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Academic project in Distributed systems

[Vipul Nayyar]

Hello,





I'm Vipul. I'm currently in my final year of computer engineering and would 
like some guidance on choosing an academic project to spend 2-3 months working 
on it. I've got a little experience in some internal Glusterfs components. But 
overall, I'm interested in contributing to the field of Distributed systems, OS 
or Cloud. Although an opportunity to contribute back to an Open source project 
including Gluster or some another would be great, but a research oriented 
project in this domain would be really exciting to work on. 

On another note, I apologize if this email perceives you to be a misuse of the 
mailing list, but I'd really be grateful, if I could get any pointers regarding 
this. 

Regards
Vipul Nayyar ___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Academic project in Distributed systems

[Vipul Nayyar]

Hello,

I'm Vipul. I'm currently in my final year of computer engineering and would 
like some guidance on choosing an academic project to spend 2-3 months working 
on it. I've got a little experience in some internal Glusterfs components. But 
overall, I'm interested in contributing to the field of Distributed systems, OS 
or Cloud. Although an opportunity to contribute back to an Open source project 
including Gluster or some another would be great, but a research oriented 
project in this domain would be really exciting to work on. 

On another note, I apologize if this email perceives you to be a misuse of the 
mailing list, but I'd really be grateful, if I could get any pointers regarding 
this. 

Regards
Vipul Nayyar 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Proposal for change regarding latency calculation

[Vipul Nayyar]

Hello,

Following is a proposal for modifying the io profiling capability of the 
io-stats xlator. I recently sent in a patch(review.gluster.org/#/c/8244/) 
regarding that, which uses the already written latency related functions in 
io-stats to dump info through meta and added some more data containers which 
would track some more fops related info each time a request goes through 
io-stats. Currently, before the io-stats' custom latency functions can run, the 
measure_latency and count_fop_hits option should be enabled. I propose to 
remove these two options entirely from io-stats.

In order to track io performance, these options should be enabled all the time, 
or removed entirely, so that a record of io requests can be kept since mount 
time, else enabling these options only when it is required will not give you 
the average statistics over the whole period since the start. This is based on 
the methodology of Linux kernel itself, since it internally maintains the io 
statistics data structures all the time and presents it via /proc filesystem 
whenever required. Enabling of any option is not required, and the data 
available represents statistics since the boot time.

I would like to know the views over this, if having io-stats profiling info 
available all the time would be a good thing?

Apart from this, I was going over latency.c in libglusterfs, which does a fine 
job of maintaining latency info for every xlator and encountered an anomaly 
which I thought should be dealt with. The function gf_proc_dump_latency_info 
which dumps the latency array for the specified xlator consists of a last line 
which in the end flushes this array through memset after every dump. That 
means, you get different latency info every time you read the profile file in 
meta. I think, flushing the data structure after every dump is wrong since, you 
don't get overall stats since one enabled the option at the top of meta, and 
more importantly, multiple applications reading this file can get wrong info, 
since it gets cleared after one read only.

If my reasons seem apt for you, I'll send a patch over for evaluation.

Regards

Vipul Nayyar 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] New project on the Forge - gstatus

[Vipul Nayyar]

Hello,


Thanks Avati for your suggestion. I tried studying the output generated my the 
meta xlator. One of the problems that I'm facing is that in order to enable 
latency measurement with meta, I need to write '1' into the the file 
.meta/measure_latency. But due to some reason I'm not able to do that. I don't 
get any permission error while writing. But the text written in the file is 
always the default one. So far I've only been able to change the digit by hard 
coding it in the meta source and building it again. I ran glusterfs under gdb 
and set a break point on  measure_file_write(). The redacted gdb output can be 
found at http://fpaste.org/104770/

As you can see when I try to write 1 (echo 1  measure_latency) into the file, 
the data received in iov is 1\n which gives the correct value in num and 
this-ctx-measure_latency. But somehow, measure_file_write is called again 
with data iov=\n\n. Therefore I feel that this write process is not doing 
it's job properly. If you think that I've made some mistake please do notify me.

Regards
Vipul Nayyar 


On Saturday, 17 May 2014 6:24 PM, Kaushal M kshlms...@gmail.com wrote:
 


Avati that's awesome! I didn't know that the meta xlator could do that already.



On Sat, May 17, 2014 at 9:30 AM, Anand Avati av...@gluster.org wrote:

KP, Vipul,


It will be awesome to get io-stats like instrumentation on the client side. 
Here are some further thoughts on how to implement that. If you have a recent 
git HEAD build, I would suggest that you explore the latency stats on the 
client side exposed through meta at $MNT/.meta/graphs/active/$xlator/profile. 
You can enable latency measurement with echo 1  $MNT/.meta/measure_latency. 
I would suggest extending these stats with the extra ones io-stats has, and 
make glusterfsiostats expose these stats. 


If you can compare libglusterfs/src/latency.c:gf_latency_begin(), 
gf_latency_end() and gf_latency_udpate() with the macros in io-stats.c 
UPDATE_PROFILE_STATS() and START_FOP_LATENCY(), you will quickly realize how a 
lot of logic is duplicated between io-stats and latency.c. If you can enhance 
latency.c and make it capture the remaining stats what io-stats is capturing, 
the benefits of this approach would be:


- stats are already getting captured at all xlator levels, and not just at the 
position where io-stats is inserted
- file like interface makes the stats more easily inspectable and consumable, 
and updated on the fly
- conforms with the way rest of the internals are exposed through $MNT/.meta


In order to this, you might want to look into:


- latency.c as of today captures fop count, mean latency, total time, whereas 
io-stats measures these along with min-time, max-time and block-size histogram.
- extend gf_proc_dump_latency_info() to dump the new stats
- either prettify that output like 'volume profile info' output, or JSONify it 
like xlators/meta/src/frames-file.c
- add support for cumulative vs interval stats (store an extra copy of 
this-latencies[])


etc..


Thanks!




On Fri, Apr 25, 2014 at 9:09 PM, Krishnan Parthasarathi kpart...@redhat.com 
wrote:

[Resending due to gluster-devel mailing list issue]

Apologies for the late reply.

glusterd uses its socket connection with brick processes (where io-stats 
xlator is loaded) to
gather information from io-stats via an RPC request. This facility is 
restricted to brick processes
as it stands today.

Some background ...
io-stats xlator is loaded, both in GlusterFS mounts and brick processes. So, 
we have the capabilities
to monitor I/O statistics on both sides. To collect I/O statistics at the 
server side, we have

# gluster volume profile VOLNAME [start | info | stop]
AND
#gluster volume top VOLNAME info [and other options]

We don't have a usable way of gathering I/O statistics (not monitoring, 
though the counters could be enhanced)
at the client-side, ie. for a given mount point. This is the gap 
glusterfsiostat aims to fill. We need to remember
that the machines hosting GlusterFS mounts may not have glusterd installed on 
them.

We are considering rrdtool as a possible statistics database because it seems 
like a natural choice for storing time-series
data. rrdtool is capable of answering high-level statistical queries on 
statistics that were logged in it by io-stats xlator
over and above printing running counters periodically.

Hope this gives some more clarity on what we are thinking.

thanks,
Krish
- Original Message -

 Probably me not understanding.

 the comment iostats making data available to glusterd over RPC - is what I
 latched on to. I wondered whether this meant that a socket could be opened
 that way to get at the iostats data flow.

 Cheers,

 PC

 - Original Message -

  From: Vipul Nayyar nayyar_vi...@yahoo.com

  To: Paul Cuzner pcuz...@redhat.com, Krishnan Parthasarathi
  kpart...@redhat.com

  Cc: Vijay Bellur vbel...@redhat.com, gluster-devel
  gluster-de...@nongnu.org

  Sent: Thursday, 20 February