[Gluster-devel] Outreachy intern's update.

[Menaka Mohan]

Hello all,


I am Menaka M from India, an Outreachy intern working under Gluster. My tasks 
for this internship is to work on the parts of this feature page - Initial 
feature description for improved io-stats [1].


I am working under the mentorship of Shyam and the program co-ordinator Amye.


So far, I have spent time on understanding the Gluster architecture and also 
about the io-stats xlator. I have also contributed few changes to the same 
under the mentor's guidance.


Initially, I have found the missing FOPs in it and then added them to it as my 
initial commit. The commit has been reviewed and merged. [2]


For adding the missing FOPs, I added the compound FOP to the list of FOPs in 
the generator.py file but later understood the problems behind that. The same 
was also explained in the commit by the community members. [3]


And then, I got a task assigned to add the code generation framework for the 
io-stats xlator. The review of the same commit is pending. [4]


My next immediate task is to do the following changes,


For now, for each FOP we get Min/Avg/Max values on profiling. I shall be 
working to also display a histogram based spread as it is just few OPs that is 
taking higher time skewing the average, etc., I shall work on that and other 
parts that are mentioned in the feature page and post regular updates in this 
mailing list.


[1] 
https://review.gluster.org/#/c/16558/
 , Feature page

[2] https://review.gluster.org/#/c/16490/ , Adding missing FOPs

[3] https://review.gluster.org/#/c/16547/ , Adding the compound FOP to 
generator file.

[4] https://review.gluster.org/#/c/16586/ , Adding Code-gen framework to 
io-stats translator


Thank you all,

Menaka M.

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Need help in understanding IOZone config file

[Menaka Mohan]

Hello,


Thank you so much for helping me in understanding it.  I have read more about 
that and with the help of other community members, I submitted a pull request 
to update it in the Gluster docs [1] for better understanding which was 
accepted and merged.


Since each record in the config file represents a thread in IOZone terminology, 
we may need to have as many records with respect to the number of threads 
defined and number of clients in our setup.


[1] : 
https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Performance%20Testing/


Thanks and Regards,

Menaka M


From: Dustin Black <dbl...@redhat.com>
Sent: Monday, October 17, 2016 8:16:27 PM
To: Menaka Mohan
Cc: gluster-devel@gluster.org
Subject: Re: [Gluster-devel] Need help in understanding IOZone config file

Hi Menaka,

We test Gluster with something like this using iozone for 4GB file sizes.

iozone -t 6 -s 4g -r 4m -+m clients.ioz -c -e -+z -+n -i 0 && iozone -t 6 -s 4g 
-r 4m -+m clients.ioz -c -e -+z -+n -i 1


Where the clients.ioz file might look like the below for 6 clients/workers:

c0 /path/to/gluster/client/mount /path/to/binary/iozone
c1 /path/to/gluster/client/mount /path/to/binary/iozone
c2 /path/to/gluster/client/mount /path/to/binary/iozone
c3 /path/to/gluster/client/mount /path/to/binary/iozone
c4 /path/to/gluster/client/mount /path/to/binary/iozone
c5 /path/to/gluster/client/mount /path/to/binary/iozone


Dustin Black, RHCA
Senior Architect, Software-Defined Storage
Red Hat, Inc.


On Mon, Oct 10, 2016 at 3:48 PM, Menaka Mohan 
<menak...@outlook.com<mailto:menak...@outlook.com>> wrote:

Hi,


I am Menaka M. I am new to this open source world. Kindly help me with the 
following query.


I have set up the Gluster development environment with two servers and one 
client. I am trying to run the basic bench test on the Gluster cluster from 
this GitHub 
repo<https://github.com/gluster/gbench/tree/master/bench-tests/bt--0001>. I 
also have IOZone installed. In that, how to generate the clients.ioz file 
(prerequisite 3) ? Does that refer to the file containing (client_name   
work_dirpath_to_IOZone_on_client) ?


I have read multiple blogs on how to analyze the IOZone results and also the 
performance testing section in docs. Kindly help me resolve this confusion. If 
I had asked a very basic thing, apologies. I will quickly learn them.


Regards,

Menaka M

___
Gluster-devel mailing list
Gluster-devel@gluster.org<mailto:Gluster-devel@gluster.org>
http://www.gluster.org/mailman/listinfo/gluster-devel

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] [Outreachy] Results of IOZone and smallfile bench tests with some changes.

[Menaka Mohan]

Hello,


I have been able to successfully run the IOZone bench test on the Gluster 
setup. Kindly find the results here [1].


But for the smallfile distributed I/O benchmark, the test fails. Since we are 
taking 5 samples in each operation, the " --operation create " fails reporting 
that the file already exits from the second sample [1462.758118, None, None, 
None, None] . Necessary changes are to be made such that the --operation create 
happens only once and others ( --operation read and --operation ls-l ) run as 
usual. Kindly correct me, If I am wrong.


Assuming that as the reason, I have made the necessary changes to the code 
[sample _size for create operation is 1] and ran the bench test. Kindly find 
the results here [2].


I will further work on it as a single script and post the output as mentioned 
in the GitHub page. Since this is a very time consuming process, I will 
simultaneously work on the next task which is to investigate the tools to use 
and analyze the system resource usage across the setup.


[1]   https://paste.fedoraproject.org/451045/55349814/
[2]   https://paste.fedoraproject.org/451953/14765795/


Regards,

Menaka M
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Outreachy] Need help with running bench test on Gluster cluster.

[Menaka Mohan]

Hello Shyam,

Thanks a lot for such a detailed reply. I am new to this and was trying hard to 
learn and got your help in the right time. After getting your reply, I have 
experimented more and following are the findings.

I am clear with all the pre-requisites except only one. I searched for any 
-easy to understand- examples online, but couldn't find any. My clients.ioz 
file has the "hostname  directory iozone_pathname". But where to specify the 
number of threads per client ? In the recent commit, Number of threads are set 
to 4 in the script. But how to do that while running our own iozone command ?

So far, I am unable to find the cause of the following errors.


1)  extract_iozone_resuslt function returning None

2)  Line 13 in sync-drop-caches.sh " source=${BASH_SOURCE[0]}" returns Bad 
substitution.

But I am able to run the iozone commands in the script on my own and get the 
results.

1)  # IOZone seq writes [https://paste.fedoraproject.org/449217/14763288/]

2)  # IOZone seq reads [https://paste.fedoraproject.org/449218/14763289/]

3)  # IOZone rand read / write 
[http://paste.fedoraproject.org/449220/76329399/]

Since we are getting the expected results, I assume covering all the 
pre-requisites rightly except the above mentioned one.

While I run the actual gbench script, I get the following log. [ 
https://paste.fedoraproject.org/449225/63310741/ ]
As I have checked, I also have bash shell as the default shell. [ 
https://paste.fedoraproject.org/449224/30577147/ ]

I am also trying to find the cause of the errors to my knowledge and will 
report any progress soon. I have made few syntactic corrections in the code and 
also updated the readme as you said in the previous email and generated a pull 
request. Kindly review it.

Now, Please let me know what all you expect in the application other than my 
understanding and the timeline and few initial contributions.

Note: I looked into few articles to reply properly into a thread in a mailing 
list. I changed the subject as Re: Re: Sub. Still, If I am unable to place it 
rightly in the thread, I am sorry. I will look into it.

Regards,
Menaka M


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Need help with running bench test on Gluster cluster.

[Menaka Mohan]

Hello Nigel B,

Thank you so much for you reply. Recently, I have noticed few syntax errors on 
the code and I have corrected them. Kindly review the Pull request.

Since I do not have a laptop with good specifications, I am unable to use VMs. 
I tried and it crashed many times. So with the support of a college friend I 
have setup the environment in a lab and accessing it through Bitvise SSH 
client. With the process becoming more complex, I am learning more things.

I have thoroughly gone through Shyam's reply and working on it.

Regards,
Menaka M


On Tue, Oct 11, 2016 at 07:53:09PM +, Menaka Mohan wrote:
> Hi,
>
>
> I am Menaka M. I am interested in participating in this round of Outreachy 
> under Gluster. I am new to this open source world. Kindly help me with the 
> following query.
>
>
> I have set up the Gluster development environment with two servers and one 
> client. I am trying to run the basic bench test on the Gluster cluster from 
> this GitHub 
> repo<https://github.com/gluster/gbench/tree/master/bench-tests/bt--0001>. 
> I also have IOZone installed. While trying to run the provided script, i get 
> the following error. I was trying to identify the cause of the error. Kindly 
> help me with that.
>
> So, I learned more about IOZone and also the performance testing section in 
> the Gluster docs. With that knowledge and to learn more, I have gone through 
> the code and running iozone commands mentioned in the GlusterBench.py script 
> individually.
>
> If I am asking a very basic thing, apologies. I will quickly learn things.
>
> Regards,
> Menaka M
>

Hi Menaka,

Thanks for taking a look at gbench. It appears that you have uncovered a bug in 
our documentation and code.

Shyam is the expert here and I'll let him chime in. You need an environment 
variable with CLIENTS and SERVER to get this working. So when you run this, you 
need at least two machines (VMs would be best. Happy to guide you on how to set 
this up if you'd like). It would look something like this:

CLIENTS="192.168.0.1 192.158.0.2" SERVERS="192.168.0.3 192.168.0.3" python 
GlusterBench.py -s 64 -f 1 -n 5 -v

I'm making a lot of assumptions here. If there's anything that is confusing, 
let me know and I'm happy to clarify. I'll let Shyam jump in later today (he's 
in EST) to chime with anything else I've missed.

--
nigelb
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Need help with running bench test on Gluster cluster.

[Menaka Mohan]

Hi,


I am Menaka M. I am interested in participating in this round of Outreachy 
under Gluster. I am new to this open source world. Kindly help me with the 
following query.


I have set up the Gluster development environment with two servers and one 
client. I am trying to run the basic bench test on the Gluster cluster from 
this GitHub 
repo. I 
also have IOZone installed. While trying to run the provided script, i get the 
following error. I was trying to identify the cause of the error. Kindly help 
me with that.

So, I learned more about IOZone and also the performance testing section in the 
Gluster docs. With that knowledge and to learn more, I have gone through the 
code and running iozone commands mentioned in the GlusterBench.py script 
individually.

If I am asking a very basic thing, apologies. I will quickly learn things.

Regards,
Menaka M

--

python GlusterBench.py -s 64 -f 1 -n 5 -v

Number threads = 4
Client list = HadoopSlave4

Running IOZone with 64KB record size and 4 threads,  Creating an 8 GB file with 
every thread.
Running smallfile with 64KB files, creating 1 files.
Running squential IOZone tests, starting with sequential writes.

About to gather sample --> 0

Iozone: Performance Test of File I/O
Version $Revision: 3.429 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
 Al Slater, Scott Rhine, Mike Wisner, Ken Goss
 Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
 Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
 Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,

Adding the current sample to the list: None
Dropping cache
/root/sync-drop-caches.sh: 13: /root/sync-drop-caches.sh: Bad substitution
Cleaning up files.

About to gather sample --> 1

/* I removed the Iozone and Contributors header */

Adding the current sample to the list: None
Dropping cache
/root/sync-drop-caches.sh: 13: /root/sync-drop-caches.sh: Bad substitution
Cleaning up files.

About to gather sample --> 2

Adding the current sample to the list: None
Dropping cache
/root/sync-drop-caches.sh: 13: /root/sync-drop-caches.sh: Bad substitution
Cleaning up files.

About to gather sample --> 3

Adding the current sample to the list: None
Dropping cache
/root/sync-drop-caches.sh: 13: /root/sync-drop-caches.sh: Bad substitution
Cleaning up files.

About to gather sample --> 4

Adding the current sample to the list: None
Dropping cache
/root/sync-drop-caches.sh: 13: /root/sync-drop-caches.sh: Bad substitution
No cleanup on 4 iteration.

The results for sequential writes are: [None, None, None, None, None]

Traceback (most recent call last):
  File "GlusterBench.py", line 255, in 
main()
  File "GlusterBench.py", line 76, in main
average_seq_write = find_average(result1)
  File "GlusterBench.py", line 250, in find_average
total = total + int(samples_in[x])
TypeError: int() argument must be a string or a number, not 'NoneType'

-
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel