Re: Proposal to add new 'export artifacts' GFSH command

[Swapnil Bawaskar]

+1 I like the idea. However, we will have to make sure that the regex is
limited to the server's working directory only.

On Mon, Jan 30, 2017 at 5:23 PM Jens Deppe  wrote:

> What about simply having --exclude and --include options that take a regex
> in order to exclude and include arbitrary files as per the users' choosing.
>
> --Jens
>
> On Mon, Jan 30, 2017 at 3:33 PM, Swapnil Bawaskar 
> wrote:
>
> > On Mon, Jan 30, 2017 at 10:15 AM, Udo Kohlmeyer 
> > wrote:
> >
> > > I think stats should always be included by default
> >
> >
> > +1 stats should be included always. If they are not needed, users can use
> > --include-stats=false.
> > I'm not sure I like the --stats-only flag since the command becomes
> "export
> > logs --stats-only", which seems a bit counterintuitive.
> >
>

Re: Proposal to add new 'export artifacts' GFSH command

[Swapnil Bawaskar]

On Mon, Jan 30, 2017 at 10:15 AM, Udo Kohlmeyer 
wrote:

> I think stats should always be included by default


+1 stats should be included always. If they are not needed, users can use
--include-stats=false.
I'm not sure I like the --stats-only flag since the command becomes "export
logs --stats-only", which seems a bit counterintuitive.

Re: Proposal to add new 'export artifacts' GFSH command

[Udo Kohlmeyer]

I think stats should always be included by default

maybe a '--stats-only' or '--logs-only'  flag...


On 1/30/17 10:08, Kevin Duling wrote:

+1 to the idea of '--include-stats'

If --include-stats is offered, there should also be a '--stats-only' flag.
That way you can get logs, logs and stats, or just stats.

On Fri, Jan 27, 2017 at 11:30 AM, Swapnil Bawaskar 
wrote:




There are some inconsistencies in the destinations of the current

GFSH

export commands.  Some commands (e.g. 'export logs’) export to a

given

directory on the executing member (server or locator).   Other

commands

(e.g. ‘export cluster-configuration’) export to a given directory on

the

GFSH client machine. (This makes far more sense, and is what I would
have
expected as a user.)

Me too :)
Rather than introduce yet another command, I would like to suggest that we
change existing 'export logs' to actually do the export gfsh client
machine. This would also mean we can use --start-time and --end-time.
For stats, I think we should introduce a flag '--include-stats' to the
export logs command.

Re: Proposal to add new 'export artifacts' GFSH command

[Kevin Duling]

+1 to the idea of '--include-stats'

If --include-stats is offered, there should also be a '--stats-only' flag.
That way you can get logs, logs and stats, or just stats.

On Fri, Jan 27, 2017 at 11:30 AM, Swapnil Bawaskar 
wrote:

> >
> >
> > 
> >  There are some inconsistencies in the destinations of the current
> GFSH
> >  export commands.  Some commands (e.g. 'export logs’) export to a
> given
> >  directory on the executing member (server or locator).   Other
> > commands
> >  (e.g. ‘export cluster-configuration’) export to a given directory on
> > the
> >  GFSH client machine. (This makes far more sense, and is what I would
> >  have
> >  expected as a user.)
> >
>
> Me too :)
> Rather than introduce yet another command, I would like to suggest that we
> change existing 'export logs' to actually do the export gfsh client
> machine. This would also mean we can use --start-time and --end-time.
> For stats, I think we should introduce a flag '--include-stats' to the
> export logs command.
>

Re: Proposal to add new 'export artifacts' GFSH command

[Jared Stewart]

The ‘export logs’ command deals with this by having —start-time and —end-time 
parameters that can be used to specify a particular time window.  I’m not sure 
whether statistics files roll over and have timestamps or not.  This could be 
one approach to limiting the size of the data returned for ‘export artifacts'.  
We could also check the size of the artifacts before sending them back, and 
prompt for confirmation from the user if it was going to exceed some given 
threshold.  

> On Jan 26, 2017, at 2:02 PM, Udo Kohlmeyer  wrote:
> 
> I agree with dan... We need to be able to limit how much data is being 
> collected and sent.
> 
> In many cases clients don't clean up the files so they might drag a lot of 
> stuff over.
> 
> 
> On 1/26/17 13:52, Dan Smith wrote:
>> +1 for export artifacts and +1 for zip-file
>> 
>> I'm also generally in favor of gathering the artifacts on the gfsh client
>> but I am a little concerned that in some cases that could be a lot of data
>> going to the client (10s or 100s of GBs), especially if logs and stats
>> aren't being cleaned up on a regular basis. Maybe we need a cap on the
>> amount of data sent back?
>> 
>> -Dan
>> 
>> On Thu, Jan 26, 2017 at 1:35 PM, John Blum  wrote:
>> 
>>> +1
>>> 
>>> On Thu, Jan 26, 2017 at 1:05 PM, Jared Stewart 
>>> wrote:
>>> 
 I would like to propose a new ‘export artifacts’ GFSH command, as well as
 some changes to existing GFSH export commands.  (See
 https://geode.apache.org/docs/guide/tools_modules/gfsh/
 command-pages/export.html  for a list of
 the current commands.)
 
 There are some inconsistencies in the destinations of the current GFSH
 export commands.  Some commands (e.g. 'export logs’) export to a given
 directory on the executing member (server or locator).   Other commands
 (e.g. ‘export cluster-configuration’) export to a given directory on the
 GFSH client machine. (This makes far more sense, and is what I would have
 expected as a user.)  I propose to reconcile all export commands to
>>> target
 an export directory on the GFSH client machine, rather than on the
 executing cluster member.
 Export cluster-configuration (which exports the existing cluster
 configuration of a cluster to a zip file) has parameters for both
 —zip-file-name and —dir.  I think it makes more sense to have one
 parameter, say —zip-file, that can take either a relative path to a zip
 file (“—zip-file=myExport.zip”, “—zip-file=logs/myExport.zip") or an
 absolute path including directories (“—zip-file=/logs/myExport.zip”)
 rather than having two separate parameters.
 I propose to add an ‘export artifacts’ GFSH command that would export all
 log files and stat files from all the members of a cluster to a single
>>> zip
 file on the GFSH client machine.  This would provide a convenient
>>> mechanism
 for an administrator to collect together all of the files necessary to
 troubleshoot problems in their Geode cluster.
 
 - Jared
>>> 
>>> 
>>> 
>>> --
>>> -John
>>> john.blum10101 (skype)
>>> 
> 

Re: Proposal to add new 'export artifacts' GFSH command

[Udo Kohlmeyer]

I agree with dan... We need to be able to limit how much data is being 
collected and sent.


In many cases clients don't clean up the files so they might drag a lot 
of stuff over.



On 1/26/17 13:52, Dan Smith wrote:

+1 for export artifacts and +1 for zip-file

I'm also generally in favor of gathering the artifacts on the gfsh client
but I am a little concerned that in some cases that could be a lot of data
going to the client (10s or 100s of GBs), especially if logs and stats
aren't being cleaned up on a regular basis. Maybe we need a cap on the
amount of data sent back?

-Dan

On Thu, Jan 26, 2017 at 1:35 PM, John Blum  wrote:


+1

On Thu, Jan 26, 2017 at 1:05 PM, Jared Stewart 
wrote:


I would like to propose a new ‘export artifacts’ GFSH command, as well as
some changes to existing GFSH export commands.  (See
https://geode.apache.org/docs/guide/tools_modules/gfsh/
command-pages/export.html  for a list of
the current commands.)

There are some inconsistencies in the destinations of the current GFSH
export commands.  Some commands (e.g. 'export logs’) export to a given
directory on the executing member (server or locator).   Other commands
(e.g. ‘export cluster-configuration’) export to a given directory on the
GFSH client machine. (This makes far more sense, and is what I would have
expected as a user.)  I propose to reconcile all export commands to

target

an export directory on the GFSH client machine, rather than on the
executing cluster member.
Export cluster-configuration (which exports the existing cluster
configuration of a cluster to a zip file) has parameters for both
—zip-file-name and —dir.  I think it makes more sense to have one
parameter, say —zip-file, that can take either a relative path to a zip
file (“—zip-file=myExport.zip”, “—zip-file=logs/myExport.zip") or an
absolute path including directories (“—zip-file=/logs/myExport.zip”)
rather than having two separate parameters.
I propose to add an ‘export artifacts’ GFSH command that would export all
log files and stat files from all the members of a cluster to a single

zip

file on the GFSH client machine.  This would provide a convenient

mechanism

for an administrator to collect together all of the files necessary to
troubleshoot problems in their Geode cluster.

- Jared




--
-John
john.blum10101 (skype)

Re: Proposal to add new 'export artifacts' GFSH command

[Dan Smith]

+1 for export artifacts and +1 for zip-file

I'm also generally in favor of gathering the artifacts on the gfsh client
but I am a little concerned that in some cases that could be a lot of data
going to the client (10s or 100s of GBs), especially if logs and stats
aren't being cleaned up on a regular basis. Maybe we need a cap on the
amount of data sent back?

-Dan

On Thu, Jan 26, 2017 at 1:35 PM, John Blum  wrote:

> +1
>
> On Thu, Jan 26, 2017 at 1:05 PM, Jared Stewart 
> wrote:
>
> > I would like to propose a new ‘export artifacts’ GFSH command, as well as
> > some changes to existing GFSH export commands.  (See
> > https://geode.apache.org/docs/guide/tools_modules/gfsh/
> > command-pages/export.html  > docs/guide/tools_modules/gfsh/command-pages/export.html> for a list of
> > the current commands.)
> >
> > There are some inconsistencies in the destinations of the current GFSH
> > export commands.  Some commands (e.g. 'export logs’) export to a given
> > directory on the executing member (server or locator).   Other commands
> > (e.g. ‘export cluster-configuration’) export to a given directory on the
> > GFSH client machine. (This makes far more sense, and is what I would have
> > expected as a user.)  I propose to reconcile all export commands to
> target
> > an export directory on the GFSH client machine, rather than on the
> > executing cluster member.
> > Export cluster-configuration (which exports the existing cluster
> > configuration of a cluster to a zip file) has parameters for both
> > —zip-file-name and —dir.  I think it makes more sense to have one
> > parameter, say —zip-file, that can take either a relative path to a zip
> > file (“—zip-file=myExport.zip”, “—zip-file=logs/myExport.zip") or an
> > absolute path including directories (“—zip-file=/logs/myExport.zip”)
> > rather than having two separate parameters.
> > I propose to add an ‘export artifacts’ GFSH command that would export all
> > log files and stat files from all the members of a cluster to a single
> zip
> > file on the GFSH client machine.  This would provide a convenient
> mechanism
> > for an administrator to collect together all of the files necessary to
> > troubleshoot problems in their Geode cluster.
> >
> > - Jared
>
>
>
>
> --
> -John
> john.blum10101 (skype)
>