Re: [go-cd] flyweight - what is it and is it safe to delete?

[Ketan Padegaonkar]

*tl;dr;*

Yes the flyweight directory is safe to delete. Make sure to shutdown your
server to avoid any issues later. Depending on the number size of
materials, your server may take several minutes to clone these again at
startup.



*The long version*

For some centralized SCMs like subversion and TFS, it's possible to do an
update-check on a remote repository URL. For distributed SCMs like git and
mercurial, the only way to check if new there are any new commits (and see
the contents of the commits) is to clone the repository. These clones live
in the flyweight directory. The server currently does not purge the
contents of the flyweight directory when a material is removed.



On Thu, Jan 11, 2018 at 9:44 AM Gopal Singhal 
wrote:

> Hi,
>
> In our Go server, flyweight
> directory /var/lib/go-server/pipelines/flyweight has grown to over 200Gb
> with some directories even older than 2 years.
>
> Is it safe to remove old directories may be like any directory older than
> 6 months?
>
> What is the significance of flyweight directories. e.g.
> (3927bc06-d4a7-4a25-a6bc-4e3e1e96d07f)? What I could make out is it is
> somewhere related to materials and polling.
>
> Thanks,
> Gopal
>
> --
> You received this message because you are subscribed to the Google Groups
> "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-cd+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-cd] flyweight - what is it and is it safe to delete?

[Gopal Singhal]

Hi,

In our Go server, flyweight
directory /var/lib/go-server/pipelines/flyweight has grown to over 200Gb
with some directories even older than 2 years.

Is it safe to remove old directories may be like any directory older than 6
months?

What is the significance of flyweight directories. e.g.
(3927bc06-d4a7-4a25-a6bc-4e3e1e96d07f)? What I could make out is it is
somewhere related to materials and polling.

Thanks,
Gopal

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-cd] logback to a logstash server

[Ketan Padegaonkar]

This script 
worked for me, I was able to confirm that logs were sent to logback.

On Wed, Jan 10, 2018 at 8:51 PM 'Barry Greenwood' via go-cd <
go-cd@googlegroups.com> wrote:

> Thanks for the update
>
> That folder didnt exist on our server, I have added with go permissions,
> I've added the files but it still isnt working.
>
> as this isnt the first issue i've had regarding this recently implemented
> feature can you please confirm that it works and should the logs show it
> being loaded ?
>
> Many thanks
>
> Barry
>
>
> On Wednesday, 10 January 2018 13:56:09 UTC, Ketan Padegaonkar wrote:
>
>> The `libs` dir is relative to the working directory of the server/agent.
>> This should typically be `/var/lib/go-server/libs` or
>> `/var/lib/go-agent/libs`. I'll update the documentation to clarify this.
>>
>>
>> On Wed, Jan 10, 2018 at 6:29 PM 'Barry Greenwood' via go-cd <
>> go...@googlegroups.com> wrote:
>>
> Hi all
>>>
>>> I've now got logback able to log to local files with access requests and
>>> other logs, I am now hoping to ship these to an external logstash server
>>>
>>> reading
>>> https://docs.gocd.org/current/advanced_usage/logging.html#advanced-logging-features
>>> this should be possible, however its a little abiguous with "download
>>> all logstash-logback-encoder jars and dependencies into libs dir:"
>>>
>>> I presume its a java library folder as these are jar files but testing
>>> with these in several places that are in the java.library.path, it isn't
>>> working.
>>>
>>> Has anyone else tried to get this working and had success ?
>>>
>>> Regards
>>>
>>> Barry
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "go-cd" group.
>>>
>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to go-cd+un...@googlegroups.com.
>>
>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-cd+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-cd] Re: Fan-In from multiple Git repos.

[Prasanna]

Thanks for the tip Varsha. I was able to resolve the problem.

1. I have to add `Deploy_Orchestrator_CI` as a dependency to 
`Deploy_Orchestrator_Pre_Prod`.

2. Even after this, I had issues of multiple deployments whenever there was 
a new material available in `git-repo1 or git-repo2 or git-repo3`. I have 
to add ignore patters on all the Service tests pipelines to ignore the 
material changes from git.

Here is the updated VSM of `Deploy_Orchestrator_Pre_Prod`




Thanks Ketan and Varsha for the comments.


- Prasanna

On Wednesday, January 10, 2018 at 5:03:34 AM UTC+1, Varsha Varadarajan 
wrote:
>
> Hi,
>
> From the attached screenshot, it looks like fan-in is not setup at all. As 
> per the docs  
> - Fan-in material resolution will ensure that a pipeline triggers only when 
> all its upstream pipelines have triggered off the *same version of an 
> ancestor pipeline* or material.  You can have the functional tests in 
> different git repositories. If you want fan in to kick in for 
> `Deploy_Orchestrator_Pre_Prod`, then the ancestor pipeline 
> `Deploy_Orchestrator_CI` must be made the upstream material for 
> `Deploy_Orchestrator_Pre_Prod` also. So, fan-in resolution will happen only 
> when the `Deploy_Orchestrator_Pre_Prod` and the functional tests pipeline 
> have a common material (be it a scm or a pipeline dependency) and for a 
> specific revision (git sha or pipeline build), all three functional tests 
> pipelines run first and then for the same revision, 
> `Deploy_Orchestrator_Pre_Prod` will run.
>
> Thanks,
> Varsha
>
> On Wednesday, 10 January 2018 02:56:45 UTC+5:30, Prasanna wrote:
>>
>>
>> Hi,
>>
>> I was referring this doc for fan-in 
>>  configuration 
>> of GoCD to set up something like the image shown below.
>>
>>
>>
>> 
>>
>>
>> I was expecting that "Deploy_Orchestrator_Pre_Prod" to be called once 
>> after all the 3 functional tests run successfully. However, each successful 
>> runs of the functional tests are triggering a deployment.
>>
>> So, if one of the functional tests fails, still the deployment happens 
>> twice.
>>
>>
>> One thing that may be an issue here is that all 3 functional tests 
>> pipelines have their own git repository. Is it the case? How can I go about 
>> resolving this? 
>>
>>
>> I would like to deploy only after all 3 build is green. Let me know if 
>> you need further details.
>>
>>
>> Thank you,
>>
>> Prasanna
>>
>>
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-cd] logback to a logstash server

['Barry Greenwood' via go-cd]

Thanks for the update

That folder didnt exist on our server, I have added with go permissions, 
I've added the files but it still isnt working.

as this isnt the first issue i've had regarding this recently implemented 
feature can you please confirm that it works and should the logs show it 
being loaded ?

Many thanks

Barry

On Wednesday, 10 January 2018 13:56:09 UTC, Ketan Padegaonkar wrote:
>
> The `libs` dir is relative to the working directory of the server/agent. 
> This should typically be `/var/lib/go-server/libs` or 
> `/var/lib/go-agent/libs`. I'll update the documentation to clarify this.
>
>
> On Wed, Jan 10, 2018 at 6:29 PM 'Barry Greenwood' via go-cd <
> go...@googlegroups.com > wrote:
>
>> Hi all
>>
>> I've now got logback able to log to local files with access requests and 
>> other logs, I am now hoping to ship these to an external logstash server
>>
>> reading 
>> https://docs.gocd.org/current/advanced_usage/logging.html#advanced-logging-features
>>  
>> this should be possible, however its a little abiguous with "download 
>> all logstash-logback-encoder jars and dependencies into libs dir:"
>>
>> I presume its a java library folder as these are jar files but testing 
>> with these in several places that are in the java.library.path, it isn't 
>> working.
>>
>> Has anyone else tried to get this working and had success ?
>>
>> Regards
>>
>> Barry
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to go-cd+un...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-cd] logback to a logstash server

[Ketan Padegaonkar]

The `libs` dir is relative to the working directory of the server/agent.
This should typically be `/var/lib/go-server/libs` or
`/var/lib/go-agent/libs`. I'll update the documentation to clarify this.


On Wed, Jan 10, 2018 at 6:29 PM 'Barry Greenwood' via go-cd <
go-cd@googlegroups.com> wrote:

> Hi all
>
> I've now got logback able to log to local files with access requests and
> other logs, I am now hoping to ship these to an external logstash server
>
> reading
> https://docs.gocd.org/current/advanced_usage/logging.html#advanced-logging-features
> this should be possible, however its a little abiguous with "download all
> logstash-logback-encoder jars and dependencies into libs dir:"
>
> I presume its a java library folder as these are jar files but testing
> with these in several places that are in the java.library.path, it isn't
> working.
>
> Has anyone else tried to get this working and had success ?
>
> Regards
>
> Barry
>
> --
> You received this message because you are subscribed to the Google Groups
> "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-cd+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-cd] logback to a logstash server

['Barry Greenwood' via go-cd]

Hi all

I've now got logback able to log to local files with access requests and 
other logs, I am now hoping to ship these to an external logstash server

reading 
https://docs.gocd.org/current/advanced_usage/logging.html#advanced-logging-features
 
this should be possible, however its a little abiguous with "download all 
logstash-logback-encoder jars and dependencies into libs dir:"

I presume its a java library folder as these are jar files but testing with 
these in several places that are in the java.library.path, it isn't working.

Has anyone else tried to get this working and had success ?

Regards

Barry

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-cd] trying to configure logback and access logs

['Barry Greenwood' via go-cd]

Deeper digging has allowed me to confirm nothing was appearing in the file 
due to it being not a new server, the issue with failing to log was related 
to an old version of the jetty.xml config file.

I can now log to files successfully, next step will be to send the data to 
a logstash server as well as log locally

Many thanks for help with this.

 Barry


On Thursday, 4 January 2018 17:02:18 UTC, Barry Greenwood wrote:
>
> Hi
>
>  I have tied these its a little better in the file is now being created, 
>
> but i'm still not getting any access logs in the file
>
> i've tried both
> cat /var/lib/go-server/config/logback-include.xml
> 
> 
>class="ch.qos.logback.core.rolling.RollingFileAppender">
> /var/log/go-server/lbinclude.log
>  class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
>   lbinclude.%d{-MM-dd}.log.gz
> 
> 7
> 1GB
> true
> 
>   %date{ISO8601} - %-4relative [%thread] %-5level %logger{35} 
> - %msg%n
> 
>   
>
>   
>
>   
> 
>   
>
> 
>
> and 
>
>
>   
> 
>   
>
> the first generates some data in the file the second leaves the file empty.
>
> logback access comes from the documentation on 
> https://logback.qos.ch/access.html which is linked from 
> https://github.com/logstash/logstash-logback-encoder
>
> Many thanks
>
> Barry
>
> On Tuesday, 2 January 2018 19:24:20 UTC, Aravind SV wrote:
>>
>> Just FYI. Yes, it looks like doing this will work:
>>
>> mkdir config
>> chown go:go config
>> ln -s /etc/go/logback-include.xml /var/lib/go-server/config/
>>
>> On Tue, Jan 2, 2018 at 11:17 AM, Aravind SV  
>> wrote:
>>
>>> Hello Barry,
>>>
>>> On Tue, Jan 2, 2018 at 9:49 AM, 'Barry Greenwood' via go-cd <
>>> go...@googlegroups.com> wrote:
>>>
 Thanks for the help however i am still having issues.

 /etc/default/logback-include.xml as described but with a with a file 
 appender in it does not create the file, that implies this isn't being read

>>>
>>> You're right. It's a bug. It'll be fixed in the next release (probably 
>>> through this PR ). Meanwhile, 
>>> temporarily, can you see if moving the file logback-include.xml into 
>>> /var/lib/go-server/config/ works?
>>>
>>> Here's what I tried, to ensure that this works:
>>>
>>> $ ls -l /var/lib/go-server/config/
>>> total 4
>>> -rw-r--r-- 1 go go 759 Jan  2 18:58 logback-include.xml
>>> $ cat /var/lib/go-server/config/logback-include.xml
>>> 
>>> 
>>>   >> class="ch.qos.logback.core.rolling.RollingFileAppender">
>>> /var/log/go-server/test.log
>>> 
>>>   %date{ISO8601} %-5level [%thread] %logger{0}:%line - 
>>> %msg%n
>>> 
>>>
>>> >> class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
>>>   
>>> /var/log/go-server/test.log.%d{-MM-dd}.%i.gz
>>>   10 MB
>>>   10
>>>   512 MB
>>> 
>>>   
>>>
>>>   
>>> 
>>>   
>>> 
>>>
>>> Once 18.1 is released, this will stop working and you can move it back 
>>> to its correct place, /etc/go/logback-include.xml. Or, you can see if a 
>>> symbolic link from /var/lib/go-server/config/logback-include.xml to 
>>> /etc/go/logback-include.xml works.
>>>
>>> Cheers,
>>> Aravind
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.