Subject: Pipeline Storage Performance Work Available For Beta - Want to kick the tires?

2017-12-21 Thread Samuel Van Oort 
Subject: Pipeline Storage Performance Work Available For Beta - Want to 
kick the tires?

Hey all, I've just released a set of plugin betas to the Experimental 
Update Center.  They have enhancements to Pipeline which CAN dramatically 
reduce I/O use and improve performance.  Please given them a try and report 
back how they work out for you.

Please note: to maintain existing behavior, these changes are OPT-IN. You 
MUST enable them to see a difference (see below).

The settings themselves have their own explanations (tooltips and help 
info), but the below gives more info.

*Will it help me?*
* Yes, if you are running complex Pipelines or Pipelines with many steps.
* Yes, if your Jenkins instance uses NFS, magnetic storage, runs many 
Pipelines at once, or shows high iowait.
* No, if your Pipelines spend almost all their time waiting for shell/batch 
scripts to run.  This isn't a magic "go fast" button for everything (I 
wish!).
* No, if you are not using Pipelines, or your system is loaded down by 
other factors.

*How do I get it?*
* You need to be on Jenkins LTS 2.73+ or higher (or a weekly 2.62+)
* Enable the experimental update center - instructions here: 
https://jenkins.io/blog/2013/09/23/experimental-plugins-update-center/
* Check for plugins updates
* You should see and install updates for the following plugins, with 
versions including the word "durability"
- Pipeline: API (workflow-api)
- Pipeline: Groovy (workflow-cps)
- Pipeline: Job (workflow-job)
- Pipeline: Supporting APIs (workflow-support)
- Pipeline: Multibranch (workflow-multibranch)
* Restart the master to use the updated plugins - note: you need all of 
them to take advantage.

*What does it do?*

This adds a performance/durability setting for Pipelines.  If you use the 
performance-optimized mode, disk writes are reduced significantly. This 
lets you improve Pipeline performance greatly (reduce I/O) at some cost to 
the running Pipelines' ability to survive if Jenkins falls over completely 
(durability).  Stability of Jenkins ITSELF is not changed, nor are there 
changes to completed Pipelines.

We also add the ability to mark Pipelines to NOT resume upon restart (a 
requested feature) - available under the properties at the top.

*How do I USE it?*

Durability settings need to be enabled (and will display in the logs when a 
job begins), either globally or per Pipeline/branch (MultiBranch). Settings 
take effect the next time the Pipeline runs. 

There are 3 ways to configure the durability setting:

**Globally**, you can choose a durability setting under "Manage Jenkins > 
Configure System", labelled "Pipeline Speed/Durability Settings".  These 
settings will take effect for Pipelines upon the next run, unless you 
override them with one of the below settings

**Per pipeline job:** at the top of the job configuration, labelled "Custom 
Pipeline Speed/Durability Level" - this overrides the global setting.  Or, 
use a "properties" step - the setting will apply to the NEXT run after the 
step is executed (same result).

**Per branch for a multibranch project:** configure a custom Branch 
Property Strategy (under the SCM) and add a property for Custom Pipeline 
Speed/Durability Level.  This overrides the global setting. 


*What are the settings?*

* Performance optimized mode ("PERFORMANCE_OPTIMIZED") - Greatly reduces 
disk I/O but running Pipelines with lower durability settings may lose 
runtime data IF they do not finish AND Jenkins is not shut down 
gracefully.  If this happens, they behave like FreeStyle builds (logs, but 
no steps to visualize). Details at bottom.

* Maximum durability ("MAX_SURVIVABILITY") - behaves just like Pipeline did 
before, slowest option.  Use this for running your most critical Pipelines.

* Less durable, a bit faster ("SURVIVABLE_NONATOMIC") - Writes data with 
every step but avoids atomic writes. On some filesytems, especially 
networked ones (i.e. NFS), this is faster than maximum durability mode, but 
it carries a small extra risk (details at bottom).


*Nitty-gritty details*

Remember: worst-case behavior reverts to something like FreeStyle builds -- 
Pipelines that cannot persist data may not be able to resume or displayed 
in Blue Ocean/Stage View/etc, but will have logs.

Running pipelines with the performance-optimized setting may lose data IF 
they do not finish AND Jenkins is not shut down gracefully. A "graceful" 
shutdown is where Jenkins goes through a full shutdown process, such as 
visiting http://[jenkins-server]/exit or using one of the gentler signals 
to kill the process.  A "dirty" shutdown is where the Jenkins process dies 
without doing shutdown tasks -- killing a Docker container or using "kill 
-9" to terminate the Java process will do this. 

The less-durable/a bit faster setting avoids atomic writes -- what this 
means is that if the Operating System fails, data that is buffered for 
writing to disk will not be flushed and will be lost.  This is quite

Jenkins Hackfest after FOSDEM

2017-12-21 Thread Mark Waite 
On 5 Feb 2018, the Monday after FOSDEM ends, the usual suspects of Jenkins
contributors  will be
gathering for a Jenkins Hackfest.  If you'd like to join us for a day of
development focused on Jenkins, RSVP HERE

,

RSVP by Monday, 15 January, please.  We'll be coordinating logistics and
working to have a great time working on the Jenkins code.

Meals, snacks, and beverages will be provided for the hackfest.  Come join
us, and let's write some code!

Questions? feel free to contact Alyssa Tong (alyton...@gmail.com) or Mark
Waite (mark.earl.wa...@gmail.com) directly.

Mark Waite

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGfSytuQJm9K%2Bw%2BrTfkuGSGr4LsZBUfHaz5tqLf1sXnbQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [JEP-200 Discussion] - Switch Remoting/XStream blacklist to a whitelist

2017-12-21 Thread Jesse Glick 
On Thu, Dec 21, 2017 at 1:38 PM, Oleg Nenashev  wrote:
> The main purpose of whitelisting for Remoting is to allow particular
> communications between agent and master by saying the class is secure to be
> sent.

Received. And/or loaded from XML files.

> For example, Jesse added several JGit classes in Jenkins PR #3120. Do
> we want these classes to be stored on the disk or submitted in config files?

IIRC some of these were in fact used in certain cases in `build.xml`
files, but I may not be remembering that right.

> Hell no, it may cause behavior and performance issues in Jenkins.

Which “behavior and performance issues” are you referring to? They
sound unrelated to security and are best addressed, if required, by
plugin developers on their own initiative.

The purpose of the JEP-200 whitelist is to protect the master JVM from
deserialization exploits (originally coming from CLI clients, now
coming from compromised agents or REST/CLI calls to upload XML). These
could come equally from Java Remoting or XStream, usually with the
same classes. (Sometimes only from XStream since there is no
requirement to implement `Serializable`; anyway is it harmless to
Remoting to include an entry which can be exploited only via XStream.)

If JBoss Marshalling were exposed to user actions, the class filter
would be applied to it as well. But `program.dat` is only ever written
by the master to begin with, and `script-security` should block
programmatic creation of payloads.

> currently there is no way to restrict classes for Remoting-only or
> XStream-only.
>
> So, I would like to put the following proposal on the table:
>
> Allow passing the "SerializationType" (or context) to the CustomClassFilter.
> Modify the implementation to pass this context

I can think of no reason to do that. It just adds complexity and
confuses the issue.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1EVa8C%2BPX2CbCOcvE4NTZosp_N3i2VbzTar4BpFVcxsw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [JEP-200 Discussion] - Switch Remoting/XStream blacklist to a whitelist

2017-12-21 Thread Oleg Nenashev 
While we are doing the JEP review in the pull requests, I would also want 
to start one topic here. The current JEP-200 design shares the whitelist 
between Remoting and XStream, and I am a bit aware of that.

The main purpose of whitelisting for Remoting is to allow particular 
communications between agent and master by saying the class is secure to be 
sent. For example, Jesse added several JGit classes in Jenkins PR #3120 
. Do we want these classes 
to be stored on the disk or submitted in config files? Hell no, it may 
cause behavior and performance issues in Jenkins. But currently there is no 
way to restrict classes for Remoting-only or XStream-only.

So, I would like to put the following proposal on the table:

   - Allow passing the "SerializationType" (or context) to the 
   CustomClassFilter. Modify the implementation to pass this context
  - Remoting ClassFilter blacklist should stay global by default, I'd 
  guess
  - Create 2 new CustomClassFilter implementations, which operate only 
   for Remoting or XStream
   - Split whitelist resource files:
  - Remoting: META-INF/hudson.remoting.ClassFilter 
  - XStream: META-INF/ihudson.util.XStream2.ClassFilter (or so)
   
It will allow managing the whitelists individually in plugins if needed.

If we eventually have another type of serialization filtering (e.g. for 
JBoss Marshalling in Pipeline), we will be able to extend the engine easily 
as well.

SerializationType could be made an internal and Restricted extension point 
in such case.


WDYT?

BR, Oleg 


понедельник, 18 декабря 2017 г., 13:41:20 UTC+1 пользователь Oleg Nenashev 
написал:
>
> Hi all,
>
> I am starting this thread in order to collect extra feedback about 
> JEP-200, which proposes switching Remoting/XStream implementations from a 
> blacklist to a whitelist. The intention is to significantly reduce risks of 
> class deserialization attacks, which was hitting Jenkins project seriously 
> over last 2 years (e.g. SECUIRTY-429 this April 
> ). This JEP is accepted 
> as a draft, and the current state is published here 
> . 
>
> I am assigned as a BDFL Delegate who makes a decision about 
> accepting/rejecting this Jenkins Enhancement Proposal (see JEP-1 
>  for more info about 
> the process). Over the next week I will be reviewing this JEP and providing 
> feedback in this thread and in pull requests. 
>
> I also call other interested contributors to comment regarding this JEP. 
> It is important, because the proposal implies a *high risk *of 
> regressions in plugins and other Jenkins components. The JEP sponsor made a 
> significant amount of testing, but there may be some gaps. Any feedback and 
> extra testing of the reference implementation will be appreciated.
>
> There are several ways to provide the feedback:
>
>- Comment in this thread
>- Create a pull request with document edits
>- Ping me (oleg-nenashev) and Jesse Glick (jglick) in IRC
>
> My current plan is to finalize the Draft reviews/edits by December 30 
> though it depends on the sponsor's availability during the Christmas break 
> if there is a discussion needed. If you have any comments or interest to 
> review the JEP deeper, please respond by this date.
>
>
> Best regards,
> Oleg Nenashev
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/376328d9-fdf7-4595-975f-8fe6afdd5967%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Some questions on plugin development

2017-12-21 Thread Ullrich Hafner 

> Am 21.12.2017 um 15:40 schrieb Jesse Glick :
> 
> On Wed, Dec 20, 2017 at 4:05 PM, Jesse Glick  wrote:
>>> Preferrably without
>>> some UI, I am trying to keep things simple.
>> 
>> `GlobalConfiguration`


Since your plugin is related to the static analysis suite:
See 
https://github.com/jenkinsci/analysis-core-plugin/blob/master/src/main/java/hudson/plugins/analysis/core/GlobalSettings.java
 


-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/7E83B707-E6F4-4A38-A2CC-88A3E4112D22%40gmail.com.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: Message signed with OpenPGP

Re: Secrets in cloudbees builds of plugins

2017-12-21 Thread Jesse Glick 
On Wed, Dec 20, 2017 at 6:36 PM, Radek Antoniuk
 wrote:
> Resurrecting an old thread, Jesse, did anything change?
> On the current infra, is there any way to provide securely the token for
> coveralls?

On ci.jenkins.io? No; only site administrators are given the ability
to define credentials, pending JENKINS-27398.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3Zag%2BuAxsapMa%3Das%2Bg_ZXv-q74gTjZYp-oW_yerQLUHg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re:

2017-12-21 Thread Jesse Glick 
On Thu, Dec 21, 2017 at 7:49 AM, Miguelangel Fernandez
 wrote:
> I've written a Jenkins plugin to implement my own authentication mechanism
> by extending hudson.security.SecurityRealm. Now I'm trying to create my own
> custom "Access Denied" page

FWIW I do not recommend you even attempt this. Implement a simple
`AbstractPasswordBasedSecurityRealm` (in the unlikely event there is
not already one able to connect to your backend authentication system)
and leave it at that.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1frWHzL3XmUVTsbF2hLvzkuOmvV7%2B5KOTHFsao9JTA-Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Some questions on plugin development

2017-12-21 Thread Jesse Glick 
On Wed, Dec 20, 2017 at 4:05 PM, Jesse Glick  wrote:
>> Preferrably without
>> some UI, I am trying to keep things simple.
>
> `GlobalConfiguration`

BTW

https://github.com/jenkinsci/archetypes/blob/master/README.md#usage

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2EGrj-V15u0%2BrGHtdOZXuSt-WkjFAmjf6v3L0o7e5H_Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re:

2017-12-21 Thread Daniel Beck 

> On 21. Dec 2017, at 13:49, Miguelangel Fernandez  
> wrote:
> 
> How can I make the static content in my plugin accessible without 
> authentication? or public?

Only specific URLs are allowed to be accessed without Overall/Read permission, 
and arbitrary resource files from plugins aren't. One of the URLs that work is 
/securityRealm, so the currently selected security realm is exposed, allowing 
users without any permissions to interact with the security realm. That means 
you can add a getter or similar with some wiring that serves your resource file 
programmatically.

When put next to the views of the security realm, the following addition to 
MySecurityRealm will serve the file at /securityRealm/logo
---
public void doLogo(StaplerRequest req, StaplerResponse rsp) throws IOException, 
ServletException {
URL resource = 
MySecurityRealm.class.getResource("MySecurityRealm/logo.png");
rsp.serveFile(req, resource);
}
---

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/D0912882-63C1-4C17-92E1-B37373EF%40beckweb.net.
For more options, visit https://groups.google.com/d/optout.

[no subject]

2017-12-21 Thread Miguelangel Fernandez 
Hi everyone,

I've written a Jenkins plugin to implement my own authentication mechanism
by extending hudson.security.SecurityRealm. Now I'm trying to create my own
custom "Access Denied" page, to display when a user types in the wrong
password or simply doesn't have access. To do this, I've created a file
called accessDenied.jelly. So let's say the name of my company is "Foo" and
the structure of my plugin (simplified here for practical reasons) is this:

src
|---main
|---java
|   |---com
|   |---foo
|   |---authn
|   |---MySecurityRealm.java
|---resources
|   |---com
|   |---foo
|   |---authn
|   |---MySecurityRealm
|   |---config.jelly
|   |---accessDenied.jelly
|---webapp
|---images
|---foo.png

Now, I want to display my company logo -The foo.png file in the tree above-
in my custom error page accessDenied.jelly. Again, for the sake of
simplicity let's assume the short name of my plugin is simply foo and this
is the content of accessDenied.jelly:





Access Denied

You don't have access to this Jenkins console.






My problem here is I'm getting a broken image link for foo.png because the
server returns a HTTP 403 Forbidden on serving it. Implying the user must
be authenticated in order to access the static content in a plugin.

If instead of my company logo I use ${app.getRootUrl()}/favicon.ico for the
source of my image, I do get the Jenkins icon displayed.

How can I make the static content in my plugin accessible without
authentication? or public?

Notice I tried setting permission="${app.ANONYMOUS}" in my Jelly file, but
this refers to the view being rendered, not to any static assets referenced
by it.
If anyone would like the extra rep points in StackOverflow, I've posted the
exact same question here:

https://stackoverflow.com/q/47830458/4124574

Thank you,

  Miguelángel Fernández M.

Life is what happens to you while you're busy making other plans.


--John Lennon.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CAK2td25JiJ%3DoMzUYBvg%2BH_17nfm0zREpM%3DYnJuKaORHJNDoyng%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.