Hi Ignasi,
I tried the code u suggested. I'm getting some error now like that file
what we have passed cannot be created in chef client. Please suggest.
[2015-02-11T18:18:32+00:00] INFO: Client key /etc/chef/client.pem is not
present - registering
^[[0m
================================================================================^[[0m
^[[31mChef encountered an error attempting to create the client
"aricloud-172.31.9.180"^[[0m
================================================================================^[[0m
^[[0m[2015-02-11T18:18:32+00:00] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
[2015-02-11T18:18:32+00:00] ERROR: The configured ssl_ca_file
/etc/chef/chef-server.crt does not exist
[2015-02-11T18:18:32+00:00] FATAL: Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)
-------- here is my full code.
private String createBootstrapAWS(String userName, String password,
InstanceBean instance, ComputeService computeService)
throws IOException {
String status = Constants.SUCCESS;
List<String> runlist = new ArrayList<String>();
String recipe = INITIAL_UPDATE_VM;
// initialising chef
ChefContext chefContext = AriServiceConfiguration
.chefServerConfigurationCloudWise(instance);
Iterable<? extends CookbookVersion> cookbookVersions = chefContext
.getChefService().listCookbookVersions();
if (any(cookbookVersions, containsRecipe(recipe))) {
runlist = new RunListBuilder().addRecipe(recipe).build();
}
if (instance.isAutoDeploy()) {
runlist = new RunListBuilder().addRecipes(
instance.getRecipes().toArray(
new String[instance.getRecipes().size()])).build();
}
LoginCredentials login = getLoginForCommandExecution(userName,
password, instance.getCloudType());
LOG.info("got the login as :" + login);
ChefService chefService = initChefServiceAWS(
MessageTranslator.getMessage("chef.client"),
MessageTranslator.getMessage("chef.validator"));
BootstrapConfig build = null;
try {
build = BootstrapConfig.builder().runList(runlist).build();
chefService.updateBootstrapConfigForGroup(instance.getGroupName(),
build);
// Build the script that will bootstrap the node
Statement bootstrap = chefService
.createBootstrapScriptForGroup(instance.getGroupName());
LOG.info("bootstrap created...");
LOG.debug("Running run script");
//
ImmutableList.Builder<Statement> withCA = ImmutableList.builder();
// Upload the Chef Server certificate to the node
withCA.add(Statements.createOrOverwriteFile("/etc/chef/chef-server.crt",
Files.readLines(new File("/tmp/apache.crt"),
Charsets.UTF_8)));
// The last statement in the generated bootstrap is the Chef client run, so
// modify the
// Chef client configuration to add the CA configuration just before that
statement
StatementList statements = (StatementList)bootstrap;
withCA.addAll(statements.subList(0, statements.size() - 1));
withCA.add(Statements.appendFile("/etc/chef/client.rb",
Collections.singleton("ssl_ca_file\"/etc/chef/chef-server.crt\"")));
withCA.add(Iterables.getLast(statements));
Statement updatedBootstrap = new StatementList(withCA.build());
//
status = runScriptOnGroup(computeService, login, instance,
updatedBootstrap);
} catch (Exception e) {
LOG.warn("error thrown for bootstrap.." + e.getMessage());
return "failure";
} finally {
chefContext.close();
}
return status;
}
Regards,
Subhadip
9741779086
-------------------------------------------------------------------------------------------------------------------
On Tue, Feb 10, 2015 at 4:11 PM, Ignasi Barrera <n...@apache.org> wrote:
> Ok, you're hitting JCLOUDS-792 [1]. It is fixed in the latest snapshot.
>
> Basically, the Chef client 12 comes with SSL verification enabled by
> default, and since the node you deployed does not trust the Chef Server's
> certificate, it aborts.
>
> If you can't upgrade to the latest snapshot, then you can workaround it by
> modifying the generated bootstrap script. Please, note that this is not
> recommended at all, as the code below might break when upgrading jclouds,
> as it makes many assumptions that may not be true in future versions. Use
> it only as a temporal workaround until you can ugprade.
>
> You will need your server's certificate in a file. Say you have that stored
> locally (where the jclouds code runs) in "/tmp/chef-server.crt".
>
> Then you could use the follogin code to generate the bootstrap script you
> will pass to the "runScript" method:
>
> Statement originalBootstrap =
> chef.createBootstrapScriptForGroup(instance.getGroupName());
>
> // https://issues.apache.org/jira/browse/JCLOUDS-792
> // We have to install the Chef Server key before running the Chef client.
>
> ImmutableList.Builder<Statement> withCA = ImmutableList.builder();
>
> // Upload the Chef Server certificate to the node
> withCA.add(Statements.createOrOverwriteFile("/etc/chef/chef-server.crt",
> Files.readLines("/tmp/chef-server.crt",
> Charsets.UTF_8)));
>
> // The last statement in the generated bootstrap is the Chef client run, so
> modify the Chef
> // client configuration to add the CA configuration just before that
> statement
> StatementList statements = (StatementList) originalBootstrap;
> withCA.addAll(statements.subList(0, statements.size() - 1));
> withCA.add(Statements.appendFile("/etc/chef/client.rb",
> Collections.singleton("ssl_ca_file
> \"/etc/chef/chef-server.crt\"")));
> withCA.add(Iterables.getLast(statements));
>
> Statement updatedBootstrap = new StatementList(withCA.build());
>
>
> And this "updatedBootstrap" is the script you can use. Basically it uploads
> the Chef server file to the node, and modifies the generated statement to
> configure the client.rb *after* jclouds has created the file and *before*
> it executes the chef run.
>
>
> HTH!
>
>
>
> [1] https://issues.apache.org/jira/browse/JCLOUDS-792
>
> On 10 February 2015 at 10:08, Subhadip Bagui <i.ba...@gmail.com> wrote:
>
> > Hi Ignasi,
> >
> > I checked in the stdout.log that is generating in /tmp/jclouds-script.../
> > and its showing the below error.. Attaching the full log also. Any idea
> how
> > to fix this.
> >
> >
> > [2015-02-08T17:41:25+00:00] INFO: Chef-client pid: 7023
> > *[2015-02-08T17:41:28+00:00] INFO: Client key /etc/chef/client.pem is not
> > present - registering*
> > *[2015-02-08T17:41:29+00:00] ERROR: SSL Validation failure connecting to
> > host: 125.16.230.216 - SSL_connect returned=1 errno=0 state=SSLv3 read
> > server certificate B: certificate verify failed*
> >
> >
> >
> > Regards,
> > Subhadip
> >
> >
> -------------------------------------------------------------------------------------------------------------------
> >
> > On Tue, Feb 10, 2015 at 1:02 AM, Subhadip Bagui <i.ba...@gmail.com>
> wrote:
> >
> >> Hi,
> >>
> >> I've added the properties for customized chef as below.
> >> chefConfig.put(ChefProperties.CHEF_USE_OMNIBUS, false);
> >> chefConfig.put(ChefProperties.CHEF_VERSION,"11.16.4");
> >>
> >> But now it seems chef is not able to download from opscode and install
> >> the rpm. I waited for 1/2 hr and no files are getting downloaded for
> chef
> >> rpm. Please suggest.
> >>
> >>
> >> 10-02-2015 00:54:19,635 DEBUG [user thread 0]
> >> net.schmizz.sshj.connection.channel.AbstractChannel 374 - Sending
> channel
> >> request for `exec`
> >> 10-02-2015 00:54:19,635 DEBUG [user thread 0]
> >> net.schmizz.concurrent.Promise 164 - Awaiting <<chan#278 / chanreq for
> >> exec>>
> >> 10-02-2015 00:54:19,682 DEBUG [reader]
> >> net.schmizz.sshj.connection.channel.AbstractChannel 328 - Received
> window
> >> adjustment for 2097152 bytes
> >> 10-02-2015 00:54:19,682 DEBUG [reader]
> >> net.schmizz.sshj.connection.channel.Window 41 - Increasing by 2097152
> up to
> >> 2097152
> >> 10-02-2015 00:54:19,682 DEBUG [reader] net.schmizz.concurrent.Promise 78
> >> - Setting <<chan#278 / chanreq for exec>> to `SOME`
> >> 10-02-2015 00:54:20,327 DEBUG [reader]
> >> net.schmizz.sshj.connection.channel.AbstractChannel 316 - Got chan
> request
> >> for `exit-status`
> >> 10-02-2015 00:54:20,327 DEBUG [reader]
> >> net.schmizz.sshj.connection.channel.AbstractChannel 410 - Got EOF
> >> 10-02-2015 00:54:20,327 DEBUG [reader]
> >> net.schmizz.sshj.connection.channel.AbstractChannel 223 - Got close
> >> 10-02-2015 00:54:20,327 DEBUG [reader]
> >> net.schmizz.sshj.connection.channel.AbstractChannel 427 - Sending EOF
> >> 10-02-2015 00:54:20,327 DEBUG [reader]
> >> net.schmizz.sshj.connection.channel.AbstractChannel 289 - Sending close
> >> 10-02-2015 00:54:20,328 DEBUG [reader]
> >> net.schmizz.sshj.connection.ConnectionImpl 84 - Forgetting `session`
> >> channel (#278)
> >> 10-02-2015 00:54:20,328 DEBUG [reader] net.schmizz.concurrent.Promise 78
> >> - Setting <<chan#278 / close>> to `SOME`
> >> *10-02-2015 00:54:20,328 DEBUG [user thread 0]
> >> net.schmizz.sshj.common.StreamCopier 139 - 0.0 KiB transferred in 0.645
> >> seconds (0.0 KiB/s)*
> >> *10-02-2015 00:54:20,328 DEBUG [user thread 0]
> >> net.schmizz.sshj.common.StreamCopier 139 - 0.0 KiB transferred in 0.0
> >> seconds (NaN KiB/s)*
> >> 10-02-2015 00:54:20,328 DEBUG [user thread 0]
> >> net.schmizz.sshj.connection.ConnectionImpl 68 - Attaching `session`
> channel
> >> (#279)
> >> 10-02-2015 00:54:20,329 DEBUG [user thread 0]
> >> net.schmizz.concurrent.Promise 164 - Awaiting <<chan#279 / open>>
> >> 10-02-2015 00:54:20,458 DEBUG [reader]
> >> net.schmizz.sshj.connection.channel.AbstractChannel 125 - Initialized -
> <
> >> session channel: id=279, recipient=2, localWin=[winSize=2097152],
> >> remoteWin=[winSize=0] >
> >>
> >>
> >>
> >> Regards,
> >> Subhadip
> >> 9741779086
> >>
> >>
> -------------------------------------------------------------------------------------------------------------------
> >>
> >> On Mon, Feb 9, 2015 at 6:17 PM, Ignasi Barrera <n...@apache.org> wrote:
> >>
> >>> The Chef version properties will only work if you install Chef
> >>> directly from the Chef gems. Support or versions when using omnibus
> >>> was added in the latest snapshot so it is not available in 1.7.3.
> >>>
> >>> You can instruct jclouds to install Chef directly from the Chef gems
> >>> by configuring the Chef properties mentioned here (look at the end, in
> >>> the "Customize how Chef is installed" section):
> >>> http://jclouds.apache.org/guides/chef
> >>>
> >>> You can start trying with the following properties, and play with the
> >>> rest (such as the ruby version, and the update gem system ones) if
> >>> these properties alone don't work.
> >>>
> >>> overrides.setProperty(ChefProperties.USE_OMNIBUS, "false");
> >>> overrides.setProperty(ChefProperties.CHEF_VERSION, "11.16.4");
> >>>
> >>> On 9 February 2015 at 13:21, Subhadip Bagui <i.ba...@gmail.com> wrote:
> >>> > Hi Ignasi,
> >>> >
> >>> > Changing the JClouds version will be difficult for me as the code is
> >>> there
> >>> > in production. I will check with changing the chef version properties
> >>> and
> >>> > let u know.
> >>> >
> >>> >
> >>> > Regards,
> >>> > Subhadip
> >>> >
> >>> >
> >>>
> -------------------------------------------------------------------------------------------------------------------
> >>> >
> >>> > On Mon, Feb 9, 2015 at 4:22 PM, Ignasi Barrera <n...@apache.org>
> >>> wrote:
> >>> >
> >>> >> The configuration seems correct. Can you try using version
> >>> >> 2.0.0-SNAPSHOT and set the chef version I suggested, and see if it
> >>> >> works? Just to isolate the issue and see if it is actually a package
> >>> >> version.
> >>> >>
> >>> >> On 9 February 2015 at 11:43, Subhadip Bagui <i.ba...@gmail.com>
> >>> wrote:
> >>> >> > Hi Ignasi,
> >>> >> >
> >>> >> > Thanks for your reply.
> >>> >> >
> >>> >> > I'm using jclouds version 1.7.3 and I'm not setting any chef
> >>> version in
> >>> >> > code. Default it's taking that opscode-omnibus chef.
> >>> >> > I have cleaned the /var/chef dir and tried but getting the same
> >>> issue.
> >>> >> >
> >>> >> > Here is my code for ChefService creation. Please let me know what
> >>> needs
> >>> >> to
> >>> >> > be done.
> >>> >> >
> >>> >> > `private ChefService initChefServiceAWS(String client, String
> >>> validator)
> >>> >> {
> >>> >> > ChefContext context = null;
> >>> >> > try {
> >>> >> >
> >>> >> > String organization = "chef";
> >>> >> > String pemFile = MessageTranslator.getMessage("user.home")
> >>> >> > + "/.chef/" + client + ".pem";
> >>> >> > String credential = Files.toString(new File(pemFile),
> >>> >> > Charsets.UTF_8);
> >>> >> >
> >>> >> > // Provide the validator information to let the nodes to
> >>> >> > // auto-register themselves
> >>> >> > // in the Chef server during bootstrap
> >>> >> > String validatorPemFile =
> MessageTranslator.getMessage("user.home")
> >>> >> > + "/.chef/" + validator + ".pem";
> >>> >> > String validatorCredential = Files.toString(new File(
> >>> >> > validatorPemFile), Charsets.UTF_8);
> >>> >> >
> >>> >> > Properties chefConfig = new Properties();
> >>> >> > chefConfig.put(ChefProperties.CHEF_VALIDATOR_NAME, validator);
> >>> >> > chefConfig.put(ChefProperties.CHEF_VALIDATOR_CREDENTIAL,
> >>> >> > validatorCredential);
> >>> >> >
> >>> >> > ContextBuilder builder = ContextBuilder
> >>> >> > .newBuilder(organization)
> >>> >> > .credentials(client, credential)
> >>> >> > .overrides(chefConfig);
> >>> >> >
> >>> >> > LOG.debug(" initializing Chef Service " +
> builder.getApiMetadata());
> >>> >> >
> >>> >> > context = builder.buildView(ChefContext.class);
> >>> >> >
> >>> >> > } catch (Exception e) {
> >>> >> > LOG.warn("error reading private key ", e);
> >>> >> > return null;
> >>> >> > }
> >>> >> > return context.getChefService();
> >>> >> > }`
> >>> >> >
> >>> >> > Regards,
> >>> >> > Subhadip
> >>> >> >
> >>> >>
> >>>
> -------------------------------------------------------------------------------------------------------------------
> >>> >> >
> >>> >> > On Mon, Feb 9, 2015 at 1:53 PM, Ignasi Barrera <n...@apache.org>
> >>> wrote:
> >>> >> >
> >>> >> >> Could you also share the code you use to create the ChefContext?
> >>> Just
> >>> >> >> to make sure the credentials and validator certificates are
> >>> properly
> >>> >> >> configured. Thanks!
> >>> >> >>
> >>> >> >> On 9 February 2015 at 09:18, Ignasi Barrera <
> >>> ignasi.barr...@gmail.com>
> >>> >> >> wrote:
> >>> >> >> > It seems to be failing to install the Chef RPM (and that would
> >>> be an
> >>> >> >> > issue with the Chef package itself). To debug what is going on,
> >>> you
> >>> >> >> > can:
> >>> >> >> >
> >>> >> >> > * Log in to the node after the script execution failure.
> >>> >> >> > * Remove the "/etc/chef" directory, to cleanup everything
> >>> jclouds has
> >>> >> >> > created there.
> >>> >> >> > * In the user's home directory you'll find a symlink pointing
> to
> >>> the
> >>> >> >> > bootstrap script.
> >>> >> >> > * That bootstrap script leaves the stdout and stderr outputs in
> >>> >> >> > "/tmp". You can read them and see if you find more details
> about
> >>> the
> >>> >> >> > error.
> >>> >> >> > * Or you can manually run the script and see if you can see
> more
> >>> >> details.
> >>> >> >> >
> >>> >> >> > jclouds by default installs Chef using the Omnibus installer
> and
> >>> >> >> > installs the latest version of Chef. You can also try to force
> a
> >>> >> >> > concrete version of the Chef Client to be installed. To do
> that:
> >>> >> >> >
> >>> >> >> > * You must use jclouds 2.0.0-SNAPSHOT version (there is not a
> >>> release
> >>> >> >> > with the version support when using Omnibus).
> >>> >> >> > * Create the ChefContext adding the following property (this
> is a
> >>> >> >> > version of the Chef client I've tested recently):
> >>> >> >> > overrides.setProperty(ChefProperties.CHEF_VERSION,
> >>> "11.16.4-1");
> >>> >> >> >
> >>> >> >> >
> >>> >> >> > HTH!
> >>> >> >> >
> >>> >> >> > I.
> >>> >> >> >
> >>> >> >> > On 8 February 2015 at 19:07, Subhadip Bagui <i.ba...@gmail.com
> >
> >>> >> wrote:
> >>> >> >> >> Hi,
> >>> >> >> >>
> >>> >> >> >> I'm trying to bootstrap one predefined recipe as runlist
> through
> >>> >> JClouds
> >>> >> >> >> boot strap. The issue I'm facing is the authentication and
> >>> >> >> chef-validator
> >>> >> >> >> pem file all the generating correctly in ec2 node, but it's
> >>> failing
> >>> >> to
> >>> >> >> >> create the client.pem in the new node. For that the recipe is
> >>> not
> >>> >> able
> >>> >> >> to
> >>> >> >> >> run. Below is my code part following the example from link
> >>> >> >> >>
> >>> >> >>
> >>> >>
> >>>
> https://github.com/jclouds/jclouds-examples/blob/master/chef-basics/src/main/java/org/apache/jclouds/examples/chef/basics/MainApp.java
> >>> >> >> >>
> >>> >> >> >> Please let me know the whats the issue here.
> >>> >> >> >>
> >>> >> >> >>
> >>> >> >> >> private String createBootstrapAWS(String userName, String
> >>> password,
> >>> >> >> >>> InstanceBean instance, ComputeService computeService)
> >>> >> >> >>> throws IOException {
> >>> >> >> >>> String status = Constants.SUCCESS;
> >>> >> >> >>> List<String> runlist = new ArrayList<String>();
> >>> >> >> >>> String recipe = INITIAL_UPDATE_VM;
> >>> >> >> >>> // initialising chef
> >>> >> >> >>> ChefContext chefContext = AriServiceConfiguration
> >>> >> >> >>> .chefServerConfigurationCloudWise(instance);
> >>> >> >> >>> Iterable<? extends CookbookVersion> cookbookVersions =
> >>> chefContext
> >>> >> >> >>> .getChefService().listCookbookVersions();
> >>> >> >> >>> if (any(cookbookVersions, containsRecipe(recipe))) {
> >>> >> >> >>> runlist = new RunListBuilder().addRecipe(recipe).build();
> >>> >> >> >>> }
> >>> >> >> >>> if (instance.isAutoDeploy()) {
> >>> >> >> >>> runlist = new RunListBuilder().addRecipes(
> >>> >> >> >>> instance.getRecipes().toArray(
> >>> >> >> >>> new String[instance.getRecipes().size()])).build();
> >>> >> >> >>> }
> >>> >> >> >>> LoginCredentials login =
> getLoginForCommandExecution(userName,
> >>> >> >> >>> password, instance.getCloudType());
> >>> >> >> >>> LOG.info("got the login as :" + login);
> >>> >> >> >>> ChefService chefService = initChefServiceAWS(
> >>> >> >> >>> MessageTranslator.getMessage("chef.client"),
> >>> >> >> >>> MessageTranslator.getMessage("chef.validator"));
> >>> >> >> >>> BootstrapConfig build = null;
> >>> >> >> >>> try {
> >>> >> >> >>> build = BootstrapConfig.builder().runList(runlist).build();
> >>> >> >> >>>
> >>> chefService.updateBootstrapConfigForGroup(instance.getGroupName(),
> >>> >> >> >>> build);
> >>> >> >> >>> // Build the script that will bootstrap the node
> >>> >> >> >>> Statement bootstrap = chefService
> >>> >> >> >>> .createBootstrapScriptForGroup(instance.getGroupName());
> >>> >> >> >>> LOG.info("bootstrap created...");
> >>> >> >> >>> LOG.debug("Running run script");
> >>> >> >> >>> status = runScriptOnGroup(computeService, login, instance,
> >>> >> >> >>> bootstrap);
> >>> >> >> >>> } catch (Exception e) {
> >>> >> >> >>> LOG.warn("error thrown for bootstrap.." + e.getMessage());
> >>> >> >> >>> return "failure";
> >>> >> >> >>> } finally {
> >>> >> >> >>> chefContext.close();
> >>> >> >> >>> }
> >>> >> >> >>> return status;
> >>> >> >> >>> }
> >>> >> >> >>
> >>> >> >> >>
> >>> >> >> >>
> >>> >> >> >> The response I'm getting from JClouds for ExecResponse is
> >>> below.
> >>> >> It's
> >>> >> >> not
> >>> >> >> >> creating the client.pem. Please let me know what is the issue
> >>> here.
> >>> >> >> >>
> >>> >> >> >> ExecResponse execResponses1 = compute.runScriptOnNode(
> >>> >> >> >>> instance.getInstanceId(), command,
> >>> >> >> >>> overrideLoginCredentials(login).runAsRoot(true));
> >>> >> >> >>
> >>> >> >> >>
> >>> >> >> >> {output=Downloading Chef for el...
> >>> >> >> >> downloading
> >>> >> >> >>
> >>> >> >>
> >>> >>
> >>>
> https://www.opscode.com/chef/metadata?v=&prerelease=false&nightlies=false&p=el&pv=6&m=x86_64
> >>> >> >> >> to file /tmp/install.sh.2830/metadata.txt
> >>> >> >> >> trying wget...
> >>> >> >> >> url
> >>> >> >> >>
> >>> >> >>
> >>> >>
> >>>
> https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-12.0.3-1.x86_64.rpm
> >>> >> >> >> md5 3634d1a3b6ae2e5977361075da0f44cc
> >>> >> >> >> sha256
> >>> >> 0ec6162b9d0ca2b2016ff02781d84905f712d64c7a81d01b0df88f977832f310
> >>> >> >> >> downloaded metadata file looks valid...
> >>> >> >> >> downloading
> >>> >> >> >>
> >>> >> >>
> >>> >>
> >>>
> https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-12.0.3-1.x86_64.rpm
> >>> >> >> >> to file /tmp/install.sh.2830/chef-12.0.3-1.x86_64.rpm
> >>> >> >> >> trying wget...
> >>> >> >> >> Comparing checksum with sha256sum...
> >>> >> >> >> Installing Chef
> >>> >> >> >> installing with rpm...
> >>> >> >> >> Preparing...
> >>> >> >> >> ##################################################
> >>> >> >> >> chef
> >>> >> >> >> ##################################################
> >>> >> >> >> Thank you for installing Chef!
> >>> >> >> >> [2015-02-08T17:41:25+00:00] INFO: Forking chef instance to
> >>> >> converge...
> >>> >> >> >> [2015-02-08T17:41:25+00:00] INFO: *** Chef 12.0.3 ***
> >>> >> >> >> [2015-02-08T17:41:25+00:00] INFO: Chef-client pid: 7023
> >>> >> >> >> [2015-02-08T17:41:28+00:00] INFO:* Client key
> >>> /etc/chef/client.pem is
> >>> >> >> not
> >>> >> >> >> present - registering*
> >>> >> >> >> *, error=warning:
> /tmp/install.sh.2830/chef-12.0.3-1.x86_64.rpm:
> >>> >> Header
> >>> >> >> V4
> >>> >> >> >> DSA/SHA1 Signature, key ID 83ef826a: NOKEY*
> >>> >> >> >> *, exitStatus=1}*
> >>> >> >> >>
> >>> >> >> >>
> >>> >> >> >> Regards,
> >>> >> >> >> Subhadip
> >>> >> >> >>
> >>> >> >> >>
> >>> >> >>
> >>> >>
> >>>
> -------------------------------------------------------------------------------------------------------------------
> >>> >> >>
> >>> >>
> >>>
> >>
> >>
> >
>
