Hi Dharam-
Understood.
My apologies, I should have clarified that when I talked about
"externalizing" the configuration of Geode into a file (outside of *Spring*
config).
I also imagined combining that with a build-time step that would set the
Geode configuration properties in the file with environment-specific
settings.
Given the following *Spring* (XML) configuration meta-data...
<*util**:properties **id=**"geodeProperties" **location=*
*"file:/resource/path/to/geode.properties"/*>
The *geode.properties* file would still contain (for example)...
# geode.properties
LOCATORS=${app.geode.server.dev.default.locators}
And then, the properties file would have values injected at build-time
using, for example...
https://maven.apache.org/plugins/maven-resources-plugin/examples/filter.html
Of course, that would require a change to your build/deployment-time
process as an interim solution, which may not be ideal.
Anyway, the fix for GEODE-8085 will be part of the Apache Geode 1.6 release.
Regards,
John
On Thu, Apr 19, 2018 at 9:40 PM, Thacker, Dharam <
[email protected]> wrote:
> Thank you so much Bruce/John! Really appreciate quick follow ups.
>
>
>
> John,
>
>
>
> <*util**:properties **id=**"geodeProperties" **location=*
> *"file:/resource/path/to/geode.properties"*>
>
>
>
> This will still not work for me. I am using dynamic properties which will
> get resolved using JVM args for different server instances.
>
>
> As an example,
>
>
> <prop key="name">${SERVER_NAME:MyGeodeServer}</prop>
> <prop key="mcast-port">0</prop>
> <prop key="locators">${LOCATORS:localhost[10334]}</prop>
> <prop key="jmx-manager">false</prop>
> <prop key="jmx-manager-start">false</prop>
> <prop
> key="http-service-bind-address">${HTTP_SERVICE_BIND_ADDRESS:localhost}</prop>
>
> <prop key="http-service-port">${HTTP_SERVICE_PORT:9301}</prop>
>
> We have only one spring boot uber jar code base with which we start
> multiple server instances.
> java -jar geode-server.jar -DSERVER_NAME=server-1
> -DLOCATORS=host1[10334],host2[10334]
>
> As you can see from above, those properties are environment specific
> currently, which in real world comes from some templatized model and gets
> appended with server start up command (java -jar geode-server.jar
> $JAVA_OPTS)
>
> *Template snippet:*
>
> app.geode.server.dev.default.locators=host-d1[10334],host-d2[10334]
>
> app.geode.server.uat.default.locators=host-d3[10334],host-d4[10334]
>
> app.geode.server.default.maxHeap=12G
>
> app.geode.server.default.extendedJvmArgs=-XX:UseG1GC +so many
>
>
>
> *Start command:*
>
> startApplication –propertyPrefix=app.geode.server –env=dev
> –servername=eventserver1 [So many overrides as per need]
>
> *Note*: startApplication is generic startup shell script which helps to
> create JAVA_OPTS from template.
>
> Thanks,
>
> Dharam
>
>
>
> *From:* Bruce Schuchardt [[email protected]]
> *Sent:* Friday, April 20, 2018 1:01 AM
> *To:* John Blum
> *Cc:* Thacker, Dharam; [email protected]
>
> *Subject:* Re: AuthenticationRequiredException on force disconnection
>
>
>
> This might work for a server. The locator sets an override configuration
> on the DistributedSystem that it creates, so files won't be accessed and
> any system properties will be ignored.
>
>
>
> On 4/19/18 12:07 PM, John Blum wrote:
>
> HI Dharam-
>
>
>
> Alternatively, (*@Bruce* - please confirm), that this problem does not
> exist when a user uses *Gfsh* to launch the server. Let's explore why
> that is.
>
>
>
> Well, this is only the case because when using the *Gfsh*, `start server`
> command, the properties (GemFire and Security properties) are typically
> "externalized", as in, put into separate (properties) files, e.g.
> gemfire.properties and gfSecurity.properties, respectively.
>
>
>
> Therefore, it would seem that Geode is able to reconnect when the
> (GemFire/Security) properties are actually kept in files since,
> non-apparently, Geode will *re-read* these files when it reconstructs the
> *Cache* (and to be more precise, the *DistributedSystem*, which is
> responsible for reading the configuration, which it delegates to an
> instance of *DistributedConfig*).
>
>
>
> Some people are quick to point out that this is only a problem when using
> SDG. However, that is simply not true and those people are not comparing
> apples to apples when using SDG to configure/bootstrap your Geode server vs
> *Gfsh*. SDG would be more equivalent to the following (i.e. using the
> public API)...
>
>
>
> Cache cache = *new* CacheFactory()
>
> .set("*locators*", "...")
>
> .set("*security-manager*", "...")
>
> .set("*security-username*", "...")
>
> .set("*security-password*", "...")
>
> .set("*disable-auto-reconnect*", "false")
>
> ...
>
> .create();
>
>
>
> I guarantee you, before the fix for GEODE-5085 [1], using the Geode public
> API would not work either, because this is exactly what SDG does (*new*
> CacheFactory(gemfireProperties)), where "gemfireProperties" are your "
> *geodeProperties*" properties specified in *geode-server-context.xml*...
>
>
>
> <*util**:properties **id=**"geodeProperties"*>
>
> ....
>
> </*util**:properties*>
>
>
>
>
>
> *So, what does this mean for your Spring configure/bootstrapped Geode
> server; How do you work around this problem?*
>
>
>
> Well, all the properties you specified in ""*geodeProperties*" simply
> need to be externalized so that Geode re-reads in the properties on a
> reconnect.
>
>
>
> <*util**:properties **id=**"geodeProperties" **location=*
> *"file:/resource/path/to/geode.properties"*>
>
>
>
> Fortunately, with *Spring*, these properties can come from any "location"
> as supported by *Spring's* Resource abstraction [2] by using the resource
> location specifiers (e.g. 'classpath:', 'file:', etc as seen in *Table 10
> - Resource strings* of [3]).
>
>
>
> Simply put all the Geode properties you specified in the
> geode-server-context.xml into a properties file, specify the location of
> this properties file in <*util**:properties*> using the *location*
> attribute, and you should be in business.
>
>
>
> Hope this helps util the fix for GEODE-5085 [1] is rolled out in Geode 1.6.
>
>
>
> Cheers,
>
> John
>
>
>
>
>
> [1] https://issues.apache.org/jira/browse/GEODE-5085
>
> [2] https://docs.spring.io/spring/docs/current/spring-
> framework-reference/core.html#resources
>
> [3] https://docs.spring.io/spring/docs/current/spring-
> framework-reference/core.html#resources-resourceloader
>
>
>
>
>
> On Thu, Apr 19, 2018 at 10:10 AM, Bruce Schuchardt <[email protected]>
> wrote:
>
> Hi Dharam
>
> This problem is fixed in the upcoming 1.6 release, which should be out
> later this month.
>
> As a workaround you could disable auto-reconnect (property
> disable-auto-reconnect) so that the process will just exit. Or you could
> get a build of the "develop" branch of the git repo and use that.
>
> The problem was that Geode was not retaining security settings along with
> other properties when it went into an auto-reconnect state. It had all of
> the other settings but security settings were lost.
>
>
>
> On 4/16/18 1:57 AM, Thacker, Dharam wrote:
>
> Hi John,
>
>
>
> *1.** What type of application are you building to configure your
> (embedded) peer cache? E.g. is this a Web application with an embedded
> peer cache instance? Or, are you simply using Spring to configure a peer
> cache, Geode server node, that is itself not an actual application?*
>
>
>
> It is client/server topology where both client & server nodes are
> configured using spring data geode.
>
> I am using spring data geode to simply configure geode server node
> [Attached actual config]
>
>
>
> *2.* I have enabled enable-auto-reconnect="true" as well.
>
>
>
> *3.* As per pulse and gfsh shell, node was thrown out of the
> distributed system as it does not show there.
>
>
>
> *4.* security-udp-dhalgo is coming as “******” on node restart
> where I am assuming that it might be setting some value by default. I am
> not sure as it’s masked.
>
>
>
>
>
> *Ok. I would request geode team to validate on issue part.*
>
>
>
> But the point for *spring data geode* is,
>
> >> *How to deal with such scenario where node is no more a part of
> distributed system but app is running?*
>
>
>
> Usually one would like to take some automated corrective action on such
> alerts.
>
>
>
> Thanks & Regards,
>
> Dharam
>
>
>
> *From:* John Blum [[email protected]]
> *Sent:* Monday, April 16, 2018 1:03 PM
> *To:* Thacker, Dharam
> *Cc:* [email protected]; Bruce Schuchardt
> *Subject:* Re: AuthenticationRequiredException on force disconnection
>
>
>
> Dharam-
>
>
>
> There is nothing *Spring* can do if Apache Geode is not respecting the
> setting for the Apache Geode property, "*security-udp-dhalgo*". That is
> all Apache Geode.
>
>
>
> Also, I suspect that the AuthenticationRequiredException is not bubbling
> out because somewhere Geode is eating the Exception in the chain of calls
> and simply logging it (at log level, "warning", no less, as in it is
> considered "fatal"). Though, I would point out that even if the Exception
> was propagated out that it is not necessarily going to cause the JVM to
> exit either. It all depends on the existing Thread types (e.g. non/daemon)
> and their state (e.g. BLOCKED).
>
>
>
> *What type of application are you building to configure your (embedded)
> peer cache? E.g. is this a Web application with an embedded peer cache
> instance? Or, are you simply using Spring to configure a peer cache, Geode
> server node, that is itself not an actual application?*
>
>
>
> A quick Thread dump on the JVM when this Exception occurs will quickly
> reveal what "non-daemon" Threads are blocking the JVM from shutting down,
> whether that is perhaps a Web application Thread (e.g. embedded Web
> server/container request processing Thread, etc) or a Geode Thread, holding
> up the JVM.
>
>
>
> Also keep in mind, that I deliberately disabled [1] the reconnecting state
> when using SDG, since I would assume that a user is using SDG to build an
> application using Apache Geode (whether as a client cache app, the common
> scenario, or even when building embedded peer cache applications). As
> such, a user would need to explicitly enable this Geode property, for
> instance, when the user is only using Spring to configure a peer cache,
> Geode server. To enable this "reconnectable" state in an application is
> futile since, if a Geode node gets "disconnected", then all Geode object
> references (from the Cache to Regions, to anything else) are not "stale".
> Therefore, if you have injected references to Geode objects into your
> application components (e.g. like a Region into a DAO/Repo), then you are
> going to have problems.
>
>
>
> As far as reporting the state, you can determine whether your peer node
> (application) is still connected by querying... cache.
> getDistributedSystem()
> <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/GemFireCache.html#getDistributedSystem-->
> [2].isConnected()
> <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/distributed/DistributedSystem.html#isConnected-->
> [3], or even isReconnecting()
> <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/distributed/DistributedSystem.html#isReconnecting-->
> [4]. As far as trying to determining whether an
> AuthenticationRequiredException has happened, great question!
>
>
>
> If I think of anything, I will let you know.
>
>
>
> Regards,
>
> John
>
>
>
>
>
> [1] https://github.com/spring-projects/spring-data-geode/
> blob/2.0.6.RELEASE/src/main/java/org/springframework/data/
> gemfire/CacheFactoryBean.java#L204-L205
>
> [2] http://geode.apache.org/releases/latest/javadoc/org/
> apache/geode/cache/GemFireCache.html#getDistributedSystem--
>
> [3] http://geode.apache.org/releases/latest/javadoc/org/
> apache/geode/distributed/DistributedSystem.html#isConnected--
>
> [4] http://geode.apache.org/releases/latest/javadoc/org/
> apache/geode/distributed/DistributedSystem.html#isReconnecting--
>
>
>
>
>
> On Sun, Apr 15, 2018 at 11:08 PM, Thacker, Dharam <
> [email protected]> wrote:
>
> Thanks John/Bruce!
>
>
>
> But it does not work as expected.
>
>
>
> I tried setting (*security-udp-dhalgo=*) in properties file in both
> locators and servers.
>
> I also confirmed the same by verifying config level logs and using locator
> with following command which explicitly mentions that *“security-udp-dhalgo”
> is empty (gemfire.sys.security-udp-dhalgo =)*
>
>
>
> >> describe config –member=locator1
>
> >> describe config –member=server1
>
>
>
> But even after that, I see following exception which is same as before.
>
>
>
> More, it looks like that once GEODE server member reboot itself after
> force disconnection, it does not respect *security-udp-dhalgo* override
> in properties file (*My assumption based on logs*)
>
> I see *security-udp-dhalgo=********* in startup configuration *after
> member’s attempt to connect to distributed system*.
>
>
>
> [warning 2018/04/15 21:23:57.095 EDT event-server-1<ReconnectThread>
> tid=0x215] Exception occurred while trying to connect the system during
> reconnect
>
> org.apache.geode.security.AuthenticationRequiredException: Failed to find
> credentials from [host-001(event-server-1:32054)<ec>:1025]
>
> at org.apache.geode.distributed.internal.membership.gms.
> membership.GMSJoinLeave.attemptToJoin(GMSJoinLeave.java:424)
>
> at org.apache.geode.distributed.internal.membership.gms.
> membership.GMSJoinLeave.join(GMSJoinLeave.java:318)
>
> at org.apache.geode.distributed.internal.membership.gms.mgr.
> GMSMembershipManager.join(GMSMembershipManager.java:656)
>
> at org.apache.geode.distributed.internal.membership.gms.mgr.
> GMSMembershipManager.joinDistributedSystem(GMSMembershipManager.java:745)
>
> at org.apache.geode.distributed.internal.membership.gms.
> Services.start(Services.java:181)
>
> at org.apache.geode.distributed.internal.membership.gms.
> GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:102)
>
> at org.apache.geode.distributed.internal.membership.MemberFactory.
> newMembershipManager(MemberFactory.java:89)
>
> at org.apache.geode.distributed.internal.DistributionManager.<
> init>(DistributionManager.java:1112)
>
> at org.apache.geode.distributed.internal.DistributionManager.<
> init>(DistributionManager.java:1160)
>
> at org.apache.geode.distributed.internal.DistributionManager.
> create(DistributionManager.java:531)
>
> at org.apache.geode.distributed.internal.
> InternalDistributedSystem.initialize(InternalDistributedSystem.java:687)
>
> at org.apache.geode.distributed.internal.
> InternalDistributedSystem.newInstance(InternalDistributedSystem.java:299)
>
> at org.apache.geode.distributed.DistributedSystem.connect(
> DistributedSystem.java:202)
>
> at org.apache.geode.distributed.internal.
> InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2675)
>
> at org.apache.geode.distributed.internal.
> InternalDistributedSystem.tryReconnect(InternalDistributedSystem.
> java:2508)
>
> at org.apache.geode.distributed.internal.
> InternalDistributedSystem.disconnect(InternalDistributedSystem.java:983)
>
> at org.apache.geode.distributed.internal.DistributionManager$
> MyListener.membershipFailure(DistributionManager.java:4307)
>
>
>
> John,
>
>
>
> This does not kill GEODE application. It still runs as it is. This makes
> APM tool to assume that application is healthy and is not facing any issues.
>
> What do you suggest to rectify this?
>
>
>
> Is there any example if I can report state of GEODE server as
> “UNHEALTHY”/”DISCONNECTED”?
>
> Is there any example if I can listen to these notifications and come up
> with some health check service?
>
>
>
> Thanks & Regards,
>
> Dharam
>
>
>
> *From:* John Blum [[email protected]]
> *Sent:* Friday, April 13, 2018 2:59 AM
> *To:* [email protected]
> *Subject:* Re: AuthenticationRequiredException on force disconnection
>
>
>
> Regarding *Spring*, not really too differently actually, see here
> <https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/resources/spring-server-cache.xml#L24-L33>
> [1]
> (XML) and here
> <https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L66-L84>
> [2]
> (*JavaConfig*) (followed by this
> <https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L91>
> [3]
> and this
> <https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L96>
> [4]).
>
>
>
> There is even an Annotation-based approach
> <https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/annotation/server/AnnotationConfiguredGeodeServerApplication.java>
> [5]
> for the curious onlooker.
>
>
>
>
>
> [1] https://github.com/jxblum/contacts-application/blob/
> master/configuration-example/src/main/resources/spring-
> server-cache.xml#L24-L33
>
> [2] https://github.com/jxblum/contacts-application/blob/
> master/configuration-example/src/main/java/example/app/spring/java/server/
> JavaConfiguredGeodeServerApplication.java#L66-L84
>
> [3] https://github.com/jxblum/contacts-application/blob/
> master/configuration-example/src/main/java/example/app/spring/java/server/
> JavaConfiguredGeodeServerApplication.java#L91
>
> [4] https://github.com/jxblum/contacts-application/blob/
> master/configuration-example/src/main/java/example/app/spring/java/server/
> JavaConfiguredGeodeServerApplication.java#L96
>
> [5] https://github.com/jxblum/contacts-application/blob/
> master/configuration-example/src/main/java/example/app/
> spring/annotation/server/AnnotationConfiguredGeodeServerApplication.java
>
>
>
>
>
> On Thu, Apr 12, 2018 at 2:17 PM, Bruce Schuchardt <[email protected]>
> wrote:
>
> The setting merely causes Geode to encrypt packets sent over UDP.
>
>
>
> On 4/11/18 10:29 AM, Thacker, Dharam wrote:
>
>
> Would there be any negative impact on disabling 'security-udp-dhalgo' on
> peer to peer members or pulse or jmx notifications ?
>
> Thanks,
> Dharam
>
>
>
>
>
>
>
> --
>
> -John
>
> john.blum10101 (skype)
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
> ---------- Forwarded message ----------
> From: Bruce Schuchardt <[email protected]>
> To: "[email protected]" <[email protected]>
> Cc:
> Bcc:
> Date: Thu, 12 Apr 2018 21:15:21 +0000
> Subject: Re: AuthenticationRequiredException on force disconnection
>
> I'm not sure what your development context is so it's hard to answer that
> question. If you're programatically creating a cache then set the cache
> property ConfigurationProperties.SECURITY_UDP_DHALGO to an empty string.
> If you're using a properties file set it to blank.
>
> security-udp-dhalgo=
>
> -or-
>
> cachefactory.set(SECURITY_UDP_DHALGO, "")
>
> I don't recall how you set properties for the cache under Spring.
>
>
>
> On 4/11/18 11:44 PM, Thacker, Dharam wrote:
>
> Hello Bruce,
>
>
>
> I have not manually specified this property to enable udp encryption using
> "security-udp-dhalgo" anywhere. I am using TCP mode only.
>
>
>
> Is it by default enabled? If yes, how can I disable it?
>
>
>
> I could not find any documentation on it.
>
>
>
> Thanks,
>
> Dharam
>
>
>
> Sent with BlackBerry Work (www.blackberry.com)
> ------------------------------
>
> *From: *"Thacker, Dharam" <[email protected]>
> *Sent: *Apr 11, 2018 10:59 PM
> *To: *[email protected]
> *Subject: *RE: AuthenticationRequiredException on force disconnection
>
> Thank you Bruce!
>
>
>
> I will surely open a JIRA soon.
>
>
>
> "Geode sends membership information, alerts and on rare occasions a PDX
>
> registration message over UDP"
>
>
>
> Would there be any negative impact on disabling 'security-udp-dhalgo' on
> peer to peer members or pulse or jmx notifications ?
>
>
>
> Thanks,
>
> Dharam
>
>
>
> Sent with BlackBerry Work (www.blackberry.com)
> ------------------------------
>
> *From: *Bruce Schuchardt <[email protected]>
> *Sent: *Apr 11, 2018 8:45 PM
> *To: *[email protected]
> *Subject: *Re: AuthenticationRequiredException on force disconnection
>
> That looks like a bug in UDP encryption. Can you open a JIRA ticket to
> track this? Set the component to "membership". Looking at the unit
> test suite I don't think there is any coverage for auto-reconnect with
> security-udp-dhalgo enabled.
>
> As a workaround you could, if you're comfortable doing so, disable
> security-udp-dhalgo until this is fixed. There are other known issues
> with this fairly new setting that people have been working on recently.
>
> Geode sends membership information, alerts and on rare occasions a PDX
> registration message over UDP. No client/server messages are sent over
> UDP so its use is confined to your server cluster. No messages
> containing application objects (keys, values, callback args etc) are
> sent over UDP unless you set disable-tcp=true to disable use of tcp/ip
> stream sockets.
>
>
> On 4/11/18 4:38 AM, Thacker, Dharam wrote:
> > warning 2018/04/10 02:40:59.541 EDT event-server-1 <ReconnectThread>
> tid=0x217] Exception occurred while trying to connect the system during
> reconnect
> > org.apache.geode.security.AuthenticationRequiredException: Failed to
> find credentials from [host001(event-server-1:3525)<ec>:1026]
> > at org.apache.geode.distributed.internal.membership.gms.
> membership.GMSJoinLeave.attemptToJoin(GMSJoinLeave.java:424)
> > at org.apache.geode.distributed.internal.membership.gms.
> membership.GMSJoinLeave.join(GMSJoinLeave.java:318)
> > at org.apache.geode.distributed.internal.membership.gms.mgr.
> GMSMembershipManager.join(GMSMembershipManager.java:656)
> > at org.apache.geode.distributed.internal.membership.gms.mgr.
> GMSMembershipManager.joinDistributedSystem(GMSMembershipManager.java:745)
> > at org.apache.geode.distributed.internal.membership.gms.
> Services.start(Services.java:181)
> > at org.apache.geode.distributed.internal.membership.gms.
> GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:102)
> > at org.apache.geode.distributed.internal.membership.
> MemberFactory.newMembershipManager(MemberFactory.java:89)
> > at org.apache.geode.distributed.internal.DistributionManager.<
> init>(DistributionManager.java:1112)
> > at org.apache.geode.distributed.internal.DistributionManager.<
> init>(DistributionManager.java:1160)
> > at org.apache.geode.distributed.internal.DistributionManager.
> create(DistributionManager.java:531)
> > at org.apache.geode.distributed.internal.
> InternalDistributedSystem.initialize(InternalDistributedSystem.java:687)
> > at org.apache.geode.distributed.internal.
> InternalDistributedSystem.newInstance(InternalDistributedSystem.java:299)
> > at org.apache.geode.distributed.DistributedSystem.connect(
> DistributedSystem.java:202)
> > at org.apache.geode.distributed.internal.
> InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2675)
> > at org.apache.geode.distributed.internal.
> InternalDistributedSystem.tryReconnect(InternalDistributedSystem.
> java:2508)
> > at org.apache.geode.distributed.internal.
> InternalDistributedSystem.disconnect(InternalDistributedSystem.java:983)
> > at org.apache.geode.distributed.internal.DistributionManager$
> MyListener.membershipFailure(DistributionManager.java:4307)
> > at org.apache.geode.distributed.internal.membership.gms.mgr.
> GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1530)
> > at org.apache.geode.distributed.internal.membership.gms.mgr.
> GMSMembershipManager.lambda$forceDisconnect$0(GMSMembershipManager.java:
> 2550)
> > at java.lang.Thread.run(Thread.java:745)
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
>
>
>
>
> --
>
> -John
>
> john.blum10101 (skype)
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
>
>
> --
>
> -John
>
> john.blum10101 (skype)
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
--
-John
john.blum10101 (skype)
