Re: 4.0 best feature/fix?

Fri, 07 May 2021 09:37:14 -0700 

My bad.  It's V4, not v4.  :)

Works fine with V4.  Spews errors on V5.

-Joe

On 5/7/2021 12:16 PM, Joe Obernberger wrote:


So I'm confused.
I get this on startup from the client:
2021-05-07 15:27:48,119 WARNÂ [com.dat.oss.dri.int.cor.poo.ChannelPool] (s1-admin-1) [s1|hercules/172.16.100.253:9042] Fatal error w hile initializing pool, forcing the node down: com.datastax.oss.driver.api.core.UnsupportedProtocolVersionException: [hercules/172.1 
6.100.253:9042] *Host does not support protocol version V5*
        at com.datastax.oss.driver.api.core.UnsupportedProtocolVersionException.forSingleAttempt(UnsupportedProtocolVersionException 
.java:46)
        at com.datastax.oss.driver.internal.core.channel.ProtocolInitHandler$InitRequest.onResponse(ProtocolInitHandler.java:335)         at com.datastax.oss.driver.internal.core.channel.ChannelHandlerRequest.onResponse(ChannelHandlerRequest.java:94)         at com.datastax.oss.driver.internal.core.channel.InFlightHandler.channelRead(InFlightHandler.java:257)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)         at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)         at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)         at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)         at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)         at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)         at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)         at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) 
        at java.base/java.lang.Thread.run(Thread.java:834)

If I specify in the application.conf to use v4, I get:

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.base/java.lang.reflect.Method.invoke(Method.java:566)         at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:48)         at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:25) Caused by: java.lang.IllegalArgumentException: Unknown protocol version name: v4 

Complete application.conf:

datastax-java-driver {
  basic.request.timeout = 60 seconds
  basic.request.consistency = LOCAL_QUORUM
  basic.contact-points = ["hercules:9042", "chaos:9042"]
  basic.load-balancing-policy {
        local-datacenter = datacenter1
  }
  advanced.protocol {
    version = v4
  }
}

-Joe


On 5/7/2021 12:05 PM, Sam Tunnicliffe wrote:
That's a driver error using protocol V5, which is the default from 4.0-rc1 but only recently added to the drivers. Can you try specifying protocol V4 with all the same parameters? Also, if it's at all possible (which it may not be, given the divergence between driver versions 3 & 4), could you try with protocol V5 and driver version 3.11.0? 

Thanks,
Sam
On 7 May 2021, at 16:12, Joe Obernberger <joseph.obernber...@gmail.com> wrote: 

I can retry Java 11.
I am seeing this error a lot - still debugging, but I'll throw it out there - using 4.11.1 driver and a 4 node RC1 cluster.  I'm seeing warning in the cassandra logs about slow queries, but no errors.  This error is client side.
Caused by: com.datastax.oss.driver.api.core.AllNodesFailedException: All 4 node(s) tried for the query failed (showing first 3 nodes, use getAllErrors() for more): Node(endPoint=/172.16.100.39:9042, hostId=93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47, hashCode=345a8431): [java.lang.NullPointerException], Node(endPoint=/172.16.100.36:9042, hostId=d9702f96-256e-45ae-8e12-69a42712be50, hashCode=4c7ac5bb): [java.lang.NullPointerException], Node(endPoint=chaos/172.16.100.37:9042, hostId=08a19658-40be-4e55-8709-812b3d4ac750, hashCode=7ba07f0e): [java.lang.NullPointerException]         at com.datastax.oss.driver.api.core.AllNodesFailedException.copy(AllNodesFailedException.java:141)         at com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)         at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:53)         at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:30)         at com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:230)         at com.datastax.oss.driver.api.core.cql.SyncCqlSession.execute(SyncCqlSession.java:54)         at com.ngc.helios.heliosingestservice.IngestService.splitOrigData(IngestService.java:55)         at com.ngc.helios.heliosingestservice.IngestService_ClientProxy.splitOrigData(IngestService_ClientProxy.zig:157)         at com.ngc.helios.heliosingestservice.IngestResource.splitOrigData(IngestResource.java:27)         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.base/java.lang.reflect.Method.invoke(Method.java:566)         at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170)         at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)         at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:643)         at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:507)         at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:457)         at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)         at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:459)         at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:419)         at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:393)         at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:68)         at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:492) 
        ... 15 more
        Suppressed: java.lang.NullPointerException
                at com.datastax.oss.protocol.internal.PrimitiveSizes.sizeOfShortBytes(PrimitiveSizes.java:59)                 at com.datastax.oss.protocol.internal.request.Execute$Codec.encodedSize(Execute.java:78)                 at com.datastax.oss.protocol.internal.FrameCodec.encodedBodySize(FrameCodec.java:272)                 at com.datastax.oss.protocol.internal.SegmentBuilder.addFrame(SegmentBuilder.java:75)                 at com.datastax.oss.driver.internal.core.protocol.FrameToSegmentEncoder.write(FrameToSegmentEncoder.java:56)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:304)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at com.datastax.oss.driver.internal.core.channel.InFlightHandler.write(InFlightHandler.java:151)                 at com.datastax.oss.driver.internal.core.channel.InFlightHandler.write(InFlightHandler.java:108)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1015)                 at io.netty.channel.AbstractChannel.write(AbstractChannel.java:289)                 at com.datastax.oss.driver.internal.core.channel.DefaultWriteCoalescer$Flusher.runOnEventLoop(DefaultWriteCoalescer.java:100)                 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)                 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)                 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)                 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)                 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)                 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)                 at java.base/java.lang.Thread.run(Thread.java:834) 
        Suppressed: java.lang.NullPointerException
                at com.datastax.oss.protocol.internal.PrimitiveSizes.sizeOfShortBytes(PrimitiveSizes.java:59)                 at com.datastax.oss.protocol.internal.request.Execute$Codec.encodedSize(Execute.java:78)                 at com.datastax.oss.protocol.internal.FrameCodec.encodedBodySize(FrameCodec.java:272)                 at com.datastax.oss.protocol.internal.SegmentBuilder.addFrame(SegmentBuilder.java:75)                 at com.datastax.oss.driver.internal.core.protocol.FrameToSegmentEncoder.write(FrameToSegmentEncoder.java:56)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:304)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at com.datastax.oss.driver.internal.core.channel.InFlightHandler.write(InFlightHandler.java:151)                 at com.datastax.oss.driver.internal.core.channel.InFlightHandler.write(InFlightHandler.java:108)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1015)                 at io.netty.channel.AbstractChannel.write(AbstractChannel.java:289)                 at com.datastax.oss.driver.internal.core.channel.DefaultWriteCoalescer$Flusher.runOnEventLoop(DefaultWriteCoalescer.java:100)                 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)                 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)                 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)                 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)                 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)                 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)                 at java.base/java.lang.Thread.run(Thread.java:834) 
        Suppressed: java.lang.NullPointerException
                at com.datastax.oss.protocol.internal.PrimitiveSizes.sizeOfShortBytes(PrimitiveSizes.java:59)                 at com.datastax.oss.protocol.internal.request.Execute$Codec.encodedSize(Execute.java:78)                 at com.datastax.oss.protocol.internal.FrameCodec.encodedBodySize(FrameCodec.java:272)                 at com.datastax.oss.protocol.internal.SegmentBuilder.addFrame(SegmentBuilder.java:75)                 at com.datastax.oss.driver.internal.core.protocol.FrameToSegmentEncoder.write(FrameToSegmentEncoder.java:56)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:304)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at com.datastax.oss.driver.internal.core.channel.InFlightHandler.write(InFlightHandler.java:151)                 at com.datastax.oss.driver.internal.core.channel.InFlightHandler.write(InFlightHandler.java:108)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1015)                 at io.netty.channel.AbstractChannel.write(AbstractChannel.java:289)                 at com.datastax.oss.driver.internal.core.channel.DefaultWriteCoalescer$Flusher.runOnEventLoop(DefaultWriteCoalescer.java:100)                 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)                 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)                 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)                 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)                 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)                 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)                 at java.base/java.lang.Thread.run(Thread.java:834) 
        Suppressed: java.lang.NullPointerException
                at com.datastax.oss.protocol.internal.PrimitiveSizes.sizeOfShortBytes(PrimitiveSizes.java:59)                 at com.datastax.oss.protocol.internal.request.Execute$Codec.encodedSize(Execute.java:78)                 at com.datastax.oss.protocol.internal.FrameCodec.encodedBodySize(FrameCodec.java:272)                 at com.datastax.oss.protocol.internal.SegmentBuilder.addFrame(SegmentBuilder.java:75)                 at com.datastax.oss.driver.internal.core.protocol.FrameToSegmentEncoder.write(FrameToSegmentEncoder.java:56)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:304)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at com.datastax.oss.driver.internal.core.channel.InFlightHandler.write(InFlightHandler.java:151)                 at com.datastax.oss.driver.internal.core.channel.InFlightHandler.write(InFlightHandler.java:108)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)                 at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)                 at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)                 at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1015)                 at io.netty.channel.AbstractChannel.write(AbstractChannel.java:289)                 at com.datastax.oss.driver.internal.core.channel.DefaultWriteCoalescer$Flusher.runOnEventLoop(DefaultWriteCoalescer.java:100)                 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)                 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)                 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)                 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)                 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)                 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)                 at java.base/java.lang.Thread.run(Thread.java:834) 


On 5/7/2021 10:00 AM, Jeff Jirsa wrote:
Cassandra 4.0 should work fine with java 11, including zgc (though zgc in jdk11 isn't meant to be production ready). 

The things I care most about:
- Much faster streaming, which you care about if you're not using EBS/Disaggregated storage - Virtual tables that make observability much more consistent (less JMX, more CQL) 
- Incremental repair finally actually works (correctly)
- There's a bunch of new defensive rate limiters and hot-tunable properties in the database that people will enjoy once they need to use them 
- JDK11�
On Fri, May 7, 2021 at 6:05 AM Joe Obernberger <joseph.obernber...@gmail.com> wrote: 

    Hi Sean - I'm using RC1 now in a research environment on bare
    metal.  The biggest drawback of Cassandra for me is that
    Cassandra has issues working with modern large servers - a
    server with >32TBytes of SSD seems to be a non-starter.

    I tried running Cassandra with java 11, and that doesn't appear
    to work.

    -Joe

    On 5/7/2021 8:47 AM, Durity, Sean R wrote:


    There is not enough 4.0 chatter here. What feature or fix of
    the 4.0 release is most important for your use
    case(s)/environment? What is working well so far? What needs
    more work? Is there anything that needs more explanation?

    �

    <image001.png>

        

    Sean Durity

    Staff Systems Engineer – Cassandra

    #cassandra - for the latest news and updates

    �
    �

    ------------------------------------------------------------------------

    The information in this Internet Email is confidential and may
    be legally privileged. It is intended solely for the
    addressee. Access to this Email by anyone else is
    unauthorized. If you are not the intended recipient, any
    disclosure, copying, distribution or any action taken or
    omitted to be taken in reliance on it, is prohibited and may
    be unlawful. When addressed to our clients any opinions or
    advice contained in this Email are subject to the terms and
    conditions expressed in any applicable governing The Home
    Depot terms of business or client engagement letter. The Home
    Depot disclaims all responsibility and liability for the
    accuracy and content of this attachment and for any damages or
    losses arising from any inaccuracies, errors, viruses, e.g.,
    worms, trojan horses, etc., or other items of a destructive
    nature, which may be contained in this attachment and shall
    not be liable for direct, indirect, consequential or special
    damages in connection with this e-mail message or its attachment.

    
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
        Virus-free. www.avg.com
    
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>


    <x-msg://7/#m_654382365824200109_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Reply via email to