Why don't you create a JIRA issue? Assertion errors, leaks and NPEs usually indicate a bug. This really should not happen.
2017-01-11 4:35 GMT+01:00 Gopal, Dhruva <dhruva.go...@aspect.com>: > Any suggestions/recommendations? Anything will help at this point. Thanks! > > Regards, > > *DHRUVA GOPAL* > > *sr. MANAGER, ENGINEERING* > > *REPORTING, ANALYTICS AND BIG DATA* > > *+1 408.325.2011 <+1%20408-325-2011>* *WORK* > > *+1 408.219.1094 <+1%20408-219-1094>* *MOBILE* > > *UNITED STATES* > > *dhruva.go...@aspect.com <dhruva.go...@aspect.com> * > > *aspect.com <http://www.aspect.com/>* > > [image: Description: http://webapp2.aspect.com/EmailSigLogo-rev.jpg] > > > > From: Dhruva Gopal > Date: Monday, January 9, 2017 at 1:42 PM > To: "user@cassandra.apache.org" > Cc: Richard Ney > Subject: Trying to identify the cause of these errors. > > My colleague (Richard Ney) has already been in touch with you on a couple > of other issues we’ve seen in the past. Our development been trying to > track down some new issues we’ve been seeing on one of our pre-prod > environments where we’ve been having consistent failures very often (every > day or every 2-3 days), even if load/number of transactions are very light. > We’re running a 2 data center deployment with 3 nodes in each data center. > Our tables are setup with replication factor = 2 and we have 16G dedicated > to the heap with the G1GC for garbage collection. Our systems are AWS > M4.2xlarge with 8 CPUs and 32GB of RAM and we have 2 general purpose EBS > volumes on each node of 500GB each. Once we hit this it seems like the only > way to recover is to shutdown the cluster and restart. Running repairs > after the restart often results in failures and we pretty much end up > having to truncate the tables before starting up clean again. We are not > sure if the two are inter-related. We pretty much see the same issue on all > the nodes. If anyone has any tips or any suggestions on how to diagnose > this further, it will help a great deal! The issues are: > > > > *Issue 1:* Once the errors occur they just repeat for a bit followed by > the errors in issue 2. > > INFO [CompactionExecutor:165] 2017-01-08 08:32:39,915 > AutoSavingCache.java:386 - Saved KeyCache (63 items) in 5 ms > INFO [IndexSummaryManager:1] 2017-01-08 08:32:41,438 > IndexSummaryRedistribution.java:74 - Redistributing index summaries > INFO [HANDSHAKE-ahldataslave4.bos.manhattan.aspect-cloud.net/10.184.8.224] > 2017-01-08 09:30:03,988 OutboundTcpConnection.java:505 - Handshaking > version with ahldataslave4.bos.manhattan.aspect-cloud.net/10.184.8.224 > INFO [IndexSummaryManager:1] 2017-01-08 09:32:41,440 > IndexSummaryRedistribution.java:74 - Redistributing index summaries > WARN [SharedPool-Worker-9] 2017-01-08 10:30:00,116 > BatchStatement.java:289 - Batch of prepared statements for > [manhattan.rcmessages] is of size 9264, exceeding specified threshold of > 5120 by 4144. > INFO [IndexSummaryManager:1] 2017-01-08 10:32:41,442 > IndexSummaryRedistribution.java:74 - Redistributing index summaries > INFO [IndexSummaryManager:1] 2017-01-08 11:32:41,443 > IndexSummaryRedistribution.java:74 - Redistributing index summaries > INFO [CompactionExecutor:162] 2017-01-08 12:32:39,914 > AutoSavingCache.java:386 - Saved KeyCache (108 items) in 4 ms > INFO [IndexSummaryManager:1] 2017-01-08 12:32:41,446 > IndexSummaryRedistribution.java:74 - Redistributing index summaries > INFO [IndexSummaryManager:1] 2017-01-08 13:32:41,448 > IndexSummaryRedistribution.java:74 - Redistributing index summaries > INFO [IndexSummaryManager:1] 2017-01-08 14:32:41,450 > IndexSummaryRedistribution.java:74 - Redistributing index summaries > INFO [IndexSummaryManager:1] 2017-01-08 15:32:41,451 > IndexSummaryRedistribution.java:74 - Redistributing index summaries > INFO [CompactionExecutor:170] 2017-01-08 16:32:39,915 > AutoSavingCache.java:386 - Saved KeyCache (109 items) in 4 ms > INFO [IndexSummaryManager:1] 2017-01-08 16:32:41,453 > IndexSummaryRedistribution.java:74 - Redistributing index summaries > WARN [SharedPool-Worker-4] 2017-01-08 17:30:45,048 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-4,5,main]: {} > java.lang.AssertionError: null > at org.apache.cassandra.db.rows.BufferCell.<init>(BufferCell.java:49) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.db.rows.BufferCell.tombstone(BufferCell.java:88) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.db.rows.BufferCell.tombstone(BufferCell.java:83) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.db.rows.BufferCell.purge(BufferCell.java:175) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.db.rows.ComplexColumnData.lambda$ > purge$107(ComplexColumnData.java:165) ~[apache-cassandra-3.3.0.jar:3.3.0] > at > org.apache.cassandra.utils.btree.BTree$FiltrationTracker.apply(BTree.java:650) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:693) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:668) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at > org.apache.cassandra.db.rows.ComplexColumnData.transformAndFilter(ComplexColumnData.java:170) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at > org.apache.cassandra.db.rows.ComplexColumnData.purge(ComplexColumnData.java:165) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at > org.apache.cassandra.db.rows.ComplexColumnData.purge(ComplexColumnData.java:43) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.db.rows.BTreeRow.lambda$purge$102(BTreeRow.java:333) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at > org.apache.cassandra.utils.btree.BTree$FiltrationTracker.apply(BTree.java:650) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:693) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:668) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.db.rows.BTreeRow.transformAndFilter(BTreeRow.java:338) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.db.rows.BTreeRow.purge(BTreeRow.java:333) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.db.partitions.PurgeFunction. > applyToRow(PurgeFunction.java:88) ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:116) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize( > UnfilteredRowIteratorSerializer.java:133) ~[apache-cassandra-3.3.0.jar: > 3.3.0] > at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize( > UnfilteredRowIteratorSerializer.java:89) ~[apache-cassandra-3.3.0.jar: > 3.3.0] > at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize( > UnfilteredRowIteratorSerializer.java:79) ~[apache-cassandra-3.3.0.jar: > 3.3.0] > at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$ > Serializer.serialize(UnfilteredPartitionIterators.java:294) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:127) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:123) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:292) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb( > ReadCommandVerbHandler.java:50) ~[apache-cassandra-3.3.0.jar:3.3.0] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_111] > at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ > ice$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ > ice$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.3.0.jar:3.3.0] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] > > > > *Issue 2:* > > WARN [SharedPool-Worker-2] 2017-01-09 05:18:58,880 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2461) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_111] > at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ > ice$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ > ice$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.3.0.jar:3.3.0] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] > Caused by: java.lang.NullPointerException: null > WARN [SharedPool-Worker-1] 2017-01-09 05:19:14,220 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]: {} > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2461) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_111] > at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ > ice$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ > ice$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.3.0.jar:3.3.0] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] > Caused by: java.lang.NullPointerException: null > WARN [SharedPool-Worker-1] 2017-01-09 05:19:29,876 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]: {} > java.lang.NullPointerException: null > WARN [SharedPool-Worker-1] 2017-01-09 05:19:45,217 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-1,5,main]: {} > java.lang.NullPointerException: null > WARN [SharedPool-Worker-2] 2017-01-09 05:20:00,241 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > > > > *Issue 3:* Periodic warnings about leaks (this was during a repair) - we > had an OOM occurrence about a week ago: > > INFO [Thread-9] 2017-01-09 15:01:11,828 RepairSession.java:237 - [repair > #7593ad41-d67c-11e6-a76f-f1d6fd321f0d] new session: will sync > ahldataslave1.bos.manhattan.aspect-cloud.net/10.184.8.151, /10.184.8.137, > /10.184.8.219, /10.184.8.224 on range > [(4815877001221178276,4825317823493984218], > (3375739423664266689,3378070249246945856], > (-7393599445401300009,-7393031105803218575], > (-8130357422229977058,-8127934975152946543], > (-4611683212167667055,-4597307445147354114], > (4829191785668845783,4847928037092763493], > (5522544388079333104,5528633388915935404], > (1021118078158195547,1032730403399741255], > (-20739833830226207,-14719054523073734], > (-6205449048248449637,-6203730539323274527], > (45435932670861622,69554714149031706], > (148647262653106287,181904829526383862], > (-854190757691833472,-854087983203765538], > (-7714687510919982336,-7700463340346726648], > (-360739232930366185,-339249720009847821], > (524595904743716879,537371267474489752], > (8185882494697143291,8187345181917158585], > (181904829526383862,190096166859570216], > (-1152328644109499972,-1134711681717758961], > (-586016578220869954,-566687033455187734], > (-4597307445147354114,-4541034774723552824], > (-4488655272471923458,-4481233629800439836], > (3111053603069111732,3121096557789915147], > (-1167831376731724655,-1152328644109499972], > (-6302582962631272172,-6290099341099600242], > (8082429902489189721,8084233962135014968], > (-2781993705200227829,-2777492754784519178], > (5996816756578981045,6022011570976925279], > (4404919629779433734,4422232637329318322], > (-643317786131863042,-611582394998065454], > (4514099138058135901,4514906323961126580], > (7273471227296986256,7316794306200362675], > (4462984376163219141,4464984714163480487], > (-2390509810947844616,-2382282465250439153], > (-7773729052257068390,-7772926086336061342], > (-161569209320145017,-149914302996366392], > (1392011488470148469,1403436343776848669], > (546940383302067460,574197777751440495], > (6624568645457217871,6629411218351717560], > (-1134711681717758961,-1133755953039077472], > (-7248281701751445296,-7243217407431962906], > (2374653705052569734,2377396978942737768], > (5619550734131192401,5628464353424177974], > (-192925818286153751,-181902834109073167], > (5896301416900157412,5911734690958039138], > (4422232637329318322,4462984376163219141], > (-4524564105581793254,-4504900851715014673], > (-2115528876758613050,-2112322288682500569], > (5504098260114685179,5515264579319185925], > (35614519826275404,45435932670861622], > (-855913784497695841,-854190757691833472], > (5736245318761313405,5738445441111493493], > (4807463771329802801,4815877001221178276], > (-2884278504117308670,-2881329894351532736], > (-207545241327995479,-207389332352346254], > (-181902834109073167,-172819893538312319], > (-7636862515759014032,-7626686102011136779], > (-8140870857704067579,-8130357422229977058], > (-2342122616740401439,-2335433185444036860], > (-2881329894351532736,-2880349360092682175], > (-611582394998065454,-587918354120388686], > (-7320730625612121443,-7317644215584737598], > (-13045270760172222,8260384715982984], > (-8668935926714005039,-8657580917928283146], > (8260384715982984,35614519826275404], > (2861067483327829799,2864748323206840859], > (3127114336475141934,3139163273099443478], > (-7657420721193956821,-7636862515759014032], > (1032730403399741255,1032951438204062821], > (-1133755953039077472,-1130095656542041882], > (5515264579319185925,5522544388079333104], > (6826479291574328528,6828593995008227175], > (658600138550738740,680809612853177633], > (2364271158745468459,2374653705052569734], > (2864748323206840859,2868871349367827459], > (-7741910906325473118,-7738712583959975589], > (2749562470526527976,2765502980561577063], > (4569656624441412194,4581277061087905130], > (-1130095656542041882,-1102822102812415509], > (2357686856375407577,2364271158745468459], > (2875146491882456097,2913694653712675567], > (-2365721528619247416,-2362417204623405805], > (-7700463340346726648,-7657420721193956821], > (-1545400958003324469,-1527145992709891047], > (6595468060656389384,6611122695943865425], > (-2362417204623405805,-2360989147149959339], > (680809612853177633,691564499046700936], > (1048818468740031469,1051848236340749987], > (-6290099341099600242,-6284139189426371408], > (-430511209136180745,-417631748808875679], > (-2127191227331961211,-2115528876758613050], > (-587918354120388686,-586016578220869954], > (5950659959556897324,5972536818860340742], > (7228681112293176899,7273471227296986256], > (-92115275177849392,-73078256610822906], > (-2150182737057335813,-2131084968707606517], > (-5931891237188071234,-5925084712702354100], > (1854063092500381970,1894128417137542926], > (3139163273099443478,3144353879892041737], > (2795130016444690338,2849989872521310320], > (69554714149031706,73259277905023853], > (-198884270246085916,-192925818286153751], > (2301867852127537258,2344138511230415627], > (-2360989147149959339,-2342122616740401439], > (2868871349367827459,2875146491882456097], > (-8520120040026037727,-8484758828657908805], > (1044175704807438060,1048818468740031469], > (3658700192210846721,3709106004630729567], > (-8715352510373593584,-8695484052773103051], > (4825317823493984218,4829191785668845783], > (5881177751245354064,5896301416900157412], > (-1557733846751863547,-1545400958003324469], > (3764340229927726801,3772352737530248438], > (5723087139361841478,5736245318761313405], > (-339249720009847821,-325063001525001753], > (4581277061087905130,4585591969590606261], > (2849989872521310320,2861067483327829799], > (-201641945389990466,-198884270246085916], > (3062119490734517386,3070339219117072282], > (6611989977644669113,6624568645457217871], > (-6303561532875621980,-6302582962631272172], > (-2880349360092682175,-2815907193866317595], > (-2210673969380244356,-2200655423201071981], > (6920406973111913689,6944396390330370813], > (3070339219117072282,3089918739923672617], > (1018479337556218583,1021118078158195547], > (-22373917471115570,-20802161777184740], > (2349224556257120691,2357686856375407577], > (1610291395978708348,1610350862682840167], > (-7774308296105089774,-7773729052257068390], > (-207389332352346254,-201641945389990466], > (-4504900851715014673,-4488655272471923458], > (-913362829471550684,-900550299179500948], > (574197777751440495,575687259162456966], > (-209852186312631406,-207545241327995479], > (3961560378473489131,4029122901768525705], > (1059898575782522398,1065913666232086022], > (6611122695943865425,6611989977644669113], > (-2815907193866317595,-2781993705200227829], > (-8149621014270042242,-8140870857704067579], > (-999997390934570392,-998678768267637866], > (2377396978942737768,2384267095612056863], > (-2267001694418802171,-2210871230482048364], > (-2131084968707606517,-2127191227331961211], > (3144353879892041737,3183343610940682057], > (-2210871230482048364,-2210673969380244356], > (4399306102077085676,4404919629779433734], > (537371267474489752,546940383302067460], > (-7722143244670988745,-7714687510919982336], > (3754196018068729018,3763367677110000435], > (-5925084712702354100,-5919412889821651249], > (4585591969590606261,4590124192625756967], > (-1189315862160445133,-1167831376731724655], > (-14719054523073734,-13045270760172222], > (3772352737530248438,3791853264732937441], > (-149914302996366392,-138596249554837825], > (-1459586834232833970,-1455782887662096513], > (-7626686102011136779,-7613607518487725198], > (-2370342707880863831,-2365721528619247416], > (1051848236340749987,1059898575782522398], > (575687259162456966,587747968172054843], > (1846675642540253784,1854063092500381970], > (5911734690958039138,5950659959556897324], > (3709106004630729567,3754196018068729018], > (2274687735800823626,2301867852127537258], > (3763367677110000435,3764340229927726801], > (2765502980561577063,2768302382460911776], > (-4835698441475193016,-4821740368643417588], > (-4840592682190099042,-4835698441475193016], > (-900550299179500948,-898509802837897450], > (5972536818860340742,5996816756578981045], > (7316794306200362675,7316948326538779302], > (-20802161777184740,-20739833830226207], > (-8695484052773103051,-8693825425156218047], > (-6228811049824135380,-6205449048248449637], > (-8522744988679262888,-8520120040026037727], > (-1527145992709891047,-1522580222898594256], > (1065913666232086022,1070185844169898505], > (3121096557789915147,3127114336475141934]] > for manhattan.[config, messages, rcmessages, csseedmessages, metadata, > rqmessages] > ERROR [Reference-Reaper:1] 2017-01-09 15:01:12,481 Ref.java:202 - LEAK > DETECTED: a reference (org.apache.cassandra.utils. > concurrent.Ref$State@8520181) to class org.apache.cassandra.io. > sstable.format.SSTableReader$InstanceTidier@110489339:/ > data/manhattan/messages-5e11ec20d49411e6ac36cb64da3c5bcf/ma-1-big was not > released before the reference was garbage collected > ERROR [Reference-Reaper:1] 2017-01-09 15:01:12,481 Ref.java:202 - LEAK > DETECTED: a reference (org.apache.cassandra.utils. > concurrent.Ref$State@2aa6af1) to class org.apache.cassandra.io. > sstable.format.SSTableReader$InstanceTidier@1202250944:/ > data/manhattan/rcmessages-5f65e360d49411e6ac36cb64da3c5bcf/ma-1-big was > not released before the reference was garbage collected > ERROR [Reference-Reaper:1] 2017-01-09 15:01:12,481 Ref.java:202 - LEAK > DETECTED: a reference (org.apache.cassandra.utils. > concurrent.Ref$State@5feebbbd) to class org.apache.cassandra.io. > sstable.format.SSTableReader$InstanceTidier@1990550669:/ > data/manhattan/rcmessages-5f65e360d49411e6ac36cb64da3c5bcf/ma-2-big was > not released before the reference was garbage collected > > > > *Issue 4:* Failure on repair (cannot complete repair) - validation > failures. > INFO [Thread-9] 2017-01-09 15:01:12,496 RepairSession.java:237 - [repair > #75f99b00-d67c-11e6-a76f-f1d6fd321f0d] new session: will sync > ahldataslave1.bos.manhattan.aspect-cloud.net/10.184.8.151, /10.184.8.136, > /10.184.8.219, /10.184.8.206 on range > [(7070473513877450523,7071785512108503562], > (7540323476844137856,7551069853951512536], > (6496851087781126402,6506947825241172690], > (4956723654110177663,4997988272614120716], > (6159728418399742238,6161785046042182181], > (-6111201816165886930,-6109358652076682027], > (6765434292326999735,6786216263996459971], > (5670991529662923445,5678922736379606122], > (-8368452819419163190,-8352559832808839636], > (-3579114434972148656,-3552744642655983590], > (7574005965138357735,7575976736757457093], > (3443342880213529041,3448513378658973391], > (-6374441519731158925,-6368083975098827965], > (1506273267531550913,1508155162580364084], > (-3841814855559153943,-3835938678769454001], > (9170965416227969215,9181174655986386320], > (-6589650497263617394,-6578994961053045565], > (-7526495575912144496,-7518448902193774650], > (3448513378658973391,3466857105667177973], > (7553098986919382391,7574005965138357735], > (-7927877550785715833,-7927648953253692861], > (8525761079018902513,8533541215797136240], > (-9020129600263662082,-9010501329825003246], > (1090724998362773160,1097939939665235297], > (6495099300248039972,6496851087781126402], > (6161785046042182181,6230901034893815059], > (-7463338857564269606,-7459175745770455573], > (-5365113529531278290,-5358403398116793300], > (2249898065901245108,2256752991177466019], > (427458615961451939,456792456721828910], > (-7518448902193774650,-7476577165771266744], > (8496964666711176665,8503454084084785332], > (1508155162580364084,1531943799366406071], > (1623939448991457330,1628591069075100791], > (2500969373198505595,2510256322219581447], > (1487072216030226303,1506273267531550913], > (-8842784826526224555,-8840216173643212475], > (2233348668129647030,2247658708769712814], > (-8289206614243730701,-8287074180401539509], > (-7476577165771266744,-7470389682785988771], > (3509709684061378867,3512938839959595162], > (1296769759309759534,1302830049359124705], > (7575976736757457093,7612135532866611803], > (7539950122318008545,7540323476844137856], > (-1860975172751451123,-1853676643512030698], > (-8116786947316919605,-8107369675963369796], > (-5576866678991811211,-5548824138592035245], > (-9211881728677655622,-9209566963086044390], > (7489367377277174970,7511259639253940249], > (1302830049359124705,1310010318365119318], > (-8750733220711433842,-8741408722769214425], > (-6060376367842620341,-6049131555099770258], > (2476529278549364684,2476659930520260878], > (-4418742343319617797,-4384333951548488071], > (1930660747072472501,1946687557062422024], > (-3702380150896445622,-3691910222307497728], > (-4419703203609450987,-4418742343319617797], > (7551697213870628887,7553098986919382391], > (-7909869864636951454,-7904683595914140744], > (2492882420790179151,2500969373198505595], > (759685512451005610,773693246847159219], > (722689136592207691,730821328907957274], > (5219025897237634110,5229566135848567231], > (4149517035748854868,4162833917677490963], > (1326132699179277905,1349642653396670633], > (-3552744642655983590,-3549073980410969181], > (-4456698363145934960,-4419703203609450987], > (-5482989026069906323,-5481213659503324794], > (279535406932511121,297415597645494493], > (-9204266745720873498,-9161795161892156657], > (6786216263996459971,6786883290959946615], > (1635462812102810232,1638975594001298412], > (1531943799366406071,1546279755200334429], > (-3851094911406737954,-3841814855559153943], > (8488130353439482734,8496964666711176665], > (9146936609495586767,9148689706364420814], > (-3674452659583777301,-3669778494255064100], > (1546279755200334429,1554992050513055527], > (2969501640373218313,2975603144417796179], > (-3865994019611705472,-3851094911406737954], > (1119818899797211952,1121114266048498658], > (5555981883036126656,5559877922798642583], > (-9161795161892156657,-9148239227447645822], > (-1988480938545038453,-1972594091279542463], > (-8457443005098123276,-8426466451426413266], > (7527886006442742638,7539950122318008545], > (1482724986864293552,1487072216030226303], > (218814708185728764,222373457441562248], > (-1844742172330357357,-1834186452017037646], > (-7914786200548736052,-7909869864636951454], > (1554992050513055527,1556471756149638584], > (2247658708769712814,2249241162600622119], > (2249241162600622119,2249898065901245108], > (-6368083975098827965,-6357129727004451888], > (1349642653396670633,1372390693080139703], > (-5628093303198956105,-5602276528881256607], > (-5602276528881256607,-5576866678991811211], > (-1972594091279542463,-1932380490869749103], > (6746187069375198686,6759453964579921534], > (-9209566963086044390,-9204266745720873498], > (2969169700471726709,2969501640373218313], > (-1846725725884736711,-1844742172330357357], > (8466845312658849111,8478632986527402947], > (6786883290959946615,6786909234664957611], > (-8105207391882034800,-8058995460560987960], > (-5481213659503324794,-5480151738028046165], > (6152943806705127166,6159728418399742238], > (-7931416471770778308,-7927877550785715833], > (-7444305315017293180,-7438698090368547807], > (-5405765462772655927,-5365113529531278290], > (7468218535666030613,7489367377277174970], > (7370990959551251700,7384554095789173771], > (9181174655986386320,9197356813903415489], > (-7891619558235712999,-7886926665775314135], > (4863660993721847898,4876110806493712229], > (5546813895398001993,5555981883036126656], > (6793672076817085638,6817463007248949484], > (730821328907957274,741073317079317196], > (1285297621466628116,1296769759309759534], > (-6109358652076682027,-6097749005908966679], > (-3160359268440132710,-3150255443388600113], > (-6067028212437311851,-6060376367842620341], > (3512938839959595162,3531669213417260709], > (-1641724835017736571,-1630212592242213627], > (-5631109618795517413,-5628093303198956105], > (6759453964579921534,6765434292326999735], > (8027648385044762960,8030142266857918084], > (1264145341476479882,1285297621466628116], > (1116504555019555851,1119818899797211952], > (-3139664283152392978,-3138393885745606369], > (-1652241560457928053,-1641724835017736571], > (8503454084084785332,8525761079018902513], > (-1853676643512030698,-1846725725884736711], > (1628591069075100791,1635462812102810232], > (-8840216173643212475,-8817712049837887184], > (-6357129727004451888,-6355060010614218705], > (-6097749005908966679,-6088616433680186631], > (-7904683595914140744,-7891619558235712999], > (-4952793941753266051,-4937049631932742604], > (-3150255443388600113,-3139664283152392978], > (2512801900471921714,2516325796899709313], > (2510256322219581447,2512801900471921714], > (-5638226176228799616,-5631109618795517413], > (-7886926665775314135,-7882937045572193145], > (6786909234664957611,6793672076817085638], > (-3691910222307497728,-3674452659583777301], > (-4867940821711541762,-4866946150328928181], > (7071785512108503562,7081793743588959944], > (4876110806493712229,4879643371134562502], > (6138845154266790671,6152943806705127166], > (-6355060010614218705,-6332373169802347251], > (-6125214696594291714,-6111201816165886930], > (-7470389682785988771,-7463338857564269606], > (-3598204681868075295,-3579114434972148656], > (-8107369675963369796,-8105207391882034800], > (7612135532866611803,7641727005250566009], > (9148689706364420814,9170965416227969215], > (6506947825241172690,6515762919020447377], > (-6088616433680186631,-6067028212437311851], > (741073317079317196,759685512451005610], > (-8352559832808839636,-8344010281020219217], > (-6049131555099770258,-6043632332003790850], > (3494099643169044463,3509709684061378867], > (3466857105667177973,3494099643169044463], > (7551069853951512536,7551697213870628887], > (2476659930520260878,2492882420790179151], > (-8842911646480810622,-8842784826526224555], > (193808271419729511,198960416318160202], > (-4873803694032119901,-4867940821711541762], > (-7569775426219162591,-7569610743922997927], > (4730391117872973381,4732570096640666262], > (9137762516596245936,9146936609495586767], > (-9148239227447645822,-9137147760354721369], > (-8296203956052422690,-8289206614243730701], > (198960416318160202,218814708185728764]] > for manhattan.[config, messages, rcmessages, csseedmessages, metadata, > rqmessages] > WARN [SharedPool-Worker-2] 2017-01-09 15:01:38,219 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2461) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_111] > at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ > ice$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ > ice$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.3.0.jar:3.3.0] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] > Caused by: java.lang.NullPointerException: null > WARN [SharedPool-Worker-2] 2017-01-09 15:04:41,678 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2461) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_111] > at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ > ice$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ > ice$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.3.0.jar:3.3.0] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.3.0.jar:3.3.0] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] > > > Regards, > > *DHRUVA GOPAL* > > *sr. MANAGER, ENGINEERING* > > *REPORTING, ANALYTICS AND BIG DATA* > > *+1 408.325.2011 <+1%20408-325-2011>* *WORK* > > *+1 408.219.1094 <+1%20408-219-1094>* *MOBILE* > > *UNITED STATES* > > *dhruva.go...@aspect.com <dhruva.go...@aspect.com> * > > *aspect.com <http://www.aspect.com/>* > > [image: Description: http://webapp2.aspect.com/EmailSigLogo-rev.jpg] > > > This email (including any attachments) is proprietary to Aspect Software, > Inc. and may contain information that is confidential. If you have received > this message in error, please do not read, copy or forward this message. > Please notify the sender immediately, delete it from your system and > destroy any copies. You may not further disclose or distribute this email > or its attachments. > -- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.com Wehrstraße 46 · 73035 Göppingen · Germany Phone +49 7161 304880-6 · Fax +49 7161 304880-1 AG Ulm · HRB 731058 · Managing Director: Jens Kammerer
