date:20170515

Re: [akka-user] Akka Cluster Failing for Huge Data File

2017-05-15 Thread Kunal Ghosh

Hey Patrik, Thanks for the help !!
your solution worked ! 
Now when I run my application in non clustered environment with round robin 
pool (no of instances = 8) it take 23 seconds to process 2 million rows of 
data. 
But when I run same application in clustered environment it took 23 minutes 
!!! The master node reads data from input file and sends it over to both 
*seed-nodes **(each node have no of instances = 4)* for evaluation 
(processing). I can not figure out the reason behind this. 
*Machine Information :*
RAM : 16 GB
Logical Process : 8
Cores : 4
CPU : Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz

Can you Help ???
*I have also attached the configuration file and code examples are attached 
with this mail please do check them out.*

*Application configuration file (without cluster)*

iCEDQDispatcher{

 worker-dispatcher {
  type = Dispatcher
 executor = "fork-join-executor"
 fork-join-executor {
 parallelism-min = 2
 parallelism-factor = 2.0
  parallelism-max = 64
  }
  throughput = 5
   }
 
   CallingThreadDispatcher {
 type = akka.testkit.CallingThreadDispatcherConfigurator
}
}


*Application configuration file (with cluster)*












*akka {  loggers = ["akka.event.slf4j.Slf4jLogger"] loglevel = "INFO"   
stdout-loglevel = "INFO"   logging-filter = 
"akka.event.slf4j.Slf4jLoggingFilter"actor.provider = 
"akka.cluster.ClusterActorRefProvider" remote {
log-remote-lifecycle-events = onlog-sent-messages = on
log-received-messages = onartery {  enabled = on  
canonical.hostname = "192.168.100.199"  canonical.port = 25520}   
}actor {kryo {  type = "graph"  idstrategy = "incremental"  
#"explicit"  buffer-size = 4096  max-buffer-size = -1  
use-manifest = false  implicit-registration-logging = true  
kryo-trace = false   mappings {  "[Ljava.lang.String;" = 20  
"[Lorg.iceengine.compare.engine.ICEColSource;" = 21  
"org.iceengine.compare.engine.ICEColSource$Type" = 22  
"java.text.DecimalFormat" = 23  "java.math.RoundingMode" = 24  
"java.text.DecimalFormatSymbols" = 25  "java.util.Locale" = 26  
"[Ljava.lang.Integer;" = 27  "[Ljava.lang.Object;" = 28  
"[Lorg.iceengine.compare.engine.ICECompareColumnData;" = 29  
"[Lorg.iceengine.common.CompareChain;" = 30  
"[Lorg.apache.commons.collections.comparators.ComparatorChain;" = 31
  "org.iceengine.compare.engine.ICEEngineContext" = 32  
"org.iceengine.compare.akka.RowData" = 33  
"org.iceengine.compare.akka.DataConsumerInspector" = 34  
"org.iceengine.compare.akka.Result" = 35  
"org.iceengine.common.pairs.IceEnginePairs" = 36 
"org.iceengine.common.pairs.IcePairKey" = 37 
"org.iceengine.common.pairs.IceEnginePairsImpl" = 38 
"org.iceengine.common.DataComparator" = 39 
"org.iceengine.common.pairs.IcePairValue" = 40 
"org.iceengine.common.MapCont" = 41 "org.iceengine.common.ObjectComparator" 
= 42 "org.iceengine.compare.akka.AtomicObject" = 43 
"org.iceengine.compare.akka.SimpleBindings" = 44 
"org.iceengine.compare.akka.Work" = 45 
"org.iceengine.compare.comparator.DKChainDiffor" = 46 
"org.iceengine.compare.comparator.DKConvertingDiffor" = 47 
"org.iceengine.compare.comparator.DKDateDiffor" = 48 
"org.iceengine.compare.comparator.DKEqualsDiffor" = 49 
"org.iceengine.compare.comparator.DKIdentityDiffor" = 50 
"org.iceengine.compare.comparator.DKNumberDiffor" = 51 
"org.iceengine.compare.comparator.DKTextDiffor" = 52 
"org.iceengine.compare.conf.DefaultDataComparison" = 53 
"org.iceengine.compare.conf.ICEBuildRule" = 54 
"org.iceengine.compare.conf.ICEDepRule" = 55 
"org.iceengine.compare.conf.ICEEngine" = 56 
"org.iceengine.compare.conf.ICEExpression" = 57 
"org.iceengine.compare.conf.ICERule" = 58 
"org.iceengine.compare.conf.ICERulePT" = 59 
"org.iceengine.compare.conf.ICERuleRuleContainer" = 60 
"org.iceengine.compare.engine.CommonRowObj" = 61 
"org.iceengine.compare.engine.DataComparison" = 62 
"org.iceengine.compare.engine.ICEColSource" = 63 
"org.iceengine.compare.engine.ICECompareColInRow" = 64 
"org.iceengine.compare.engine.ICECompareColumn" = 65 
"org.iceengine.compare.engine.ICECompareColumnData" = 66 
"org.iceengine.compare.engine.ICECompareEngine" = 67 
"org.iceengine.compare.engine.ICECompareRow" = 68 
"org.iceengine.compare.engine.ICEExprCompare" = 69 
"org.iceengine.compare.engine.ICEFinalExpressionDiff" = 70 
"org.iceengine.compare.engine.ICESourceUtil" = 71 
"org.iceengine.compare.engine.ICEUniqueSource" = 72 
"org.iceengine.compare.sources.EngineWritterFormat" = 73 
"org.iceengine.compare.sources.ExternalSortingForUserDefiendColumn" = 74 
"org.iceengine.compare.sources.FileSort" = 75 
"org.iceengine.compare.sources.FinalWritter" = 76 
"org.iceengine.compare.sources.ICEDBWritter" = 77 
"org.iceengine.compare.sources.IceFileWritter" = 78 
"org.iceengine.compare.sources.ICESourceDataBase" = 79

[akka-user] [akka-streams] StreamConverters.fromOutputStream(..) stops processing messages

2017-05-15 Thread Alexey Shuksto

Hello hAkkers,

Our app spawns multiple external processes (via java.util.Runtime.exec(..)) 
and then writes to STDIN of such processes via:
```
StreamConverters
  .fromOutputStream(() => process.getOutputStream, autoFlush = true)
  .runWith(MergeHub.source[])
```

MergeHub is attached to output stream to prevent closing of such stream 
after one of client done writing to it.

While there are up to, say, four external processes everything is working 
fine, but as soon as they number increases, OutputStreamSubscriber just 
stops to request next elements for all of them.
I've checked via adding `.confate` for incoming ByteStrings -- buffer just 
keeps growing.

It is possible to replace StreamConverters with simple
```
Sink
  .foreach[ByteString] { 
process.getOutputStream.write(bs.toArray)
process.getOutputStream.flush()
  }
```
and everything would work, but it's blocking and stuff.

What may be the cause of this? I've tried to change settings of 
"default-blocking-io-dispatcher", but to no effect.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Re: Akka Http how to add charset utf-8 to the content-type header

2017-05-15 Thread johannes . rudolph

Hi Thibault,

you are right, there's currently no built-in way to do this. To achieve it, 
you could e.g. copy the Jackson marshaller from the sources to use a custom 
media type. See 
here: 
https://github.com/akka/akka-http/blob/5932237a86a432d623fafb1e84eeeff56d7485fe/akka-http-marshallers-java/akka-http-jackson/src/main/java/akka/http/javadsl/marshallers/jackson/Jackson.java#L27-L27

Johannes

On Saturday, May 6, 2017 at 8:12:06 AM UTC+2, Thibault Meyer wrote:
>
> Hi Greg,
>
>
> how to do this ? I see no arguments in completeOKWithFuture 
> or Jackson.marshaller() to do this. I'm using Java version of akka-http.
>
> Thanks
>
> Le samedi 6 mai 2017 00:49:14 UTC+2, Greg Methvin a écrit :
>>
>> There is no charset parameter defined for application/json. See 
>> https://www.iana.org/assignments/media-types/application/json
>>
>> The encoding should not be determined by looking at the charset 
>> parameter, but rather by looking at the first four octets: 
>> https://tools.ietf.org/html/rfc4627#section-3
>>
>> If you absolutely have to deal with non-conforming parsers, the custom 
>> media type is probably the right way to go.
>>
>> On Fri, May 5, 2017 at 2:23 PM, Alan Burlison  
>> wrote:
>>
>>> On 05/05/17 18:42, Alex Cozzi wrote:
>>>
>>> funny, I was looking into this yesterday!
 I am by no means an expert, but I think that what it is happening is the
 following:
 application/json implies UTF-8: if you look at the MediaType class from
 akka http you se the declaration:

>>>
>>> I believe you are correct, http://www.ietf.org/rfc/rfc4627.txt says:
>>>
>>> 3.  Encoding
>>> JSON text SHALL be encoded in Unicode.  The default encoding is UTF-8.
>>>
>>> so an explicit MIME type isn't required if the content is UTF-8 - but 
>>> other JSON frameworks do often seem to include the UTF-8 charset.
>>>
>>> As far as I know there's no "proper" way of forcing the addition of the 
>>> charset to the application/json content type in Akka-HTTP but it is 
>>> possible to hack around it:
>>>
>>> val ct = ContentType(MediaType.custom("application/json", 
>>> false).asInstanceOf[MediaType.WithOpenCharset], HttpCharset.custom("utf-8"))
>>>
>>> -- 
>>> Alan Burlison
>>> --
>>>
>>>
>>> -- 
>>>
  Read the docs: http://akka.io/docs/
>  Check the FAQ: 
> http://doc.akka.io/docs/akka/current/additional/faq.html
>  Search the archives: 
> https://groups.google.com/group/akka-user
>
 --- You received this message because you are subscribed to the 
>>> Google Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> -- 
>> *Greg Methvin*
>> *Tech Lead - Play Framework*
>> Lightbend, Inc. 
>>
>>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Re: Akka-http: how to deal with incorrect http headers

2017-05-15 Thread johannes . rudolph

Hi Florian,

can you clarify what needs improvement? Is that about client or server side?

Johannes

On Saturday, May 13, 2017 at 10:47:52 AM UTC+2, Florian Rosenberg wrote:
>
> I'm seeing a similar problem, but not wit the URI but with the 
> Strict-Transport-Header, it seems to be invalid, changing the conf to 
> ignore it has no effect. I'm using a websocket though.
>
> After debugging around for a while, I instrumented some code and see this:
>
> Illegal 'strict-transport-security' header: Invalid input 'EOI', expected 
> OWS, 'i' or token0 (line 1, column 18): max-age=31536000;
>
> It may well be that the endpoint I'm sending it to does not treat this 
> correctly and returns a 400 (Bad request), but I need to be able to ignore 
> this somehow. 
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Seed node auto down

2017-05-15 Thread 谌浏洋

Sorry for my pool English
Recently my seed node always shutdown it self. But I  never do 
"system.terminate()" in my code. 
akka version 2.4.18

Any help will be appreciate.


02:21:12.037 [c-akka.actor.default-dispatcher-18] INFO 
org.appsugar.cluster.service.akka.share.ActorShareCenter - member event 
status DOWN member Member(address = akka.tcp://c@192.168.0.16:2555, status 
= Removed)
[WARN] [05/14/2017 02:21:14.315] 
[c-akka.remote.default-remote-dispatcher-14] 
[akka.tcp://c@192.168.0.16:9413/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fc%40192.168.0.16%3A2558-16]
 
Association with remote system [akka.tcp://c@192.168.0.16:2558] has failed, 
address is now gated for [5000] ms. Reason: [Disassociated] 
[INFO] [05/14/2017 02:21:14.454] [c-akka.actor.default-dispatcher-5] 
[akka.cluster.Cluster(akka://c)] Cluster Node 
[akka.tcp://c@192.168.0.16:9413] - Shutting down myself
[INFO] [05/14/2017 02:21:14.460] [c-akka.actor.default-dispatcher-5] 
[akka.cluster.Cluster(akka://c)] Cluster Node 
[akka.tcp://c@192.168.0.16:9413] - Shutting down...
02:21:14.494 [c-akka.actor.default-dispatcher-17] INFO 
org.appsugar.cluster.service.akka.share.ActorShareCenter - member event 
status DOWN member Member(address = akka.tcp://c@192.168.0.16:2554, status 
= Removed)
02:21:14.542 [c-akka.actor.default-dispatcher-17] INFO 
org.appsugar.cluster.service.akka.share.ActorShareCenter - member event 
status DOWN member Member(address = akka.tcp://c@192.168.0.16:2558, status 
= Removed)
02:21:14.543 [c-akka.actor.default-dispatcher-17] INFO 
org.appsugar.cluster.service.akka.share.ActorShareCenter - member event 
status DOWN member Member(address = akka.tcp://c@192.168.0.16:2552, status 
= Removed)
02:21:14.588 [c-akka.actor.default-dispatcher-17] DEBUG 
org.appsugar.cluster.service.akka.share.ActorShareCenter - self member 
event  do nothing
02:21:14.604 [c-akka.actor.default-dispatcher-17] INFO 
org.appsugar.cluster.service.akka.share.ActorShareCenter - member event 
status DOWN member Member(address = akka.tcp://c@192.168.0.16:39866, status 
= Removed)
[INFO] [05/14/2017 02:21:14.724] [c-akka.actor.default-dispatcher-5] 
[akka.cluster.Cluster(akka://c)] Cluster Node 
[akka.tcp://c@192.168.0.16:9413] - Successfully shut down

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Akka Cluster Failing for Huge Data File

[akka-user] [akka-streams] StreamConverters.fromOutputStream(..) stops processing messages

Re: [akka-user] Re: Akka Http how to add charset utf-8 to the content-type header

Re: [akka-user] Re: Akka-http: how to deal with incorrect http headers

[akka-user] Seed node auto down

5 matches

Site Navigation

Mail list logo

Footer information