Ah no sorry, so in the load image, the crawl has just kicked off on the driver node which is why its flagged red and the load is spiking. https://pasteboard.co/K5QHOJN.png here's the cluster now its been running a while. The red node is still (and is always every time I tested it) the driver node.
Tom On Wed, Jun 9, 2021 at 8:03 PM Sean Owen <sro...@gmail.com> wrote: > Where do you see that ... I see 3 executors busy at first. If that's the > crawl then ? > > On Wed, Jun 9, 2021 at 1:59 PM Tom Barber <t...@spicule.co.uk> wrote: > >> Yeah :) >> >> But it's all running through the same node. So I can run multiple tasks >> of the same type on the same node(the driver), but I can't run multiple >> tasks on multiple nodes. >> >> On Wed, Jun 9, 2021 at 7:57 PM Sean Owen <sro...@gmail.com> wrote: >> >>> Wait. Isn't that what you were trying to parallelize in the first place? >>> >>> On Wed, Jun 9, 2021 at 1:49 PM Tom Barber <t...@spicule.co.uk> wrote: >>> >>>> Yeah but that something else is the crawl being run, which is triggered >>>> from inside the RDDs, because the log output is slowly outputting crawl >>>> data. >>>> >>>> >> Spicule Limited is registered in England & Wales. Company Number: >> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston >> Road, Brighton, England, BN1 6AF. VAT No. 251478891. >> >> >> All engagements are subject to Spicule Terms and Conditions of Business. >> This email and its contents are intended solely for the individual to whom >> it is addressed and may contain information that is confidential, >> privileged or otherwise protected from disclosure, distributing or copying. >> Any views or opinions presented in this email are solely those of the >> author and do not necessarily represent those of Spicule Limited. The >> company accepts no liability for any damage caused by any virus transmitted >> by this email. If you have received this message in error, please notify us >> immediately by reply email before deleting it from your system. Service of >> legal notice cannot be effected on Spicule Limited by email. >> > -- Spicule Limited is registered in England & Wales. Company Number: 09954122. Registered office: First Floor, Telecom House, 125-135 Preston Road, Brighton, England, BN1 6AF. VAT No. 251478891. All engagements are subject to Spicule Terms and Conditions of Business. This email and its contents are intended solely for the individual to whom it is addressed and may contain information that is confidential, privileged or otherwise protected from disclosure, distributing or copying. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Spicule Limited. The company accepts no liability for any damage caused by any virus transmitted by this email. If you have received this message in error, please notify us immediately by reply email before deleting it from your system. Service of legal notice cannot be effected on Spicule Limited by email.
