Where do you see that ... I see 3 executors busy at first. If that's the crawl then ?
On Wed, Jun 9, 2021 at 1:59 PM Tom Barber <t...@spicule.co.uk> wrote: > Yeah :) > > But it's all running through the same node. So I can run multiple tasks of > the same type on the same node(the driver), but I can't run multiple tasks > on multiple nodes. > > On Wed, Jun 9, 2021 at 7:57 PM Sean Owen <sro...@gmail.com> wrote: > >> Wait. Isn't that what you were trying to parallelize in the first place? >> >> On Wed, Jun 9, 2021 at 1:49 PM Tom Barber <t...@spicule.co.uk> wrote: >> >>> Yeah but that something else is the crawl being run, which is triggered >>> from inside the RDDs, because the log output is slowly outputting crawl >>> data. >>> >>> > Spicule Limited is registered in England & Wales. Company Number: > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston > Road, Brighton, England, BN1 6AF. VAT No. 251478891. > > > All engagements are subject to Spicule Terms and Conditions of Business. > This email and its contents are intended solely for the individual to whom > it is addressed and may contain information that is confidential, > privileged or otherwise protected from disclosure, distributing or copying. > Any views or opinions presented in this email are solely those of the > author and do not necessarily represent those of Spicule Limited. The > company accepts no liability for any damage caused by any virus transmitted > by this email. If you have received this message in error, please notify us > immediately by reply email before deleting it from your system. Service of > legal notice cannot be effected on Spicule Limited by email. >
