RE: adding shutdownmanagerhook to spark.

2019-05-13 Thread Nasrulla Khan Haris 
Thanks Sean

Nasrulla

-Original Message-
From: Sean Owen  
Sent: Monday, May 13, 2019 4:16 PM
To: Nasrulla Khan Haris 
Cc: dev@spark.apache.org
Subject: Re: adding shutdownmanagerhook to spark.

Spark just adds a hook to the mechanism that Hadoop exposes. You can do the 
same. You shouldn't use Spark's.

On Mon, May 13, 2019 at 6:11 PM Nasrulla Khan Haris 
 wrote:
>
> HI All,
>
>
>
> I am trying to add shutdown hook, but looks like shutdown manager object 
> requires the package to be spark only, is there any other API that can help 
> me to do this ?
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Fapache%2Fspark%2Fblob%2Fv2.4.0%2Fcore%2Fsrc%2Fmain%2Fscala%2F
> org%2Fapache%2Fspark%2Futil%2FShutdownHookManager.scaladata=01%7C
> 01%7CNasrulla.Khan%40microsoft.com%7C8aadef9c8cf844139eca08d6d7f90058%
> 7C72f988bf86f141af91ab2d7cd011db47%7C1sdata=Si4C7iF8PN5%2FJDDQy8F
> BzzuFh2XsYD%2BfjMtT5wLxbr4%3Dreserved=0
>
>
>
> I can see that hiveserver2 in 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fspark%2Fcommit%2F515708d5f33d5acdb4206c626192d1838f8e691fdata=01%7C01%7CNasrulla.Khan%40microsoft.com%7C8aadef9c8cf844139eca08d6d7f90058%7C72f988bf86f141af91ab2d7cd011db47%7C1sdata=AGNpXhauHp4Wkbe%2BVesAbO5H%2BJFhwApMCRcAi2oROl0%3Dreserved=0
>  uses ShutdownHookManager in different package. But It expects me to have the 
> package name “org.apache.spark”
>
>
>
> Thanks,
>
> Nasrulla
>
>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: adding shutdownmanagerhook to spark.

2019-05-13 Thread Sean Owen 
Spark just adds a hook to the mechanism that Hadoop exposes. You can
do the same. You shouldn't use Spark's.

On Mon, May 13, 2019 at 6:11 PM Nasrulla Khan Haris
 wrote:
>
> HI All,
>
>
>
> I am trying to add shutdown hook, but looks like shutdown manager object 
> requires the package to be spark only, is there any other API that can help 
> me to do this ?
>
> https://github.com/apache/spark/blob/v2.4.0/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala
>
>
>
> I can see that hiveserver2 in 
> https://github.com/apache/spark/commit/515708d5f33d5acdb4206c626192d1838f8e691f
>  uses ShutdownHookManager in different package. But It expects me to have the 
> package name “org.apache.spark”
>
>
>
> Thanks,
>
> Nasrulla
>
>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org