>But here, we depend on Hadoop so we cannot isolate it. User does not necessarily depend on Hadoop. The platform pulls the Hadoop dependency into application code irrespective of user's influence. The platform also pulls in other non-hadoop related N random dependencies from application's perspective.
The complexity comes from the effort needed to refactor parts of the code to isolate the UDF and its dependencies from platform dependencies. It¹s a very large initiative (that¹s where the motivation part comes) but not an impossible one. Shading is a shady approach but it¹s an easy way out at smaller scale. At larger scale it starts foraying into the pain around application jar/platform jar deployment discussion and JVM PermGen memory issues. ‹ Chetan
