Github user petermaxlee commented on the issue:

    https://github.com/apache/spark/pull/15122
  
    I looked into this. I think there are two ways that you can intercept any 
calls to HDFS.
    
    The first way is slightly hacky but pretty simple. 
FileSystem.addFileSystemForTesting is a package private method that can be used 
to inject a mock file system. You can create an implementation of 
FilterFileSystem and pass it in as "file" schema. Then all accesses to local 
file system will go through your implementation. Of course, you can also use a 
mocking library to do that, but that is not as clean since FilterFileSystem is 
a public interface.
    
    The second way is more robust and does not depend on any private APIs. 
Create an implementation of FilterFileSystem by pointing to LocalFileSystem, 
e.g. call it MockFileSystem. MockFileSystem.getScheme should return 
"mockfs://". You can then use this as the path when passing to structured 
streaming. This is probably a more robust, generic solution.
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to