Github user dongjoon-hyun commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20590#discussion_r167803208
  
    --- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala
 ---
    @@ -160,6 +162,25 @@ abstract class OrcSuite extends OrcTest with 
BeforeAndAfterAll {
           }
         }
       }
    +
    +  // This should be tested manually because it raises OOM intentionally
    +  // in order to cause `Leaked filesystem connection`. The test suite 
dies, too.
    +  ignore("SPARK-23399 Register a task completion listner first for 
OrcColumnarBatchReader") {
    +    withSQLConf(SQLConf.ORC_VECTORIZED_READER_BATCH_SIZE.key -> 
s"${Int.MaxValue}") {
    +      withTempDir { dir =>
    +        val basePath = dir.getCanonicalPath
    +        Seq(0).toDF("a").write.format("orc").save(new Path(basePath, 
"first").toString)
    +        Seq(1).toDF("a").write.format("orc").save(new Path(basePath, 
"second").toString)
    +        val df = spark.read.orc(
    +          new Path(basePath, "first").toString,
    +          new Path(basePath, "second").toString)
    +        val e = intercept[SparkException] {
    +          df.collect()
    +        }
    +        assert(e.getCause.isInstanceOf[OutOfMemoryError])
    +      }
    +    }
    +  }
    --- End diff --
    
    Hi, All. 
    The above test case generates the same leakage reported in JIRA.
    And, this PR fixes that. Please try this in IntelliJ with the original code.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to