Hi Venkata, I'm not aware of the FileSuite test failures. In fact I just
tried it locally on the master branch and the tests are all passing. Could
you provide more details?
The reason we want to disable the LZ4 test is because it requires the
native LZ4 library when running with Hadoop 2.x, which the Spark CI doesn't
have.
On Tue, Sep 21, 2021 at 3:46 PM Venkatakrishnan Sowrirajan <vsowr...@asu.edu>
wrote:
> Hi Chao,
>
> But there are tests in core as well failing. For
> eg: org.apache.spark.FileSuite But these tests are passing in 3.1, why do
> you think we should disable these tests for hadoop version < 3.x?
>
> Regards
> Venkata krishnan
>
>
> On Tue, Sep 21, 2021 at 3:33 PM Chao Sun <sunc...@apache.org> wrote:
>
>> I just created SPARK-36820 for the above LZ4 test issue. Will post a PR
>> there soon.
>>
>> On Tue, Sep 21, 2021 at 2:05 PM Chao Sun <sunc...@apache.org> wrote:
>>
>>> Mridul, is the LZ4 failure about Parquet? I think Parquet currently uses
>>> Hadoop compression codec while Hadoop 2.7 still depends on native lib for
>>> the LZ4. Maybe we should run the test only for Hadoop 3.2 profile.
>>>
>>> On Tue, Sep 21, 2021 at 10:08 AM Mridul Muralidharan <mri...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> Signatures, digests, etc check out fine.
>>>> Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes, this
>>>> worked fine.
>>>>
>>>> I found that including "-Phadoop-2.7" failed on lz4 tests ("native lz4
>>>> library not available").
>>>>
>>>> Regards,
>>>> Mridul
>>>>
>>>> On Tue, Sep 21, 2021 at 10:18 AM Gengliang Wang <ltn...@gmail.com>
>>>> wrote:
>>>>
>>>>> To Stephen: Thanks for pointing that out. I agree with that.
>>>>> To Sean: I made a PR
>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/34059__;!!IKRxdwAv5BmarQ!O-njQDJjvUEKCXotXCcks-Bp6M5Hvwm2lVAdEvN7Wdi_DsazPKxBtqP5St4gRBM$>
>>>>> to
>>>>> remove the test dependency so that we can start RC4 ASAP.
>>>>>
>>>>> Gengliang
>>>>>
>>>>> On Tue, Sep 21, 2021 at 8:14 PM Sean Owen <sro...@gmail.com> wrote:
>>>>>
>>>>>> Hm yeah I tend to agree. See
>>>>>> https://github.com/apache/spark/pull/33912
>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/33912__;!!IKRxdwAv5BmarQ!O-njQDJjvUEKCXotXCcks-Bp6M5Hvwm2lVAdEvN7Wdi_DsazPKxBtqP5nHr4Dvc$>
>>>>>> This _is_ a test-only dependency which makes it less of an issue.
>>>>>> I'm guessing it's not in Maven as it's a small one-off utility; we
>>>>>> _could_ just inline the ~100 lines of code in test code instead?
>>>>>>
>>>>>> On Tue, Sep 21, 2021 at 12:33 AM Stephen Coy
>>>>>> <s...@infomedia.com.au.invalid> wrote:
>>>>>>
>>>>>>> Hi there,
>>>>>>>
>>>>>>> I was going to -1 this because of the
>>>>>>> com.github.rdblue:brotli-codec:0.1.1 dependency, which is not available
>>>>>>> on
>>>>>>> Maven Central, and therefore is not available from our repository
>>>>>>> manager
>>>>>>> (Nexus).
>>>>>>>
>>>>>>> Historically most places I have worked have avoided other public
>>>>>>> maven repositories because they are not well curated. i.e artifacts with
>>>>>>> the same GAV have been known to change over time, which never happens
>>>>>>> with
>>>>>>> Maven Central.
>>>>>>>
>>>>>>> I know that I can address this by changing my settings.xml file.
>>>>>>>
>>>>>>> Anyway, I can see this biting other people so I thought that I would
>>>>>>> mention it.
>>>>>>>
>>>>>>> Steve C
>>>>>>>
>>>>>>> On 19 Sep 2021, at 1:18 pm, Gengliang Wang <ltn...@gmail.com> wrote:
>>>>>>>
>>>>>>> Please vote on releasing the following candidate as
>>>>>>> Apache Spark version 3.2.0.
>>>>>>>
>>>>>>> The vote is open until 11:59pm Pacific time September 24 and passes
>>>>>>> if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>>>>>>
>>>>>>> [ ] +1 Release this package as Apache Spark 3.2.0
>>>>>>> [ ] -1 Do not release this package because ...
>>>>>>>
>>>>>>> To learn more about Apache Spark, please see
>>>>>>> http://spark.apache.org/
>>>>>>> <https://urldefense.com/v3/__https://aus01.safelinks.protection.outlook.com/?url=http*3A*2F*2Fspark.apache.org*2F&data=04*7C01*7Cscoy*40infomedia.com.au*7C40d4b33b156b46c92cd808d97b1c3142*7C45d5407150f849caa59f9457123dc71c*7C0*7C0*7C637676183289473704*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&sdata=mf1Z69isdBZnI7I5MS0ss3GmCmN*2FiyHqfrnKrG4U4qk*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUl!!IKRxdwAv5BmarQ!O-njQDJjvUEKCXotXCcks-Bp6M5Hvwm2lVAdEvN7Wdi_DsazPKxBtqP5GeuooqI$>
>>>>>>>
>>>>>>> The tag to be voted on is v3.2.0-rc3 (commit
>>>>>>> 96044e97353a079d3a7233ed3795ca82f3d9a101):
>>>>>>> https://github.com/apache/spark/tree/v3.2.0-rc3
>>>>>>> <https://urldefense.com/v3/__https://aus01.safelinks.protection.outlook.com/?url=https*3A*2F*2Fgithub.com*2Fapache*2Fspark*2Ftree*2Fv3.2.0-rc3&data=04*7C01*7Cscoy*40infomedia.com.au*7C40d4b33b156b46c92cd808d97b1c3142*7C45d5407150f849caa59f9457123dc71c*7C0*7C0*7C637676183289473704*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&sdata=0QYDm8FEE9Zikf8*2F6x2SvFfjlsqNyarpMd9*2B2xjwnhY*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJQ!!IKRxdwAv5BmarQ!O-njQDJjvUEKCXotXCcks-Bp6M5Hvwm2lVAdEvN7Wdi_DsazPKxBtqP50OS0eBg$>
>>>>>>>
>>>>>>> The release files, including signatures, digests, etc. can be found
>>>>>>> at:
>>>>>>> https://dist.apache.org/repos/dist/dev/spark/v3.2.0-rc3-bin/
>>>>>>> <https://urldefense.com/v3/__https://aus01.safelinks.protection.outlook.com/?url=https*3A*2F*2Fdist.apache.org*2Frepos*2Fdist*2Fdev*2Fspark*2Fv3.2.0-rc3-bin*2F&data=04*7C01*7Cscoy*40infomedia.com.au*7C40d4b33b156b46c92cd808d97b1c3142*7C45d5407150f849caa59f9457123dc71c*7C0*7C0*7C637676183289483662*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&sdata=OAS4bFev*2FZNByxF4*2Bs8*2FcZCv*2BxZwd2D3K6ayeKGYxhs*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSU!!IKRxdwAv5BmarQ!O-njQDJjvUEKCXotXCcks-Bp6M5Hvwm2lVAdEvN7Wdi_DsazPKxBtqP5d8KSciE$>
>>>>>>>
>>>>>>> Signatures used for Spark RCs can be found in this file:
>>>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>> <https://urldefense.com/v3/__https://aus01.safelinks.protection.outlook.com/?url=https*3A*2F*2Fdist.apache.org*2Frepos*2Fdist*2Fdev*2Fspark*2FKEYS&data=04*7C01*7Cscoy*40infomedia.com.au*7C40d4b33b156b46c92cd808d97b1c3142*7C45d5407150f849caa59f9457123dc71c*7C0*7C0*7C637676183289483662*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&sdata=hLzYUBYBg0NQLnFKnCX1iD2HRM1zyqaVuyVQF82UKaQ*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUl!!IKRxdwAv5BmarQ!O-njQDJjvUEKCXotXCcks-Bp6M5Hvwm2lVAdEvN7Wdi_DsazPKxBtqP5XgT-Irw$>
>>>>>>>
>>>>>>> The staging repository for this release can be found at:
>>>>>>>
>>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1390
>>>>>>> <https://urldefense.com/v3/__https://aus01.safelinks.protection.outlook.com/?url=https*3A*2F*2Frepository.apache.org*2Fcontent*2Frepositories*2Forgapachespark-1390&data=04*7C01*7Cscoy*40infomedia.com.au*7C40d4b33b156b46c92cd808d97b1c3142*7C45d5407150f849caa59f9457123dc71c*7C0*7C0*7C637676183289493632*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&sdata=vMnpXpJZuq49WKmnuvAmaiXzTi9dvSbFQxr4yL8cGDI*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJQ!!IKRxdwAv5BmarQ!O-njQDJjvUEKCXotXCcks-Bp6M5Hvwm2lVAdEvN7Wdi_DsazPKxBtqP5kXJJdV0$>
>>>>>>>
>>>>>>> The documentation corresponding to this release can be found at:
>>>>>>> https://dist.apache.org/repos/dist/dev/spark/v3.2.0-rc3-docs/
>>>>>>> <https://urldefense.com/v3/__https://aus01.safelinks.protection.outlook.com/?url=https*3A*2F*2Fdist.apache.org*2Frepos*2Fdist*2Fdev*2Fspark*2Fv3.2.0-rc3-docs*2F&data=04*7C01*7Cscoy*40infomedia.com.au*7C40d4b33b156b46c92cd808d97b1c3142*7C45d5407150f849caa59f9457123dc71c*7C0*7C0*7C637676183289493632*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&sdata=vG*2Fdyv3k*2F6*2FMk7I*2FMFx*2FTyCvZIfQ0AewqEiv28uyFbY*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!IKRxdwAv5BmarQ!O-njQDJjvUEKCXotXCcks-Bp6M5Hvwm2lVAdEvN7Wdi_DsazPKxBtqP5eWe5Ot0$>
>>>>>>>
>>>>>>> The list of bug fixes going into 3.2.0 can be found at the following
>>>>>>> URL:
>>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/12349407
>>>>>>>
>>>>>>> <https://urldefense.com/v3/__https://aus01.safelinks.protection.outlook.com/?url=https*3A*2F*2Fissues.apache.org*2Fjira*2Fprojects*2FSPARK*2Fversions*2F12349407&data=04*7C01*7Cscoy*40infomedia.com.au*7C40d4b33b156b46c92cd808d97b1c3142*7C45d5407150f849caa59f9457123dc71c*7C0*7C0*7C637676183289503577*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&sdata=HgrjsbZT6Hu6kQB17X3HhGo6qiq95e9nKS5O1z*2F4*2BOA*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSU!!IKRxdwAv5BmarQ!O-njQDJjvUEKCXotXCcks-Bp6M5Hvwm2lVAdEvN7Wdi_DsazPKxBtqP5WeYogC4$>
>>>>>>>
>>>>>>> This release is using the release script of the tag v3.2.0-rc3.
>>>>>>>
>>>>>>>
>>>>>>> FAQ
>>>>>>>
>>>>>>> =========================
>>>>>>> How can I help test this release?
>>>>>>> =========================
>>>>>>> If you are a Spark user, you can help us test this release by taking
>>>>>>> an existing Spark workload and running on this release candidate,
>>>>>>> then
>>>>>>> reporting any regressions.
>>>>>>>
>>>>>>> If you're working in PySpark you can set up a virtual env and install
>>>>>>> the current RC and see if anything important breaks, in the
>>>>>>> Java/Scala
>>>>>>> you can add the staging repository to your projects resolvers and
>>>>>>> test
>>>>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>>>>> you don't end up building with a out of date RC going forward).
>>>>>>>
>>>>>>> ===========================================
>>>>>>> What should happen to JIRA tickets still targeting 3.2.0?
>>>>>>> ===========================================
>>>>>>> The current list of open tickets targeted at 3.2.0 can be found at:
>>>>>>> https://issues.apache.org/jira/projects/SPARK
>>>>>>> <https://urldefense.com/v3/__https://aus01.safelinks.protection.outlook.com/?url=https*3A*2F*2Fissues.apache.org*2Fjira*2Fprojects*2FSPARK&data=04*7C01*7Cscoy*40infomedia.com.au*7C40d4b33b156b46c92cd808d97b1c3142*7C45d5407150f849caa59f9457123dc71c*7C0*7C0*7C637676183289503577*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&sdata=bdvunkTLnI4iJuvRkJzq8P*2ByGa9LeEqoTvvXRvxlfIY*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSU!!IKRxdwAv5BmarQ!O-njQDJjvUEKCXotXCcks-Bp6M5Hvwm2lVAdEvN7Wdi_DsazPKxBtqP5-kZ7Cx4$>
>>>>>>> and
>>>>>>> search for "Target Version/s" = 3.2.0
>>>>>>>
>>>>>>> Committers should look at those and triage. Extremely important bug
>>>>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>>>>> be worked on immediately. Everything else please retarget to an
>>>>>>> appropriate release.
>>>>>>>
>>>>>>> ==================
>>>>>>> But my bug isn't fixed?
>>>>>>> ==================
>>>>>>> In order to make timely releases, we will typically not hold the
>>>>>>> release unless the bug in question is a regression from the previous
>>>>>>> release. That being said, if there is something which is a regression
>>>>>>> that has not been correctly targeted please ping me or a committer to
>>>>>>> help target the issue.
>>>>>>>
>>>>>>>
>>>>>>> This email contains confidential information of and is the copyright
>>>>>>> of Infomedia. It must not be forwarded, amended or disclosed without
>>>>>>> consent of the sender. If you received this message by mistake, please
>>>>>>> advise the sender and delete all copies. Security of transmission on the
>>>>>>> internet cannot be guaranteed, could be infected, intercepted, or
>>>>>>> corrupted
>>>>>>> and you should ensure you have suitable antivirus protection in place.
>>>>>>> By
>>>>>>> sending us your or any third party personal details, you consent to (or
>>>>>>> confirm you have obtained consent from such third parties) to
>>>>>>> Infomedia’s
>>>>>>> privacy policy. http://www.infomedia.com.au/privacy-policy/
>>>>>>> <https://urldefense.com/v3/__http://www.infomedia.com.au/privacy-policy/__;!!IKRxdwAv5BmarQ!O-njQDJjvUEKCXotXCcks-Bp6M5Hvwm2lVAdEvN7Wdi_DsazPKxBtqP5K8y8EOo$>
>>>>>>>
>>>>>>
