`compute_flat_in_array_if_unknown()` computes the flat in array property if we don't know it. We additionally added some assertions that when we have `FlatInArray::NotFlat` or `FlatInArray::Flat`, we are either dealing with a non-value class or a value class, respectively. This worked in most cases.
We now found cases with type speculation where this breaks: We could speculatively assume that something is `FlatInArray::NotFlat/Flat`, propagate that further and then later find that this contradicts the flat in array property of some non-speculative type. I've first tried to avoid introducing these inconsistencies when starting to use a speculative type. This sometimes work but not always: We could find an inconsistency way further down in the graph later which is then hard to fix. I therefore propose to accept the fact that the flat in array property can be inconsistent (i.e. being not flat in array, when actually dealing with an always flat in array value class etc.): - We set not flat in array even though the type could theoretically be flat in array. - We set flat in array even though the type could theoretically be non flat in array. We only use the flat in array property for `SubTypeCheck` and `CmpP` nodes in `Value()`. When facing such an inconsistency, we could end up "wrongly" folding a compare node. But in that case, we would trap earlier because the speculated type turns out to be wrong (see `testSuperNotFlatSubFlat()` as an example). To make this work properly, we also need to adjust `SubTypeCheck::sub()` to not only check the case when the super class is flat in array and the sub class is not but also the other way round (this missing piece was now also found as a missed optimization opportunity in the CI independently of these changes). Thanks, Christian ------------- Commit messages: - 8373878: [lworld] C2: Assertions about flat in array properties are too strong in TypePtr::compute_flat_in_array_if_unknown() Changes: https://git.openjdk.org/valhalla/pull/2162/files Webrev: https://webrevs.openjdk.org/?repo=valhalla&pr=2162&range=00 Issue: https://bugs.openjdk.org/browse/JDK-8373878 Stats: 275 lines in 5 files changed: 255 ins; 7 del; 13 mod Patch: https://git.openjdk.org/valhalla/pull/2162.diff Fetch: git fetch https://git.openjdk.org/valhalla.git pull/2162/head:pull/2162 PR: https://git.openjdk.org/valhalla/pull/2162
