Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/21146
FYI, I just did a kinesis test run with this PR on a JVM with the unlimited
JCE installed (explicitly verified by shasum of the relevant JARs); failure
with cert errors.
```
success on second attempt after a Kinesis throttling exception
- retry success on second attempt after a Kinesis dependency exception
- retry failed after a shutdown exception
- retry failed after an invalid state exception
- retry failed after unexpected exception
- retry failed after exhausting all retries
WithAggregationKinesisBackedBlockRDDSuite:
[2018-08-13 22:01:12.872175] [0x000070000b6f9000] [info]
[kinesis_producer.cc:79] Created pipeline for stream
"KinesisTestUtils-3116048591482212471"
[2018-08-13 22:01:12.872918] [0x000070000b6f9000] [info] [shard_map.cc:83]
Updating shard map for stream "KinesisTestUtils-3116048591482212471"
[2018-08-13 22:01:13.040026] [0x000070000b905000] [error]
[http_client.cc:148] Failed to open connection to
monitoring.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:13.054034] [0x000070000b905000] [error]
[http_client.cc:148] Failed to open connection to
kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:15.943229] [0x000070000b905000] [error]
[http_client.cc:148] Failed to open connection to
kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:18.924473] [0x000070000b905000] [error]
[http_client.cc:148] Failed to open connection to
kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:21.919673] [0x000070000b905000] [error]
[http_client.cc:148] Failed to open connection to
kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:25.121685] [0x000070000b905000] [error]
[http_client.cc:148] Failed to open connection to
kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:27.925785] [0x000070000b905000] [error]
[http_client.cc:148] Failed to open connection to
kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:30.917030] [0x000070000b905000] [error]
[http_client.cc:148] Failed to open connection to
kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:33.960962] [0x000070000b905000] [error]
[http_client.cc:148] Failed to open connection to
kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:36.926987] [0x000070000b905000] [error]
[http_client.cc:148] Failed to open connection to
kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:39.912528] [0x000070000b905000] [error]
[http_client.cc:148] Failed to open connection to
kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:42.911743] [0x000070000b905000] [error]
[http_client.cc:148] Failed to open connection to
kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
org.apache.spark.streaming.kinesis.WithAggregationKinesisBackedBlockRDDSuite
*** ABORTED ***
java.lang.IllegalArgumentException: requirement failed: Need data to be
sent to multiple shards
at scala.Predef$.require(Predef.scala:224)
at
org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests$$anonfun$beforeAll$1.apply$mcV$sp(KinesisBackedBlockRDDSuite.scala:47)
at
org.apache.spark.streaming.kinesis.KinesisFunSuite$class.runIfTestsEnabled(KinesisFunSuite.scala:41)
at
org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:25)
at
org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:42)
at
org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:212)
at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:52)
at org.scalatest.Suite$class.callExecuteOnSuite$1(Suite.scala:1210)
at org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1257)
...
the assert failure is possibly a followon from the previous problem
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
