[GitHub] incubator-predictionio pull request #371: [PIO-61] Add S3 Model Data Reposit...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-predictionio/pull/371


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-predictionio pull request #371: [PIO-61] Add S3 Model Data Reposit...

[marevol]

Github user marevol commented on a diff in the pull request:


https://github.com/apache/incubator-predictionio/pull/371#discussion_r113695041
  
--- Diff: storage/s3/build.sbt ---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import PIOBuild._
+
+name := "apache-predictionio-data-s3"
+
+libraryDependencies ++= Seq(
+  "org.apache.predictionio" %% "apache-predictionio-core" % version.value 
% "provided",
+  "com.google.guava"% "guava" % "14.0.1"  
% "provided",
+  "com.amazonaws"   % "aws-java-sdk-s3"   % "1.11.118",
+  "org.scalatest"   %% "scalatest"% "2.1.7" % 
"test")
+
+parallelExecution in Test := false
+
+pomExtra := childrenPomExtra.value
+
+assemblyOption in assembly := (assemblyOption in 
assembly).value.copy(includeScala = false)
+
+assemblyShadeRules in assembly := Seq(
+  ShadeRule.rename("org.apache.http.**" -> 
"shadeio.data.s3.http.@1").inAll,
+  ShadeRule.rename("com.fasterxml.**" -> 
"shadeio.data.s3.fasterxml.@1").inAll
+)
--- End diff --

Checking on Scala 2.11/Spark 2.1/ES 5.1&5.3, I could not reproduce it yet...
My test script file is 
[test_all.sh](https://github.com/jpioug/incubator-predictionio/blob/express-v1/test_all.sh)
 to build pio and run recommender template.
Is it reproduced in Docker tests? I'd like to know steps to reproduce it...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-predictionio pull request #371: [PIO-61] Add S3 Model Data Reposit...

[marevol]

Github user marevol commented on a diff in the pull request:


https://github.com/apache/incubator-predictionio/pull/371#discussion_r113576948
  
--- Diff: storage/s3/build.sbt ---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import PIOBuild._
+
+name := "apache-predictionio-data-s3"
+
+libraryDependencies ++= Seq(
+  "org.apache.predictionio" %% "apache-predictionio-core" % version.value 
% "provided",
+  "com.google.guava"% "guava" % "14.0.1"  
% "provided",
+  "com.amazonaws"   % "aws-java-sdk-s3"   % "1.11.118",
+  "org.scalatest"   %% "scalatest"% "2.1.7" % 
"test")
+
+parallelExecution in Test := false
+
+pomExtra := childrenPomExtra.value
+
+assemblyOption in assembly := (assemblyOption in 
assembly).value.copy(includeScala = false)
+
+assemblyShadeRules in assembly := Seq(
+  ShadeRule.rename("org.apache.http.**" -> 
"shadeio.data.s3.http.@1").inAll,
+  ShadeRule.rename("com.fasterxml.**" -> 
"shadeio.data.s3.fasterxml.@1").inAll
+)
--- End diff --

Thank you for the info.
My environment is Scala 2.10 and Spark 1.6.3.
I'll check it on Spark 2.1.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-predictionio pull request #371: [PIO-61] Add S3 Model Data Reposit...

[mars]

Github user mars commented on a diff in the pull request:


https://github.com/apache/incubator-predictionio/pull/371#discussion_r113494975
  
--- Diff: storage/s3/build.sbt ---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import PIOBuild._
+
+name := "apache-predictionio-data-s3"
+
+libraryDependencies ++= Seq(
+  "org.apache.predictionio" %% "apache-predictionio-core" % version.value 
% "provided",
+  "com.google.guava"% "guava" % "14.0.1"  
% "provided",
+  "com.amazonaws"   % "aws-java-sdk-s3"   % "1.11.118",
+  "org.scalatest"   %% "scalatest"% "2.1.7" % 
"test")
+
+parallelExecution in Test := false
+
+pomExtra := childrenPomExtra.value
+
+assemblyOption in assembly := (assemblyOption in 
assembly).value.copy(includeScala = false)
+
+assemblyShadeRules in assembly := Seq(
+  ShadeRule.rename("org.apache.http.**" -> 
"shadeio.data.s3.http.@1").inAll,
+  ShadeRule.rename("com.fasterxml.**" -> 
"shadeio.data.s3.fasterxml.@1").inAll
+)
--- End diff --

Hi @marevol,

We're building a [predictionio-incubating branch to add authentication to 
the Elasticsearch REST 
client](https://github.com/apache/incubator-predictionio/pull/372) with Scala 
2.11 & Spark 2.1.

I believe it was during a runtime ES REST client call to `performRequest` 
suddenly an underlying method signature could not be located in 
`org.apache.http` package. @dszeto helped me to discover that the 
`shadeio.data` name was being incorrectly resolved. So, I tried [removing the 
shade 
rule](https://github.com/apache/incubator-predictionio/pull/372/files#diff-55cfeb297edd310e1efa8b6ac8bdbae6L39)
 and it started working. No more errors for that branch.

Maybe the difference is that we're using Spark 2.1?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-predictionio pull request #371: [PIO-61] Add S3 Model Data Reposit...

[marevol]

Github user marevol commented on a diff in the pull request:


https://github.com/apache/incubator-predictionio/pull/371#discussion_r113334667
  
--- Diff: storage/s3/build.sbt ---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import PIOBuild._
+
+name := "apache-predictionio-data-s3"
+
+libraryDependencies ++= Seq(
+  "org.apache.predictionio" %% "apache-predictionio-core" % version.value 
% "provided",
+  "com.google.guava"% "guava" % "14.0.1"  
% "provided",
+  "com.amazonaws"   % "aws-java-sdk-s3"   % "1.11.118",
+  "org.scalatest"   %% "scalatest"% "2.1.7" % 
"test")
+
+parallelExecution in Test := false
+
+pomExtra := childrenPomExtra.value
+
+assemblyOption in assembly := (assemblyOption in 
assembly).value.copy(includeScala = false)
+
+assemblyShadeRules in assembly := Seq(
+  ShadeRule.rename("org.apache.http.**" -> 
"shadeio.data.s3.http.@1").inAll,
+  ShadeRule.rename("com.fasterxml.**" -> 
"shadeio.data.s3.fasterxml.@1").inAll
+)
--- End diff --

@mars could you provide info?
ES storage also conflicts with httpclient jar file of Spark.
I think an exception occurs if removing it, so it will be needed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-predictionio pull request #371: [PIO-61] Add S3 Model Data Reposit...

[dszeto]

Github user dszeto commented on a diff in the pull request:


https://github.com/apache/incubator-predictionio/pull/371#discussion_r113326594
  
--- Diff: storage/s3/build.sbt ---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import PIOBuild._
+
+name := "apache-predictionio-data-s3"
+
+libraryDependencies ++= Seq(
+  "org.apache.predictionio" %% "apache-predictionio-core" % version.value 
% "provided",
+  "com.google.guava"% "guava" % "14.0.1"  
% "provided",
+  "com.amazonaws"   % "aws-java-sdk-s3"   % "1.11.118",
+  "org.scalatest"   %% "scalatest"% "2.1.7" % 
"test")
+
+parallelExecution in Test := false
+
+pomExtra := childrenPomExtra.value
+
+assemblyOption in assembly := (assemblyOption in 
assembly).value.copy(includeScala = false)
+
+assemblyShadeRules in assembly := Seq(
+  ShadeRule.rename("org.apache.http.**" -> 
"shadeio.data.s3.http.@1").inAll,
+  ShadeRule.rename("com.fasterxml.**" -> 
"shadeio.data.s3.fasterxml.@1").inAll
+)
--- End diff --

I think @mars is seeing some issues with 
https://github.com/apache/incubator-predictionio/blob/develop/storage/elasticsearch/build.sbt#L42.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-predictionio pull request #371: [PIO-61] Add S3 Model Data Reposit...

[marevol]

Github user marevol commented on a diff in the pull request:


https://github.com/apache/incubator-predictionio/pull/371#discussion_r112782773
  
--- Diff: storage/s3/build.sbt ---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import PIOBuild._
+
+name := "apache-predictionio-data-s3"
+
+libraryDependencies ++= Seq(
+  "org.apache.predictionio" %% "apache-predictionio-core" % version.value 
% "provided",
+  "com.google.guava"% "guava" % "14.0.1"  
% "provided",
+  "com.amazonaws"   % "aws-java-sdk-s3"   % "1.11.118",
+  "org.scalatest"   %% "scalatest"% "2.1.7" % 
"test")
+
+parallelExecution in Test := false
+
+pomExtra := childrenPomExtra.value
+
+assemblyOption in assembly := (assemblyOption in 
assembly).value.copy(includeScala = false)
+
+assemblyShadeRules in assembly := Seq(
+  ShadeRule.rename("org.apache.http.**" -> 
"shadeio.data.s3.http.@1").inAll,
+  ShadeRule.rename("com.fasterxml.**" -> 
"shadeio.data.s3.fasterxml.@1").inAll
+)
--- End diff --

aws-java-sdk-s3 depends on them, but they conflict with different version 
files in spark-assembly jar.
To avoid this conflict, the package replacement is needed in storage/s3.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-predictionio pull request #371: [PIO-61] Add S3 Model Data Reposit...

[marevol]

GitHub user marevol opened a pull request:

https://github.com/apache/incubator-predictionio/pull/371

[PIO-61] Add S3 Model Data Repository



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/marevol/incubator-predictionio storage_s3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-predictionio/pull/371.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #371


commit 853d39939c737ddd1ada572db1d3454bb2d4e1aa
Author: Shinsuke Sugaya 
Date:   2017-04-17T09:43:25Z

add S3 model data repository




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---