[
https://issues.apache.org/jira/browse/NIFI-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391548#comment-16391548
]
ASF GitHub Bot commented on NIFI-4946:
--------------------------------------
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2521#discussion_r173216342
--- Diff:
nifi-nar-bundles/nifi-spark-bundle/nifi-livy-processors/src/main/java/org/apache/nifi/processors/livy/ExecuteSparkInteractive.java
---
@@ -83,6 +83,62 @@
.expressionLanguageSupported(true)
.build();
+ public static final PropertyDescriptor IS_BATCH_JOB = new
PropertyDescriptor.Builder()
+ .name("exec-spark-iactive-is_batch_job")
+ .displayName("Is Batch Job")
+ .description("If true, the `Code` part is ignored and the flow
file from previous stage is considered "
+ + "as a triggering event and not as code for Spark
session. When `Wait` state is self routed"
+ + "the livy json response flow file from previous
Spark job is used to poll the job status"
+ + "for sucess or failure")
+ .required(true)
+ .allowableValues("true", "false")
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor PY_FILES = new
PropertyDescriptor.Builder()
+ .name("exec-spark-iactive-pyfiles")
+ .displayName("pyFiles")
+ .description("Python files to be used in this batch session
that includes *.py, *.zip files")
+ .required(false)
+ .addValidator(StandardValidators.createURLorFileValidator())
+ .expressionLanguageSupported(false)
+ .build();
+
+ public static final PropertyDescriptor JAR_FILES = new
PropertyDescriptor.Builder()
+ .name("exec-spark-iactive-jarfiles")
+ .displayName("jars")
+ .description("jars to be used in this batch session")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(false)
+ .build();
+
+ public static final PropertyDescriptor NAME = new
PropertyDescriptor.Builder()
--- End diff --
Is this supposed to be the Spark app name? Looks it is never used anywhere
other than adding to the `PropertyDescriptor` list
> nifi-spark-bundle : Adding support for pyfiles, file, jars options
> ------------------------------------------------------------------
>
> Key: NIFI-4946
> URL: https://issues.apache.org/jira/browse/NIFI-4946
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
> Affects Versions: 1.6.0
> Environment: Ubuntu 16.04, IntelliJ
> Reporter: Mageswaran
> Priority: Major
> Fix For: 1.6.0
>
> Attachments: nifi-spark-options.png, nifi-spark.png
>
>
> Adding support for submitting PySpark based Sparks jobs (which is normally
> structured as modules) over Livy on existing "ExecuteSparkInteractive"
> processor.
> This is done by reading file paths for pyfiles and file and an option from
> user whether the processor should trigger a batch job or not.
> [https://livy.incubator.apache.org/docs/latest/rest-api.html]
> *Current Work flow Logic ( [https://github.com/apache/nifi/pull/2521
> )|https://github.com/apache/nifi/pull/2521]*
> * Check whether the processor has to handle code or submit a Spark job
> * Read incoming flow file
> ** If batch == true
> *** If flow file matches Livy `batches` JSON response through `wait` loop
> **** Wait for Status Check Interval
> **** Read the state
> **** If state is `running` route it to `wait` or if it is `success` or
> `dead` route it accordingly
> *** Else
> **** Ignore the flow file
> **** Trigger the Spark job over Livy `batches` endpoint
> **** Read the state of the submitted job
> **** If state is `running` route it to `wait` or if it is `success` or
> `dead` route it accordingly
> ** Else:
> *** Existing Logic to handle `Code`
>
> !nifi-spark-options.png!
> !nifi-spark.png!
>
> Thanks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
