Katie Horn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/159761
Change subject: WIP: DO NOT MERGE - Generic audit logic and worldpay audit job
......................................................................
WIP: DO NOT MERGE - Generic audit logic and worldpay audit job
Change-Id: I4020e3ac9e84b1b2c3bfbb44a874abd95e8e5df8
TODO: Pretty much everything.
---
A sites/all/modules/wmf_audit/wmf_audit.info
A sites/all/modules/wmf_audit/wmf_audit.module
A sites/all/modules/wmf_audit/worldpay/worldpay_audit.drush.inc
A sites/all/modules/wmf_audit/worldpay/worldpay_audit.info
A sites/all/modules/wmf_audit/worldpay/worldpay_audit.module
5 files changed, 2,289 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm
refs/changes/61/159761/1
diff --git a/sites/all/modules/wmf_audit/wmf_audit.info
b/sites/all/modules/wmf_audit/wmf_audit.info
new file mode 100644
index 0000000..d82630a
--- /dev/null
+++ b/sites/all/modules/wmf_audit/wmf_audit.info
@@ -0,0 +1,9 @@
+name = WMF Audit
+description = Auditing framework for contacts and contributions that come in
through nightly reconciliation files
+core = 7.x
+dependencies[] = queue2civicrm
+dependencies[] = civicrm
+dependencies[] = wmf_common
+dependencies[] = wmf_civicrm
+package = WMF Audit
+configure = admin/config/wmf_audit
diff --git a/sites/all/modules/wmf_audit/wmf_audit.module
b/sites/all/modules/wmf_audit/wmf_audit.module
new file mode 100644
index 0000000..a827510
--- /dev/null
+++ b/sites/all/modules/wmf_audit/wmf_audit.module
@@ -0,0 +1,1489 @@
+<?php
+
+define( 'WMF_AUDIT_PAYMENTS_LOGS_DIR', '/usr/local/src/logs/' );
+
+/**
+ * Implementation of hook_menu()
+ */
+function wmf_audit_menu() {
+ $items = array();
+
+ $items['admin/config/wmf_audit'] = array(
+ 'title' => 'WMF Audit',
+ 'description' => t('Configure WMF audit settings.'),
+ 'access arguments' => array('administer wmf_audit'),
+ 'page callback' => 'drupal_get_form',
+ 'page arguments' => array( 'wmf_audit_settings' ),
+ );
+
+ return $items;
+}
+
+/**
+ * Callback for menu
+ */
+function wmf_audit_settings() {
+ $form['wmf_audit_log_files_dir'] = array(
+ '#type' => 'textfield',
+ '#title' => t( 'Path to directory containing WMF payments logs' ),
+ '#required' => TRUE,
+ '#default_value' => variable_get( 'wmf_audit_log_files_dir',
WMF_AUDIT_PAYMENTS_LOGS_DIR ),
+ );
+ return system_settings_form( $form );
+}
+
+
+/** Refactor Line Starts Here **/
+/**
+ *
+ * First piece would seem to be a thing that takes recon files, and returns
the distilled (and normal!) version of only what's missing from the db. This
crosses the line of universal and specific several times. Enjoy the drupal
submodule voodoo problem.
+ Open, parse, normalize one recon file
+ Throw out everything that we already know about at this stage, because
by *far*, the most processor-intensive part is finding the missing information
from these dang things in the payments logs. We want to not have to do that as
frequently as possible.
+ ...and if there isn't anything in that file we don't know about,
move the whole recon file to completed.
+ Pass back the rest for processing in the second piece.
+ Dumb Current Thing: Negative transactions. I did something
extremely silly there... like add a negative sign before the oid as the array
key, or some other BS. Stop it.
+ Controller decides to go for more recon files or not, based on run
parameters. Maybe the default run is only... the last three days or so, to
appropriately handle logrotate timing issues.
+ Second piece takes all the missing transactions, rebuilds them, and stuffs
them into the donations or refund queue
+ The Great Log Hunt Mechanism. This is going to take a great deal of
investigation and tuning, as it's the majority of processor time spent.
+ Make sure all the right logs are available in distilled format in
the working dir.
+ Get the distribution of dates for when we think the payments were
initiated. <--ow. Hopefully we've already built this in the process of finding
all the missings.
+ Open the most likely... f***. How does this work? Are we happy
with that? (no)
+ Don't re-investigate log files you've already looked at.
+ Don't look at everydamnthing. In fact, delete really old cruft.
+ If we can't find the relevant log no matter what we do, *and the
transaction itself is of a certain age*, we should rebuild with defaults and be
glad about that too.
+ Rebuilding the message
+ Send the message to the right queue (donations or refunds)
+ Third... output? Reporting? File moving aroundness? Cleanup?
+ As stated... somewhere above, it would be sort of neat to somehow
target payments log files that haven't had any action in a long time, and
delete them.
+ but that's really fiddly from here. I think.
+ *
+ * TODO: Figure out other drush params...
+ * ** Run all, or run latest
+ * ** Make missing, or... don't do that.
+ *
+ */
+
+/**
+ * The main function.
+ * @param string $submod_prefix The prefix for the... submodule?
+ * @param array $drush_params Runtime parameters supplied by the drush job
+ */
+function wmf_audit_main($submod_prefix, $drush_params) {
+
+ //take care of run modes...
+ $run_all = false;
+ if ( array_key_exists('run_all', $drush_params ) ){
+ $run_all = true;
+ }
+
+ //find out what the situation is with the available recon files, by date
+ $recon_files = wmf_audit_get_all_recon_files($submod_prefix);
+
+ //get missing transactions from one or more recon files
+ //let's just assume that the default mode will be to pop off the top three
(most recent) at this point. :)
+ //...Three, because Shut Up.
+ $count = count($recon_files);
+ if ( $count > 3 && !$run_all ){
+ $count = 3;
+ }
+
+ for ( $i = 0; $i < $count; ++$i ){
+ //process them.
+ $file = array_pop($recon_files);
+ $results = wmfa_execute('parse_recon_file', $submod_prefix, array('file'
=> $file));
+ echo count($results) . 'results found';
+ die(__FUNCTION__ . ' is where we currently drop dead. WIP: Anything in
this function below this line is probably BS.');
+ }
+
+ //get the date distribution
+ //go transaction hunting
+ //send rebuilt messages
+ //output something nice, possibly to fredge if we're sufficiently cool.
+}
+
+/**
+ * Returns an array of the full paths to all valid reconciliation files
+ * directory.
+ * @return array full paths to all wr1-type files
+ */
+function wmf_audit_get_all_recon_files( $submod_prefix ){
+ $files_directory = wmfa_execute( 'get_recon_directory', $submod_prefix
);
+ //foreach file in the directory, if it matches our pattern, add it to
the array.
+ $files = array();
+ if ( $handle = opendir( $files_directory ) ){
+ while ( ( $file = readdir( $handle ) ) !== false ){
+ if ( strpos( $file, wmfa_execute(
'get_recon_filename_pattern', $submod_prefix ) ) ){
+ $filedate = wmfa_execute( 'get_recon_file_date', $submod_prefix,
array( $file ) ); //which is not the same thing as the edited date... probably.
+ $files[$filedate] = $files_directory . $file;
+ }
+ }
+ closedir($handle);
+ ksort($files); //@TODO: Do we want this to be reverse or forward by key
(date)? Forward so we can array_pop, I think...
+ return $files;
+
+ } else {
+ //can't open the directory at all. Problem.
+ wmf_audit_log_error( "Can't open directory $files_directory",
'FILE_ISSUES' ); //should be fatal
+ }
+ return false;
+}
+
+/**
+ * Short function name because I'm tired of it.
+ * @param type $function_name_base
+ * @param type $submod_prefix
+ * @param type $parameters
+ * @return boolean
+ */
+function wmfa_execute($function_name_base, $submod_prefix = null, $parameters
= array()) {
+ $function_name = $submod_prefix . '_audit_' . $function_name_base;
+ if ( function_exists( $function_name ) ) {
+ return call_user_func_array($function_name, $parameters );
+ } else {
+ //serious issues, you have.
+ wmf_audit_log_error( "Missing required function $function_name",
'MISSING_FUNCTION' ); //should be fatal
+ }
+ return false;
+}
+
+/** EVERYTHING BELOW THIS LINE IS NONSENSE. * */
+
+/**
+ * Find Globalcollect transactions not recorded in civicrm_contribution
+ */
+function wmf_audit_find_missing_gc_trxns( $transactions ) {
+ $missing_trxns = array( );
+
+ $dbs = wmf_civicrm_get_dbs();
+
+ // check for matching ids - if it doesnt match, it's missing
+ wmf_audit_echo( "Checking for missing transactions" );
+ foreach ( array_keys( $transactions ) as $order_id ) {
+ $error = false;
+ $found = false;
+ if ( recon_is_negative_txn( $transactions[$order_id] ) ) {
+ //look for this transaction's positive counterpart
before we go
+ //looking for the refund, becuase we need the whole
civi transaction ID.
+ //Remember to regrind the order_id, though. At this
point, it's got a negative in front of it.
+
+// $matches =
wmf_civicrm_get_contributions_from_gateway_id('globalcollect',
recon_order_id($transactions[$order_id]));
+ $matches =
wmf_audit_get_contributions_from_whatever_this_is(recon_order_id($transactions[$order_id]));
+
+ if ( !$matches ) {
+ wmf_audit_log_error("Could not find parent (or
any related) transaction for refund/chargeback " .
recon_order_id($transactions[$order_id]) . print_r($transactions[$order_id],
true), 'MISSING_PARENT');
+ $error = true;
+ } else {
+ //Make sure the match we have is a base
transaction.
+ $parent = null;
+ foreach ( $matches as $index => $match ) {
+ if ( trim(
$match['parent_contribution_id'] ) === '' ) {
+ if ( !is_null( $parent ) ) {
+ wmf_audit_log_error(
"Too many parents for refund/chargeback $order_id", 'PARENT_OVERFLOW' );
+ $error = true;
+ } else {
+ $parent = $match;
+ $copy_fields = array(
+ 'entity_id',
//for the parent!
+
'contribution_type_id',
+
'payment_instrument_id',
+
'gateway_account',
+ );
+ foreach ( $copy_fields
as $field ){
+
$transactions[$order_id][$field] = trim( $match[$field] );
+ }
+ }
+ }
+ }
+
+ if ( !$error && is_null( $parent ) ){
+ wmf_audit_log_error( "Could not find
parent transaction for refund/chargeback " . recon_order_id(
$transactions[$order_id] ) . print_r( $transactions[$order_id], true ),
'MISSING_PARENT' );
+ $error = true;
+ }
+
+ if ( !$error && !is_null(
$parent['contribution_recur_id'] ) ){
+ //this is a recurring transaction, and
we're not cool enough to handle that yet. Defer to later.
+ wmf_audit_log_error( "Not cool enough
to handle recurring refunds/chargebacks yet for " . recon_order_id(
$transactions[$order_id] ) . print_r( $transactions[$order_id], true ),
'RECURRING_CBK' );
+ $error = true; //well, not really an
"error", per se, but we do want to ignore this for now.
+ }
+
+ $matches =
wmf_audit_get_child_contributions_from_whatever_this_is(recon_order_id($transactions[$order_id]));
+ if ( !$error && is_array( $matches ) && count( $matches ) ) {
+ $im_a = wmf_civicrm_get_civi_id(
'contribution_type_id', recon_get_negative_type( $transactions[$order_id] ) );
+ foreach ( $matches as $match ) {
+ //None of these are themselves
parents.
+ //check to make sure they're
not the same type of thing we are.
+ if ( $im_a ===
$match['contribution_type_id'] ) {
+ $found = true;
//already went there.
+ } else {
+ wmf_audit_log_error(
"Refund/Chargeback Collision on order id " . recon_order_id(
$transactions[$order_id] ), 'CBK+RFD' );
+ $error = true;
+ //TODO: Seems like some
kind of alert should happen
+ //here if there's a
sibling transaction, but at this
+ //point in time for all
we know, this is totally normal...
+ }
+ }
+ }
+ }
+ } elseif
(wmf_audit_get_contributions_from_whatever_this_is($order_id)) {
+ $found = true;
+ }
+
+ if ( $found && !$error ){
+ wmf_audit_echo( '.' );
+ watchdog( 'wmf_audit', 'Found Globalcollect: @id',
array( '@id' => $order_id ), WATCHDOG_DEBUG );
+ }
+ if ( $error && !$found ){
+ wmf_audit_echo( 'X' );
+ //the exact thing that happened should already have
been logged up the page a bit.
+ }
+ if ( !$found && !$error ) {
+ wmf_audit_echo( '!' );
+ $missing_trxns[$order_id] = $transactions[$order_id];
+ watchdog( 'wmf_audit', 'Missing Globalcollect: @tx_id
@pm, @pc', array(
+ "@tx_id" => $order_id,
+ "@pm" =>
$transactions[$order_id]['Payment-method'],
+ "@pc" => recon_currency(
$transactions[$order_id] ) ), WATCHDOG_DEBUG );
+ }
+ if ( $found && $error ){
+ wmf_audit_echo( 'Hit the programmer with a rolled-up
newspaper. This shouldn\'t be possible.' );
+ die();
+ }
+ }
+
+ wmf_audit_echo( count( $missing_trxns ) . " missing records from a
possible " . count( $transactions ) . " total records found." );
+ return $missing_trxns;
+}
+
+
+function wmf_audit_parse_all_wr1(){
+ civicrm_initialize();
+
+ //output data about the current user if we're running verbose
+ wmf_audit_echo( print_r( posix_getpwuid( posix_getuid() ), true ), true
);
+
+ $actual_start_time = microtime(true);
+ $test_mode = variable_get( 'wmf_audit_test_mode',
CONTRIBUTION_AUDIT_TEST_MODE );
+
+ $recon_files = wmf_audit_get_all_recon_files();
+
+ $missing = array();
+ $errorful_transactions = array();
+
+ $missing = wmf_audit_find_more_missing($recon_files);
+ if ( !$missing ){
+ wmf_audit_echo("No missing transactions found. Done!");
+ return;
+ }
+
+ $counts = array();
+
+ //Here is the place where we should take care of all the negatives.
+ $missing = wmf_audit_mark_rfd_cbk( $missing );
+
+ while ( count( $missing ) && wmf_audit_can_continue() ) {
+ $local_found_ids = array();
+ $log = wmf_audit_get_next_log( $missing, $counts );
+ if ( is_null( $log ) ){
+ continue;
+ }
+
+ wmf_audit_echo("Trying log $log");
+ $tried_count = 0;
+ foreach ( $missing as $order_id => $data ){
+ ++$tried_count;
+ $date = recon_date( $data );
+ if ( is_null($date) ){
+ wmf_audit_echo('X');
+ continue;
+ }
+ $logdata = wmf_audit_get_log_data_by_order_id(
$order_id, $log );
+ if ( is_null( $logdata ) ){
+ wmf_audit_echo('X');
+ continue;
+ }
+ if ($logdata === false){
+ wmf_audit_echo('.');;
+ continue;
+ }
+
+ //This isn't going to work unless you're connecting to
a database that has the row you're looking for.
+ $missing_txn_message = wmf_audit_format_data_for_stomp(
$data, $logdata['donor_data'], $logdata['contribution_id'] );
+ if ( is_null( $missing_txn_message ) ){
+ //log this out in a list of troubled
transactions.
+ $info = array(
+ 'contribution_id' =>
$logdata['contribution_id'],
+ 'log' => $log,
+ 'data' => $data,
+ 'logdata' => $logdata,
+ );
+ $errorful_transactions[] = $info;
+ wmf_audit_echo('X');
+ continue;
+ }
+
+ if ( $test_mode ){
+ $local_found_ids[] = $order_id;
+ wmf_audit_echo('!');
+ } else {
+ if ( wmf_audit_send_stomp( 'donations',
$missing_txn_message ) ){
+ watchdog('wmf_audit', __FUNCTION__ . ':
Message sent to stomp successfully: ' . print_r( $missing_txn_message, true ),
array(), WATCHDOG_INFO);
+ $local_found_ids[] = $order_id;
+ wmf_audit_echo('!');
+ } else {
+ $wd_message = __FUNCTION__ . ': Sending
message to stomp failed: ' . print_r( $missing_txn_message, true );
+ $drush_message = "Failed sending STOMP
message to queue.";
+ wmf_audit_log_error( $wd_message,
"STOMP_BAD_SEND", $drush_message );
+ return;
+ }
+ }
+
+ }
+ wmf_audit_echo("\n");
+
+ //Output here, and add these stats to the data ball.
+ $logdate = wmf_audit_get_log_date_by_file($log);
+ $counts[$logdate] = count( $local_found_ids );
+ wmf_audit_echo($counts[$logdate] . " records found in $log");
+ if ( $counts[$logdate] > 0 ){ //stop the log noise...
+ wmf_audit_results_ball( array('Transactions Found',
$log), $counts[$logdate] );
+ }
+
+ //now, we unset the array keys we found, so as not to corrupt
the foreach we were in.
+ foreach ($local_found_ids as $id) {
+ unset($missing[$id]);
+ }
+
+ }
+ //And loop.
+
+ //if we're running in make_missing mode, do that.
+ $make_missing = variable_get( 'wmf_audit_make_missing', false );
+ if ( $make_missing && count( $missing ) ){
+ //make all the remaining missing transactions with data
defaults.
+ foreach ( $missing as $order_id => $data ) {
+ $missing_txn_message = wmf_audit_format_data_for_stomp(
$data );
+
+ //but then we have to send the transaction...
+ if ( $test_mode ){
+ $local_built_ids[] = $order_id;
+ wmf_audit_echo('!');
+ } else {
+ if ( wmf_audit_send_stomp( 'donations',
$missing_txn_message ) ){
+ watchdog('wmf_audit', __FUNCTION__ . ':
Message sent to stomp successfully: ' . print_r( $missing_txn_message, true ),
array(), WATCHDOG_INFO);
+ $local_built_ids[] = $order_id;
+ wmf_audit_echo('!');
+ } else {
+ $wd_message = __FUNCTION__ . ': Sending
message to stomp failed: ' . print_r( $missing_txn_message, true );
+ $drush_message = "Failed sending STOMP
message to queue.";
+ wmf_audit_log_error( $wd_message,
"STOMP_BAD_SEND", $drush_message );
+ return;
+ }
+ }
+ }
+ }
+
+ if ( count( $errorful_transactions ) ){
+ wmf_audit_echo("Errorful Transactions: " . print_r(
$errorful_transactions, true ));
+ wmf_audit_results_ball( array('Final', 'Errors'),
count($errorful_transactions) );
+ }
+
+ //more log noise killing
+ foreach ( $counts as $log => $count ){
+ if ( $count === 0 ){
+ unset( $counts[$log] );
+ }
+ }
+
+ wmf_audit_echo("Transactions found by log:" . print_r( $counts, true ));
+ wmf_audit_echo("Still Missing: " . count($missing) . ".");
+ if ( count($missing) ){
+ wmf_audit_echo(" Good luck with that.");
+ wmf_audit_results_ball( array('Final', 'Missing'),
count($missing) );
+ }
+ wmf_audit_echo("\n");
+
+ $time = microtime(true) - $actual_start_time;
+ wmf_audit_echo("Total Runtime: $time");
+ wmf_audit_results_ball(array('Final', 'Runtime'), $time);
+}
+
+/**
+ * Finds more missing transactions to rebuild, in the remaining wr1 files.
+ * @staticvar bool $run_all_wr1 Whether or not we're configured to operate in
+ * Run All mode.
+ * @param array $recon_files An array of full paths to wr1 files
+ * @return mixed an array of entries in the wr1 files that are not present in
+ * civicrm
+ */
+function wmf_audit_find_more_missing(&$recon_files){
+ static $run_all_wr1 = null;
+
+ if (is_null($run_all_wr1)){
+ $run_all_wr1 = variable_get( 'wmf_audit_run_all_wr1', false );
+ }
+
+ $missing = array();
+
+ while ( ( $run_all_wr1 || !count( $missing ) ) && count( $recon_files )
){
+ $local_missing = array();
+ $file = array_shift( $recon_files );
+ $recon_data = wmf_audit_parse_recon_file($file);
+
+ if ( $recon_data ){
+ $local_missing = wmf_audit_find_missing_gc_trxns(
$recon_data );
+ $missing += $local_missing;
+ wmf_audit_results_ball(array('Missing at Start',
$file), count($local_missing));
+ }
+
+ if ( !count( $local_missing ) ){
+ wmf_audit_move_completed_wr1( $file );
+ }
+ }
+
+ if ( !count($missing) ){
+ return false;
+ } else {
+ return $missing;
+ }
+}
+
+/**
+ * Just parse one wr1 file.
+ * @staticvar string $parser_directory The directory where the parser lives.
+ * @param string $file Absolute location of the wr1 you want to parse
+ * @return mixed An array of wr1 data, or false
+ */
+function wmf_audit_parse_recon_file( $file ){
+ static $parser_directory = null;
+ if (is_null($parser_directory)){
+ $parser_directory = variable_get( 'wmf_audit_recon_parser_dir',
CONTRIBUTION_AUDIT_WR1_PARSER_DIR );
+ }
+ require_once( $parser_directory . 'Wr1DataRecord.php' );
+
+ $recon_data = array();
+ wmf_audit_echo("Parsing $file");
+
+ $recon_parser = new Wr1DataFileParser( $file );
+ $start_time = microtime(true);
+ foreach( $recon_parser->getRecordIterator() as $record ) {
+ wmf_audit_echo( wmf_audit_echochar( $record ) );
+ $order_id = recon_order_id( $record );
+ if ( recon_is_negative_txn( $record ) ){
+ //have to do this to avoid collisions between the
original and the chargeback / refund / whatever
+ $order_id = '-' . $order_id;
+ }
+ $recon_data[ $order_id ] = $record;
+ }
+ wmf_audit_echo("\n");
+ $time = microtime(true) - $start_time;
+ wmf_audit_echo("$file parsed in $time");
+ if ( count( $recon_data ) ){
+ return $recon_data;
+ }
+ return false;
+}
+
+
+function wmf_audit_format_data_for_stomp( $wr1, $donor_data = null,
$contribution_tracking_id = null ){
+ $fake_db = variable_get( 'wmf_audit_fake_db',
CONTRIBUTION_AUDIT_FAKE_DB );
+ $make_missing = variable_get( 'wmf_audit_make_missing', false );
+ //you should probably make sure the data between the two sources match
up before we do anything else
+
+ //just knowing we _can_ make missing transactions isn't enough. We need
to
+ //know if this particular one is still missing.
+ $this_transaction_is_missing = false;
+ if ( is_null($donor_data) ){
+ //set this as true down the page a bit when ctid is missing. We
could be
+ //running in some kind of half-missing mode.
+ $this_transaction_is_missing = true;
+ }
+
+ $unstaged_amount = 0;
+
+ if ( $this_transaction_is_missing && $make_missing ) {
+ $donor_data = wmf_audit_make_default_donor_data( $wr1 );
+ }
+
+ //if we have $donor_data, make sure it's rational
+ if ( !$this_transaction_is_missing && ( (int)recon_amount($wr1) !==
(int)$donor_data['AMOUNT'] ) ){
+ $amount_problem = true;
+ if ( array_key_exists( 'Over-under-amount-local', $wr1 ) &&
(int)$wr1['Over-under-amount-local'] > 0 ){
+ //then the mismatch is expected, and with a little more
checking...
+ // Invoice-amount-deliv = what we think it is.
+ // Payment-amount = what actually happened.
+ if ((int) $wr1['Invoice-amount-deliv'] === (int)
$donor_data['AMOUNT'] || (int) $wr1['Order-amount-deliv'] === (int)
$donor_data['AMOUNT']) {
+ $unstaged_amount = (int)recon_amount($wr1)/100; //it's a GC thing.
+ $amount_problem = false;
+ }
+ }
+ if ( $amount_problem ) {
+ $message = __FUNCTION__ . " $contribution_tracking_id
Payment amount mismatch! Aborting.";
+ wmf_audit_log_error( $message, "WR1_DATA_FORMAT" );
+ return null;
+ }
+ }
+
+ $unstaged_amount = (int)recon_amount($wr1)/100; //it's a GC thing.
+
+ if ( !$this_transaction_is_missing && ( recon_currency( $wr1 ) !==
$donor_data['CURRENCYCODE'] ) ){
+ $message = __FUNCTION__ . " $contribution_tracking_id Currency
code mismatch! Aborting. ";
+ wmf_audit_log_error( $message, "WR1_DATA_FORMAT" );
+ return null;
+ }
+
+ $recurring = false;
+ if ( array_key_exists( 'ORDERTYPE', $donor_data ) &&
$donor_data['ORDERTYPE'] == '4' ){
+ $recurring = true;
+ }
+
+ //in case only our second half is missing...
+ if ( is_null( $contribution_tracking_id ) ){
+ $this_transaction_is_missing = true;
+ }
+
+ if ( $this_transaction_is_missing && $make_missing ){
+ $contribution_tracking =
wmf_audit_make_contribution_tracking_data( $wr1 );
+ }
+
+ if ( !$this_transaction_is_missing ) {
+ if ( !$fake_db ) {
+ $contribution_tracking =
wmf_audit_get_contribution_tracking_data( $contribution_tracking_id );
+ } else {
+ $contribution_tracking =
wmf_audit_get_contribution_tracking_data( 1 );
+ }
+ if ( $contribution_tracking ){
+ $payment_method_utm = explode( '.',
$contribution_tracking['utm_source'] );
+ $payment_method_utm = $payment_method_utm[2];
+
+ $idiotic_date_format = $contribution_tracking['ts'];
+ $real_timestamp = date_parse( $idiotic_date_format );
+
+ $contribution_tracking['ts'] = mktime(
+ $real_timestamp['hour'],
+ $real_timestamp['minute'],
+ $real_timestamp['second'],
+ $real_timestamp['month'],
+ $real_timestamp['day'],
+ $real_timestamp['year']
+ );
+ //echo " $idiotic_date_format = $real_timestamp \n";
+
+ } else {
+ $payment_method_utm = null;
+ }
+ }
+
+ $payment_submethod = false;
+ if ( !$this_transaction_is_missing && array_key_exists(
'PAYMENTPRODUCTID', $donor_data ) ){
+ $payment_info =
wmf_audit_get_payment_submethod($donor_data['PAYMENTPRODUCTID']);
+ $payment_method = $payment_info['group'];
+ $payment_submethod = $payment_info['code'];
+ if ( !$fake_db && $payment_method != $payment_method_utm ){
+ $message = __FUNCTION__ . " Payment Method Mismatch on
$contribution_tracking_id. UTM indicates '$payment_method_utm', but xml was
sent for '$payment_method'. Using xml value.";
+ wmf_audit_log_error( $message, "WR1_DATA_FORMAT" );
+ //this is not fatal, so don't return here.
+ }
+ }
+
+ if ( $this_transaction_is_missing ){
+ $payment_info = wmf_audit_make_payment_method_info( $wr1 );
+ if ( array_key_exists( 'payment_method', $payment_info ) ){
+ $payment_method = $payment_info['payment_method'];
+ } else {
+ $message = __FUNCTION__ . " Making missing records:
Failed to find a payment method for " . $wr1['Payment-method'];
+ wmf_audit_log_error( $message, "WR1_DATA_FORMAT" );
+ return null;
+ }
+ if ( array_key_exists( 'payment_submethod', $payment_info ) ){
+ $payment_submethod = $payment_info['payment_submethod'];
+ }
+ }
+
+ //start it off with the stuff that's always there...
+ $stomp_data = array(
+ 'contribution_tracking_id' => $contribution_tracking_id,
+ 'gateway' => 'globalcollect',
+ 'gross' => $unstaged_amount,
+ 'payment_method' => $payment_method,
+ );
+
+ if ( $payment_submethod ){
+ $stomp_data['payment_submethod'] = $payment_submethod;
+ }
+
+ if( $recurring ){
+ $stomp_data['recurring'] = 1;
+ }
+
+ if ( $make_missing ){
+ $stomp_data['audit_hole'] = true;
+ }
+
+
+ //now, we have three sources for this mess: $wr1, $donor_data, and
$contribution_tracking. Map 'em.
+ $stomp_field_map = array(
+ 'optout' => array( 'contribution_tracking', 'optout' ),
+ 'anonymous' => array( 'contribution_tracking', 'anonymous' ),
+ 'comment' => array( 'contribution_tracking', 'note' ), //I
think.
+ 'size',
+ 'premium_language',
+ 'utm_source' => array( 'contribution_tracking', 'utm_source' ),
+ 'utm_medium' => array( 'contribution_tracking', 'utm_medium' ),
+ 'utm_campaign' => array( 'contribution_tracking',
'utm_campaign' ),
+ 'language' => array( 'contribution_tracking', 'language' ),
//probably the best one to use.
+ 'referrer' => array( 'contribution_tracking', 'referrer' ),
+ 'email' => array( 'donor_data', 'EMAIL' ),
+ 'first_name' => array( 'donor_data', 'FIRSTNAME' ),
+ 'middle_name',
+ 'last_name' => array( 'donor_data', 'SURNAME' ),
+ 'street_address' => array( 'donor_data', 'STREET' ),
+ 'city' => array( 'donor_data', 'CITY' ),
+ 'state' => array( 'donor_data', 'STATE' ),
+ 'country' => array( 'donor_data', 'COUNTRYCODE' ),
+ 'postal_code' => array( 'donor_data', 'ZIP' ),
+ 'gateway_txn_id' => array( 'donor_data', 'ORDERID' ),
+ 'response', //? Maybe some dummy value for "found it in the
audit phase"
+ 'currency' => array( 'donor_data', 'CURRENCYCODE' ),
+ 'date' => array( 'contribution_tracking', 'ts' ),
//double-check how the dates work
+ );
+
+ foreach ( $stomp_field_map as $key => $location ){
+ if( is_array( $location ) ){
+ $source = $location[0];
+ $source_key = $location[1];
+ if ( is_array( ${$source} ) && array_key_exists(
$source_key, ${$source} ) ){
+ $stomp_data[$key] = ${$source}[$source_key];
+ } else {
+ $stomp_data[$key] = '';
+ }
+ } else {
+ $stomp_data[$location] = '';
+ }
+ }
+
+ return $stomp_data;
+
+
+}
+
+/**
+ * Makes default donor data according to current run config, if the message
+ * cannot be rebuilt from logs.
+ * @param array $wr1 Transaction parsed into an array
+ * @return array Donor data with appropriate keys
+ */
+function wmf_audit_make_default_donor_data( $wr1 ){
+
+ $donor_data = array(
+ 'AMOUNT' => (int)recon_amount($wr1),
+ 'EMAIL' => '',
+ 'FIRSTNAME' => '',
+ 'SURNAME' => '',
+ 'STREET' => '',
+ 'CITY' => '',
+ );
+
+ $donor_data['ORDERID'] = recon_order_id($wr1);
+
+ //deal with the complete insanity that is the way currency codes are
dealt with in the wr1 files...
+ //cc always comes in as...
+ if (array_key_exists('Transaction-currency', $wr1)){
+ $donor_data['CURRENCYCODE'] = $wr1['Transaction-currency'];
+ } else {
+ //absolute madness. But I'm curious. So... here we go.
+ $currency_code_fields = array(
+ 'Invoice-currency-deliv',
+ 'Order-currency-deliv',
+ 'Invoice-currency-local',
+ 'Order-currency-local',
+ 'Payment-currency',
+ 'Currency-due',
+ 'Over-under-currency-local', //the heck you say.
+ );
+
+ $currency = null;
+ foreach ( $currency_code_fields as $field ){
+ if (array_key_exists( $field, $wr1 )){
+ if ( is_null($currency) ){
+ $currency = trim($wr1[$field]);
+ } else {
+ if ( trim($wr1[$field]) != '' &&
trim($currency) != trim($wr1[$field]) ){
+ //Alert the press: It's in five
different fields for a reason!
+ //I want this to email me, but
for right now...
+ wmf_audit_echo(print_r($wr1,
true));
+ wmf_audit_echo("Check it: The
currencies are all messed up!");
+ die();
+ }
+ }
+ }
+ }
+ $donor_data['CURRENCYCODE'] = $currency;
+ }
+
+
+// TODO: Figure out if/how WR1 files mark recurring and set:
+// $donor_data['ORDERTYPE'] = '4'
+
+ return $donor_data;
+}
+
+/**
+ * Makes everything we need to fake contribution tracking data.
+ * So: Mostly the timestamp.
+ * @param array $wr1 Transaction parsed into an array
+ * @return array
+ */
+function wmf_audit_make_contribution_tracking_data( $wr1 ){
+
+ $return = array(
+ //don't care about the actual values here: They're not saved
downstream.
+ //It's just so we can eyeball where the stomp message came
from, really.
+ 'utm_source' => 'gc_wr1',
+ 'utm_medium' => 'gc_wr1',
+ 'utm_campaign' => 'gc_wr1',
+ );
+
+ $return['ts'] = recon_date_to_timestamp( $wr1 );
+ return $return;
+
+}
+
+
+function wmf_audit_get_contribution_tracking_data( $contribution_tracking_id ){
+ $dbs = wmf_civicrm_get_dbs();
+ $dbs->push( 'donations' );
+
+ $query = "SELECT * FROM contribution_tracking WHERE id = :id";
+ $result = db_query( $query, array( ':id' => $contribution_tracking_id )
);
+ if ( !$result->rowCount() ) {
+ watchdog( 'wmf_audit', 'Missing Contribution Tracking data: @id
', array(
+ "@id" => $contribution_tracking_id ), WATCHDOG_DEBUG );
+ } else {
+ watchdog( 'wmf_audit', 'Found Contribution: @id', array( '@id'
=> $contribution_tracking_id ), WATCHDOG_DEBUG );
+ }
+
+ $result = $result->fetchAssoc();
+
+ return $result;
+}
+
+
+/**
+ * Goes through the log file directory, and does a few things:
+ * * gunzips the .gz files
+ * * Renames the raw unzips to something we're expecting
+ * * Kills off log data that has been distilled already into .iop files
+ * @return array A list of full paths to the files in their buckets
+ */
+function wmf_audit_groom_logs(){
+ $files_directory = variable_get( 'wmf_audit_log_files_dir',
WMF_AUDIT_PAYMENTS_LOGS_DIR );
+ $files_directory_alt = variable_get(
'wmf_audit_log_files_dir_secondary', WMF_AUDIT_PAYMENTS_LOGS_DIR );
+
+ $files = array(
+ 'gz' => array(),
+ 'rename' => array(),
+ 'log' => array(),
+ 'iop' => array()
+ );
+ wmf_audit_echo("Grooming log files");
+ $dates = array();
+ if ( $handle = opendir( $files_directory ) ){
+ while ( ( $file = readdir( $handle ) ) !== false ){
+ $full_path = $files_directory . $file;
+ $date = wmf_audit_get_log_date_by_file($full_path);
+ if ( !$date ){
+ continue;
+ }
+ $dates[] = $date;
+ if ( strpos( $file, '.gz' ) ){
+ $files['gz'][] = $full_path;
+ continue;
+ }
+ if ( preg_match( '/\d{8}_iop\.log/', $file ) ){
+ $files['iop'][$date] = $full_path;
+ //rekey
+ continue;
+ }
+ if ( preg_match( '/globalcollect_\d{8}\.log/', $file )
){
+ $files['log'][$date] = $full_path;
+ //rekey
+ continue;
+ }
+ if ( preg_match( '/globalcollect.\d{8}/', $file ) ){
+ $files['rename'][] = $full_path;
+ continue;
+ }
+ }
+ }
+ closedir($handle);
+
+ if ( $files_directory_alt != $files_directory ){
+ //and if we're missing any files from between what's in that
dir, and NOW,
+ //go get them at the alt and add them to the .gz list.
+ sort($dates, SORT_NUMERIC);
+ //we have to add one to the next date, because the dates in the
audit script are all -1.
+ $earliest_log_date = wmf_common_date_add_days( $dates[0], 1);
+ $today_date = wmf_common_date_get_today_string();
+
+ $copy_us = wmf_common_date_get_date_gap( $earliest_log_date,
$today_date );
+ $copy_us = array_diff( $copy_us, $dates ); //get all date holes.
+
+ foreach ( $copy_us as $missing_date ){
+ $copy_me = 'payments-globalcollect-' . $missing_date .
'.gz';
+ if ( file_exists( $files_directory_alt . $copy_me ) ){
+ $cmd = "cp $files_directory_alt$copy_me
$files_directory";
+ exec( escapeshellcmd($cmd), $ret, $errorlevel );
+ if ( !file_exists( $files_directory . $copy_me
) ){
+ wmf_audit_echo("FILE PROBLEM: Something
went wrong with $cmd");
+ } else {
+ $files['gz'][] = $files_directory .
$copy_me;
+ }
+ } else {
+ wmf_audit_echo("Missing log file
$files_directory_alt$copy_me does not exist!");
+ }
+ }
+ }
+
+ //handle the gz files first
+ foreach ($files['gz'] as $key => $gzfile){
+ wmf_audit_echo("Gunzipping $gzfile");
+ $cmd = "gunzip $gzfile";
+ exec( escapeshellcmd($cmd), $ret, $errorlevel );
+ //now check to make sure the file you expect exists
+ $newfile = substr( $gzfile, 0, strlen($gzfile) - 3 );
+ if (!file_exists($newfile)){
+ wmf_audit_echo("FILE PROBLEM: Something went wrong with
$cmd : $newfile doesn't exist.");
+ } else {
+ //add it to the other array.
+ $files['rename'][] = $newfile;
+ if (file_exists($gzfile)){ //still?
+ unlink($gzfile);
+ }
+ unset($files['gz'][$key]);
+ }
+ }
+
+ //handle the renames
+ foreach ($files['rename'] as $key => $rename_me){
+ wmf_audit_echo("Renaming $rename_me");
+ //'/globalcollect.\d{8}_\d{6}/'
+ //becomes
+ //'/globalcollect_\d{8}\.log/'
+ $new_name = explode('/', $rename_me);
+ //and the file name should be the last array key.
+ $just_file = array_pop($new_name);
+ $just_file = preg_replace(
+ array( '/(payments.)?globalcollect./' ),
+ array( 'globalcollect_' ),
+ $just_file );
+ $new_name = implode( '/', $new_name ) . '/' . $just_file .
'.log';
+ rename( $rename_me, $new_name );
+ $files['log'][] = $new_name;
+ unset($files['rename'][$key]);
+ }
+
+ //delete redundant data
+ foreach ( $files['log'] as $key => $big_log ){
+ //'/\d{8}_iop\.log/'
+ //is a distilled version of
+ //'/globalcollect_\d{8}\.log/'
+ $smallened_log = preg_replace(
+ array('/globalcollect_/', '/\.log/'),
+ array('', '_iop.log'),
+ $big_log );
+ if ( in_array($smallened_log, $files['iop']) ){
+ wmf_audit_echo("Deleting $big_log");
+ unlink( $big_log );
+ unset($files['log'][$key]);
+ }
+ }
+
+ foreach ($files as $type => $dontrekeyme){
+ ksort($files[$type]);
+ }
+
+ return $files;
+}
+
+
+function wmf_audit_mark_rfd_cbk( $missing ){
+ wmf_audit_echo('Marking refunds and chargebacks');
+ $count = 0;
+ foreach ( $missing as $index => $wr1 ){
+ if ( recon_is_negative_txn( $wr1 ) ){
+ $send_message = array(
+ 'gateway_refund_id' => 'RFD' . recon_order_id(
$wr1 ), //after intense deliberation, we don't actually care what this is at
all.
+ 'gateway_parent_id' => recon_order_id( $wr1 ),
//gateway transaction ID
+ 'gross_currency' => recon_currency( $wr1 ),
//currency code
+ 'gross' => recon_amount( $wr1 ) / 100, //amount
+ 'date' => recon_date_to_timestamp( $wr1 ),
//timestamp
+ 'gateway' => 'globalcollect', //lcase
+ 'gateway_account' => $wr1['gateway_account'],
//from DI. ?
+ 'payment_method' => '', //Argh. Not telling you.
+ 'payment_submethod' => '', //Still not telling
you.
+ 'type' => recon_get_negative_type( $wr1 ),
//refund or chargeback
+ );
+ echo print_r( $send_message, true );
+ wmf_audit_send_stomp( 'refund', $send_message );
+ $count += 1;
+ wmf_audit_echo('.');
+
+ unset( $missing[$index] );
+ }
+ }
+ wmf_audit_results_ball( 'refunds/chargebacks', $count );
+ return $missing;
+}
+
+/**
+ * Tries to intelligently select the next most likely log to knock off the
+ * majority of the remaining missing transactions, which drastically cuts down
+ * on runtime.
+ * @staticvar array $tried The dates we have tried
+ * @staticvar string $available The groomed files that are currently available
+ * in the logs directory.
+ * @param array $missing Parsed missing transaction data form the wr1 files
+ * @param array $counts Array of log data we have found, indexed by the actual
+ * date it was initiated (logdate - 1)
+ * @return type
+ */
+function wmf_audit_get_next_log( $missing, $counts ){
+ static $tried = array();
+ static $available = null;
+ if ( is_null( $available ) ){
+ $available = wmf_audit_groom_logs();
+ //$available['iop'] and $available['log'] are the ones we want.
+ //don't really care about the order: All we want at this point
are the
+ //keys, so we have to flip first or array_merge will think
they're
+ //numeric and rekey. I hate php.
+ $available = array_merge( array_flip($available['log']),
array_flip($available['iop']) );
+ rsort( $available );
+ }
+
+ //this is the var that tells us how far to cluster around the dates
we're
+ //missing, before we simply slide backwards into history.
+ $plusminus = variable_get( 'wmf_audit_plusminus',
CONTRIBUTION_AUDIT_PLUSMINUS );
+
+ //haven't found anything yet.
+ $file = null;
+
+ //If this is the first time around, kill the future.
+ if ( empty($tried) ){
+ $today = wmf_common_date_get_today_string();
+ for ( $i=0; $i<$plusminus + 1; ++$i ){
+ $tried[] = wmf_common_date_add_days( $today, $i );
+ }
+ }
+
+ //look for the next file until we either find something, or hit a fatal
error.
+ while ( wmf_audit_can_continue() && is_null( $file ) ){
+ $tryme = false;
+
+ //Find the most popular date in the missing transactions that
has not been
+ //tried yet, and look there.
+ if ( !$tryme ){
+ $dates = array();
+ foreach ( $missing as $transaction ){
+ $date = recon_date( $transaction );
+ if (is_null($date)){
+ return null;
+ }
+ if ( array_key_exists( $date, $dates ) ){
+ $dates[$date] += 1;
+ } else {
+ $dates[$date] = 1;
+ }
+ }
+
+ //now turn the array inside out and sort it by most
popular date
+ array_flip($dates);
+ arsort($dates);
+
+ foreach ( $dates as $date => $count ){
+ if ( !in_array($date, $tried) ){
+ $tryme = $date;
+ break;
+ }
+ }
+ }
+
+ //Buffer out both directions from all the dates we've found
something on.
+ if ( !$tryme ){
+ //we've tried all the log dates explicitly present in
the wr1. Now try more.
+ $more_tries = array();
+ for($i=1; $i < ($plusminus + 1); ++$i ){
+ $more_tries[] = $i;
+ $more_tries[] = $i * -1;
+ }
+
+ foreach ( $dates as $date => $count ){
+ foreach( $more_tries as $add ){
+ $newdate = wmf_common_date_add_days(
$date, $add );
+ if ( !in_array($newdate, $tried) ){
+ $tryme = $newdate;
+ break 2;
+ }
+ }
+ }
+ }
+
+ //Check everything else we have, that we haven't tried yet.
+ if ( !$tryme ){
+ foreach ( $available as $date ){
+ if ( !in_array( $date, $tried ) ){
+ $tryme = $date;
+ break;
+ }
+ }
+ }
+
+ //We have now tried everything. Kick out of the loop with an
error code that will cause can_continue to say "no".
+ if ( !$tryme ){
+ $message = __FUNCTION__ . " No 'next' log identified.
Current Missing: " . print_r( $dates, true );
+ wmf_audit_log_error( $message, "WR1_LOG_STOP" );
+ return null;
+ }
+
+ $tried[] = $tryme;
+ //now see if the file actually exists.
+ $file = wmf_audit_get_log_file_by_date( $tryme );
+ }
+ return $file;
+
+}
+
+/**
+ * Returns the log file that will contain transactions from the date passed in.
+ * @param string $date The target date of the actual log entry
+ * @return string The full path to the reduced iop file
+ */
+function wmf_audit_get_log_file_by_date( $date ){
+ //I know somebody is going to try to come back in time just to punch me
for this one, but...
+ $date = wmf_common_date_add_days( $date, 1 );
+ //I swear it makes sense. Most of the data in any given log are
actually from the previous day.
+
+ $log_directory = variable_get( 'wmf_audit_log_files_dir',
WMF_AUDIT_PAYMENTS_LOGS_DIR );
+ $file = $log_directory . $date . '_iop.log';
+ if ( file_exists( $file ) ){
+ return $file;
+ } else {
+ //try to make it.
+ $original_file = $log_directory . 'globalcollect_' . $date .
'.log';
+ if ( file_exists( $original_file ) ){
+
+ $cmd = escapeshellcmd("grep 'INSERT_ORDERWITHPAYMENT'
$original_file") . " > " . escapeshellcmd($file);
+
+ wmf_audit_echo( $cmd );
+ $ret = array();
+ exec( $cmd, $ret, $errorlevel );
+
+ if ( file_exists( $file ) ) {
+ return $file;
+ } else {
+ $message = __FUNCTION__ . " $file could not be
created. Something went wrong with the grep (permissions?)";
+ wmf_audit_log_error( $message, "WR1_LOG_FATAL"
);
+ return null;
+ }
+
+ } else {
+ $message = "$original_file not found. Please copy that
file to the specified directory and re-run.";
+ wmf_audit_log_error( $message, "WR1_LOG" );
+ return null;
+ }
+ }
+
+}
+
+/**
+ * Given a file name, should return the Ymd date portion of the filename.
+ * This should work on both .log files, and .iop files generated by this
script.
+ * @param string $file Name of the file. Either a full path, or not.
+ * @return string The file's date in the format Ymd
+ */
+function wmf_audit_get_log_date_by_file( $file ){
+ $file = explode('/', $file);
+ $file = $file[count($file) - 1];
+
+ $date = false;
+ if ( preg_match( '/\d{8}_iop\.log/', $file ) ){
+ $date = str_replace('_iop.log', '', $file);
+ }
+ if ( preg_match( '/globalcollect_\d{8}\.log/', $file ) ){
+ $date = str_replace(array('globalcollect_', '.log'),
array('',''), $file);
+ }
+ if ( preg_match( '/payments-globalcollect-\d{8}\.gz/', $file ) ){
+ $date = str_replace(array('payments-globalcollect-', '.gz'),
array('',''), $file);
+ }
+ if ( preg_match( '/globalcollect.\d{8}_\d{6}/', $file ) ){ //this
happens sometimes
+ $date = str_replace('globalcollect.', '', $file);
+ $date = substr($date, 8);
+ }
+
+ if ( !is_numeric( $date ) ){
+ return false;
+ }
+
+ //Subtract one from the date, because the logs rotate at midnight, so
all
+ //the actual transactions are from the previous day.
+ $date = wmf_common_date_add_days( $date, -1 );
+ return $date;
+}
+
+
+/**
+ * Logs the errors we get in a consistent way
+ * @param type $watchdog_message
+ * @param type $drush_code
+ * @param type $drush_message
+ */
+function wmf_audit_log_error( $watchdog_message, $drush_code, $drush_message =
null ){
+ global $gc_audit_errors;
+ if ( is_null( $drush_message ) ){
+ $drush_message = $watchdog_message;
+ }
+
+ //hijacking this for missing logs. These are clogging the errors, and
I'd rather have the ball at the end anyway.
+ if ( $drush_code === 'WR1_LOG' ){
+ $watchdog_message = explode(' ', $watchdog_message);
+ $watchdog_message = explode('/', $watchdog_message[0]);
+ wmf_audit_results_ball('Missing Logs',
$watchdog_message[count($watchdog_message) - 1]);
+ return;
+ } else {
+ //ball it up anyway.
+ wmf_audit_results_ball( $drush_code, $drush_message );
+ }
+
+ watchdog('wmf_audit', $watchdog_message, NULL, WATCHDOG_ERROR);
+
+ //STOP EXPLODING.
+ if (wmf_audit_error_isfatal( $drush_code )) {
+ $gc_audit_errors[$drush_code] = $drush_message;
+ }
+}
+
+/**
+ * Gather output info in a static, because that's... less bad than a global
+ * @param mixed $key Key denoting the type of data we are storing. Could be a
+ * string, or an array with two values (key and subkey)
+ * @param string $value The value we want to add to that key. Default null
+ * @param bool $reset Set to true, if you want to wipe all values in $key (and
+ * set $reset to true and $key to false if you want to wipe everything).
+ * Default false.
+ * @return array The entire ball of final results so far.
+ */
+function wmf_audit_results_ball($key = null, $value = null, $reset = false){
+ static $data_ball = array();
+ $subkey = null;
+ if ($key && is_array($key)){
+ $subkey = $key[1];
+ $key = $key[0];
+ }
+
+ //reset first
+ if ( $reset ){
+ if ( $key && array_key_exists( $key, $data_ball ) ){
+ if ( $subkey && array_key_exists( $subkey,
$data_ball[$key] ) ){
+ unset($data_ball[$key][$subkey]);
+ }
+ if ( !$subkey ) {
+ unset($data_ball[$key]);
+ }
+ }
+ if ( !$key ){
+ $data_ball = array();
+ }
+ }
+
+ //then add the value
+ if ( $key && !is_null( $value ) ){
+ if ( $subkey ){
+ //direct assignment
+ $data_ball[$key][$subkey] = $value;
+ } else {
+ //add a value to the array
+ $data_ball[$key][] = $value;
+ }
+ }
+
+ return $data_ball;
+}
+
+/**
+ * Function determines if the audit can continue.
+ * Will return false if we've thrown an error that is of a type that should be
+ * fatal.
+ * @return bool True if the audit can soldier on, else false
+ */
+function wmf_audit_can_continue(){
+ //check for errors
+ global $gc_audit_errors;
+ if ( count( $gc_audit_errors ) ){
+ //nonfatal errors:
+ //"WR1_DATA_FORMAT"
+
+ if ( array_key_exists( 'WR1_LOG_STOP', $gc_audit_errors ) ){
+ //If the only thing in the errors is a log stop, don't
blow up.
+ //If there's other stuff, we should blow up with all of
it.
+ if ( count( $gc_audit_errors ) !== 1 ){
+ foreach ( $gc_audit_errors as $code => $thing ){
+ drush_set_error($code, $thing);
+ }
+ }
+ return false;
+ }
+
+ foreach ( $gc_audit_errors as $code => $thing){
+ if (wmf_audit_error_isfatal( $code )) {
+ return false;
+ }
+ }
+ }
+ return true;
+}
+
+/**
+ * Returns an array of errors that should not cause the script to blow up.
+ * @return type
+ */
+function wmf_audit_error_isfatal( $error ){
+ $nonfatal = array(
+ 'WR1_DATA_FORMAT',
+ 'MISSING_MANDATORY_DATA',
+ 'WR1_LOG',
+ 'MISSING_PARENT',
+ 'PARENT_OVERFLOW',
+ 'RECURRING_CBK',
+ 'CBK+RFD',
+ );
+
+ if ( in_array( $error, $nonfatal) ){
+ return false;
+ } else {
+ return true;
+ }
+}
+
+function wmf_audit_get_partial_xml_node_value( $node, $xml ){
+ $node1 = "<$node>";
+ $node2 = "</$node>";
+
+ $valstart = strpos( $xml, $node1 ) + strlen( $node1 );
+ if ( !$valstart ){
+ return null;
+ }
+
+ $valend = strpos( $xml, $node2 );
+ if ( !$valend ){ //it cut off in that node. This next thing is
therefore safe(ish).
+ $valend = strpos( $xml, '</' );
+ }
+
+ if ( !$valend ){
+ return null;
+ }
+
+ $value = substr( $xml, $valstart, $valend - $valstart );
+ return $value;
+
+}
+
+
+function wmf_audit_cc_company_to_submethod( $cc_company ){
+ $cc_array = array(
+ 'AMEX' => 'amex',
+ 'ECMC' => 'mc',
+ 'VISA' => 'visa',
+ 'DISC' => 'discover',
+ );
+
+ if ( array_key_exists( $cc_company, $cc_array ) ){
+ return $cc_array[$cc_company];
+ } else {
+ //I am dying here, because if you're running in make_missing
mode
+ //(which is currently the only way to get here)
+ //you dang well better be doing it manually.
+ wmf_audit_echo("$cc_company is not a known Credit Card Company
code! Fixit.");
+ die();
+ }
+}
+
+
+function wmf_audit_echo( $echo, $verbose = false ) {
+ if ( $verbose && variable_get( 'wmf_audit_verbose', false ) === false )
{
+ return;
+ }
+ static $chars = 0;
+ static $limit = null;
+ if (is_null($limit)){
+ $limit = variable_get( 'wmf_audit_charlimit', 0);
+ }
+
+ if ( strlen( $echo ) === 1 ){
+ echo $echo;
+ ++$chars;
+ if ($limit > 0 && $chars > $limit){
+ echo "\n";
+ $chars = 0;
+ }
+ } else {
+ //echo a whole line. Gets a little tricky.
+ if ( $chars != 0 ){
+ echo "\n";
+ }
+ echo "$echo\n";
+ $chars = 0;
+ }
+}
+
+
+function gca_recon_key_grinder( $record, $keys, $name, $error_on_mismatch =
false ){
+
+ if ( $error_on_mismatch ){
+ $ret = null;
+ $first = '';
+ foreach ( $keys as $key ){
+ if (array_key_exists( $key, $record )){
+ if ( is_null($ret) ){
+ $ret = trim($record[$key]);
+ $first = $key;
+ } else {
+ if ( trim($record[$key]) != '' && $ret
!= trim($record[$key]) ){
+ wmf_audit_echo(print_r($record,
true));
+ wmf_audit_echo("Check it: The
$name is all f'd. $first does not match $key (at least).");
+
+ //insert special "don't die"
rules here.
+ //On occasion, these are
predictable and we can cope with them going in "wrong"...
+ $die = true;
+ if ( $name === 'currency'
+ && $record['WbC Payment
method ID'] === '8'
+ && $record['WbC Payment
product ID'] === '841' ){
+ $ret =
$record['Payment-currency'];
+ wmf_audit_echo("...but
we're (sort of) fine with the Webmoney mismatch. Setting $name to $ret");
+ $die = false;
+ }
+
+ if ( $die && !( variable_get(
'wmf_audit_mismatch_override', false ) ) ){
+ die();
+ } else {
+ //short-circuit the
rest of this generalized business. We know what's up.
+ return $ret;
+ }
+ }
+ }
+ }
+ }
+ return $ret;
+
+ } else {
+ foreach ( $keys as $key ){
+ if ( array_key_exists( $key, $record ) &&
trim($record[$key]) !== '' ){
+ return $record[$key];
+ }
+ }
+ //implicit else
+ //TODO: Remove this mess when you know it's not going to...
fail to kill things.
+ $checked = array(
+ 'sign',
+ );
+ if ( !in_array( $name, $checked ) ){
+ wmf_audit_echo("No $name found for the following
record: " . print_r( $record, true ));
+ die();
+ }
+ return null;
+ }
+}
+
+function wmf_audit_send_stomp( $queueId, $body ) {
+ static $q;
+
+ if ( !$q ) {
+ $q = new Queue( queue2civicrm_stomp_url() );
+ }
+
+ // FIXME: register the queue mapping somewhere sane
+ if ( $queueId === 'donations' ) {
+ $queuePath = variable_get( 'queue2civicrm_subscription', '/queue/test'
);
+ } elseif ( $queueId === 'refund' ) {
+ $queuePath = variable_get( 'refund_queue',
'/queue/refund-notifications_test' );
+ } else {
+ throw new Exception( "What kind of a queue is this??: {$queueId}" );
+ }
+
+ $headers = array();
+ wmf_common_set_message_source( $headers, 'audit', 'GlobalCollect WR1
Auditor' );
+
+ return $q->enqueue( json_encode( $body ), $headers, $queuePath );
+}
+
+/**
+ * If the old "order id" is actually a newstyle merchantref
+ * ($contribution_tracking . ':' . $rando_whatever)
+ * return the contribution_tracking id portion. Otherwise, return false.
+ * @param string $order_id The thing we used to think was always order_id
+ * @return mixed The contribution_tracking ID | false
+ */
+function gca_recon_get_ctid_from_old_order_id($order_id) {
+ if (strpos($order_id, '.') > 1) {
+ $contribution_id = explode('.', $order_id);
+ return $contribution_id[0];
+ } else {
+ return false;
+ }
+}
+
+/**
+ * Continuation of the exceedingly funny story in which the field we thought
was
+ * order_id, turned out to be Merchant Reference.
+ * Please burn this entire script to the ground at your earliest convenience.
+ * kthnx.
+ * @param type $oid_or_ctid_iono
+ * @return The db matches if they exist, or false
+ */
+function wmf_audit_get_contributions_from_whatever_this_is($oid_or_ctid_iono) {
+ $ctid = gca_recon_get_ctid_from_old_order_id($oid_or_ctid_iono);
+ $things = false;
+ if ($ctid) {
+ //get the contribution from contribution_tracking
+ $ct_row = wmf_audit_get_contribution_tracking_data($ctid);
+ if (!$ct_row) {
+ return false;
+ }
+ if (isset($ct_row['contribution_id'])) {
+ $cc_id = trim($ct_row['contribution_id']);
+ $things =
wmf_civicrm_get_contributions_from_contribution_id($cc_id);
+ }
+ } else {
+ //oldschool
+ $order_id = $oid_or_ctid_iono;
+ $things =
wmf_civicrm_get_contributions_from_gateway_id('globalcollect', $order_id);
+ }
+
+ return $things;
+}
+
+/**
+ * ...I thought the last one was the punch line.
+ * @param type $oid_or_ctid_iono
+ * @return The db matches if they exist, or false
+ */
+function
wmf_audit_get_child_contributions_from_whatever_this_is($oid_or_ctid_iono) {
+ $ctid = gca_recon_get_ctid_from_old_order_id($oid_or_ctid_iono);
+ $things = false;
+ if ($ctid) {
+ //get the contribution from contribution_tracking
+ $ct_row = wmf_audit_get_contribution_tracking_data($ctid);
+ if (!$ct_row) {
+ return false;
+ }
+ if (isset($ct_row['contribution_id'])) {
+ //get the contribution, and from there, the order_id...
+ $cc_id = $ct_row['contribution_id'];
+ $things =
wmf_civicrm_get_child_contributions_from_contribution_id($cc_id);
+ }
+ } else {
+ //oldschool
+ $order_id = $oid_or_ctid_iono;
+ $things =
wmf_civicrm_get_child_contributions_from_gateway_id('globalcollect', $order_id);
+ }
+
+ return $things;
+}
diff --git a/sites/all/modules/wmf_audit/worldpay/worldpay_audit.drush.inc
b/sites/all/modules/wmf_audit/worldpay/worldpay_audit.drush.inc
new file mode 100644
index 0000000..eee18f9
--- /dev/null
+++ b/sites/all/modules/wmf_audit/worldpay/worldpay_audit.drush.inc
@@ -0,0 +1,117 @@
+<?php
+/**
+ * @file wmf_audit.drush.inc
+ * Parses .wr1 files from globalcollect and inserts any missing transactions
+ * into the regular stomp message queue.
+ * @author Katie Horn <[email protected]>
+ * @TODO print some useful info to STDOUT
+ */
+
+/**
+ * Implementation of hook_drush_command()
+ */
+function worldpay_audit_drush_command() {
+ $items = array();
+
+ $items['worldpay-audit'] = array(
+ 'description' =>
+ 'Worldpay Audit tool: Parses worldpay reconciliation files and inserts
any missing transactions into the message queues.',
+ 'examples' => array(
+ 'drush worldpay-audit' => '# Run the audit and rebuild job',
+ ),
+ 'options' => array(
+ 'run_all' => 'Batch search for all missing transactions across all
available recon files.',
+ 'test' => 'Test the audit and rebuild job, but do not generate stomp
messages. No data will be changed.',
+ 'fakedb' => 'Fake the database information. This will cause the script
to avoid looking up the actual contribution tracking id.',
+ 'makemissing' => 'Will reconstruct the un-rebuildable transactions found
in the recon file, with default values. USE WITH CAUTION: Currently this
prevents real data from entering the system if we ever get it.',
+ 'charlimit' => 'Will cause echoing to line break after the given number
of characters',
+ 'mismatch_override' => 'Won\'t die on a keygrind mismatch',
+ 'verbose' => 'Verbose output',
+ ),
+ 'aliases' => array('wp_audit'),
+ );
+
+ return $items;
+}
+
+/**
+ * Implementation of hook_drush_help()
+ */
+function worldpay_audit_drush_help($section) {
+ switch ( $section ) {
+ case 'worldpay-audit':
+ return dt("Worldpay Audit tool: Parses worldpay reconciliation files and
inserts any missing transactions into the message queues.");
+ }
+}
+
+function worldpay_audit_reset_drush_vars() {
+ static $vars = null;
+ if ( is_null( $vars ) ){
+ $vars = array(
+ 'worldpay_audit_test_mode' =>
variable_get('worldpay_audit_test_mode', WP_AUDIT_TEST_MODE),
+ 'worldpay_audit_fake_db' => false,
+ 'worldpay_audit_make_missing' => false,
+ 'worldpay_audit_run_all_wr1' => false,
+ 'worldpay_audit_charlimit' => 0,
+ 'worldpay_audit_mismatch_override' => false,
+ 'worldpay_audit_verbose' => false,
+ );
+ }
+
+ foreach ( $vars as $var => $default ){
+ variable_set( $var, $default );
+ }
+}
+
+/**
+ * Fires the 'wmf_audit_main' method with the appropriate parameters
+ */
+function drush_worldpay_audit() {
+ //if the last execution died before we got back here
+ worldpay_audit_reset_drush_vars();
+
+ $simple_opts = array(
+ 'test' => 'Running in test mode: No stomp messages will be sent',
+ 'fakedb' => 'Faking Database',
+ 'make_missing' => 'Making missing transactions---',
+ 'run_all' => 'Running all WR1 Files! This might take a while...',
+ 'mismatch_override' => 'Overriding keygrind mismatches. Problem records
will not be fatal, but they will still be logged.',
+ 'verbose' => 'Outputting verbose.',
+ );
+
+ $args = array();
+
+ foreach ($simple_opts as $key => $message) {
+ if (drush_get_option($key)) {
+ echo "$message\n";
+ $args[$key] = true;
+ }
+ }
+
+ if (drush_get_option('charlimit')){
+ echo "Char limit of " . drush_get_option('charlimit') . " is in
effect.\n";
+ $args['charlimit'] = drush_get_option('charlimit');
+ }
+
+ module_invoke('wmf_audit', 'main', 'worldpay', $args);
+
+ //Be Kind: Rewind (your run settings).
+ worldpay_audit_reset_drush_vars();
+
+ echo "\n***RESULTS***";
+ $results = wmf_audit_results_ball();
+ echo "\n" . print_r($results, true);
+
+ $errors = drush_get_error_log();
+ if (!empty($errors)){
+ echo "\n***ERRORS***";
+ foreach($errors as $error=>$msgarray){
+ echo "\n$error: ";
+ foreach ($msgarray as $count=>$message){
+ echo "\n $message";
+ }
+ }
+ echo "\n\n";
+ exit(drush_get_error());
+ }
+}
diff --git a/sites/all/modules/wmf_audit/worldpay/worldpay_audit.info
b/sites/all/modules/wmf_audit/worldpay/worldpay_audit.info
new file mode 100644
index 0000000..e989061
--- /dev/null
+++ b/sites/all/modules/wmf_audit/worldpay/worldpay_audit.info
@@ -0,0 +1,6 @@
+name = WorldPay Audit
+description = Auditing framework for contacts and contributions that come in
through nightly reconciliation files, for WorldPay
+core = 7.x
+dependencies[] = wmf_audit
+package = WMF Audit
+configure = admin/config/wmf_audit/worldpay_audit
diff --git a/sites/all/modules/wmf_audit/worldpay/worldpay_audit.module
b/sites/all/modules/wmf_audit/worldpay/worldpay_audit.module
new file mode 100644
index 0000000..9b65d53
--- /dev/null
+++ b/sites/all/modules/wmf_audit/worldpay/worldpay_audit.module
@@ -0,0 +1,668 @@
+<?php
+
+define( 'WP_AUDIT_RECON_PARSER_DIR', '/usr/local/src/WP_Parser_Lib/' );
+define( 'WP_AUDIT_RECON_FILES_DIR', '/usr/local/src/recon_files/' );
+define( 'WP_AUDIT_RECON_COMPLETED_DIR',
'/usr/local/src/recon_files-completed/' );
+define( 'WP_AUDIT_PLUSMINUS', 7 );
+define( 'WP_AUDIT_TEST_MODE', true );
+define( 'WP_AUDIT_FAKE_DB', false );
+
+/**
+ * Implementation of hook_menu()
+ */
+function worldpay_audit_menu() {
+ $items = array();
+
+ $items['admin/config/wmf_audit/worldpay_audit'] = array(
+ 'title' => 'WorldPay Audit',
+ 'description' => t('Configure WMF audit settings.'),
+ 'access arguments' => array('administer wmf_audit'),
+ 'page callback' => 'drupal_get_form',
+ 'page arguments' => array( 'worldpay_audit_settings' ),
+ );
+
+ return $items;
+}
+
+/**
+ * Callback for menu
+ */
+function worldpay_audit_settings() {
+ $form['worldpay_audit_recon_parser_dir'] = array(
+ '#type' => 'textfield',
+ '#title' => t( 'Path to directory containing reconciliation file parser
classes' ),
+ '#required' => TRUE,
+ '#default_value' => variable_get( 'worldpay_audit_recon_parser_dir',
WP_AUDIT_RECON_PARSER_DIR ),
+ );
+ $form['worldpay_audit_recon_files_dir'] = array(
+ '#type' => 'textfield',
+ '#title' => t( 'Directory containing incoming reconciliation files' ),
+ '#required' => TRUE,
+ '#default_value' => variable_get( 'worldpay_audit_recon_files_dir',
WP_AUDIT_RECON_FILES_DIR ),
+ );
+ $form['worldpay_audit_recon_completed_dir'] = array(
+ '#type' => 'textfield',
+ '#title' => t( 'Directory for completed reconciliation files' ),
+ '#required' => TRUE,
+ '#default_value' => variable_get( 'worldpay_audit_recon_completed_dir',
WP_AUDIT_RECON_COMPLETED_DIR ),
+ );
+ $form['worldpay_audit_test_mode'] = array(
+ '#type' => 'checkbox',
+ '#title' => t( 'When this box is checked, no stomp messages will be sent.'
),
+ '#required' => FALSE,
+ '#default_value' => variable_get( 'worldpay_audit_test_mode',
WP_AUDIT_TEST_MODE ),
+ );
+ $form['worldpay_audit_plusminus'] = array(
+ '#type' => 'textfield',
+ '#title' => t( 'Plus or minus log search (in days)' ),
+ '#required' => TRUE,
+ '#default_value' => variable_get( 'worldpay_audit_plusminus',
WP_AUDIT_PLUSMINUS ),
+ );
+ return system_settings_form( $form );
+}
+
+//A pattern we'll cheaply strpos to determine which files in the recon
directory
+//we actually care about.
+function worldpay_audit_get_recon_filename_pattern(){
+ return '.RECON.WIKI.';
+}
+
+
+function worldpay_audit_get_recon_directory() {
+ return variable_get( 'worldpay_audit_recon_files_dir',
WP_AUDIT_RECON_FILES_DIR );
+}
+
+function worldpay_audit_get_recon_file_date( $file ){
+ //WP recon files look like the following thing:
+ // MA.PISCESSW.#M.RECON.WIKI.D280514
+ //For that, we'd want to return 20140518
+ $parts = explode('.', $file);
+ $end_piece = $parts[count($parts)-1];
+ $date = '20' . substr( $end_piece, 5, 2) . substr( $end_piece, 3, 2) .
substr( $end_piece, 1, 2);
+ return $date;
+}
+
+/**
+ * Just parse one wr1 file.
+ * @staticvar string $parser_directory The directory where the parser lives.
+ * @param string $file Absolute location of the wr1 you want to parse
+ * @return mixed An array of wr1 data, or false
+ */
+function worldpay_audit_parse_recon_file($file) {
+ static $parserClass = null;
+ if (is_null($parserClass)) {
+ $parser_directory = variable_get('worldpay_audit_recon_parser_dir',
WP_AUDIT_RECON_PARSER_DIR);
+
+ //TODO: Maybe something less cheap later.
+ require_once( $parser_directory . 'WorldPayAudit.php' );
+ require_once( $parser_directory . 'TransactionReconciliationFile.php' );
+ $parserClass = 'SmashPig\PaymentProviders\WorldPay\Audit\WorldPayAudit';
//yeesh.
+ }
+
+
+ $recon_data = array();
+ wmf_audit_echo("Parsing $file");
+
+ $recon_parser = new $parserClass();
+
+ $start_time = microtime(true);
+
+ $data = null;
+ try {
+ $data = $recon_parser->parseFile($file);
+ } catch (Excepction $e) {
+ error_log("thing went wrong...");
+ }
+
+ echo print_r(count($data) . ' rows parsed in the thingy.');
+
+ echo 'sample: ' . print_r($data[2], true);
+
+ die(__FUNCTION__ . ' is where we currently drop dead. WIP: Anything in this
function below this line is probably BS.');
+
+ foreach( $recon_parser->getRecordIterator() as $record ) {
+ wmf_audit_echo(recon_echochar($record));
+ $order_id = recon_order_id( $record );
+ if ( recon_is_negative_txn( $record ) ){
+ //have to do this to avoid collisions between the
original and the chargeback / refund / whatever
+ $order_id = '-' . $order_id;
+ }
+ $recon_data[ $order_id ] = $record;
+ }
+ worldpay_audit_echo("\n");
+ $time = microtime(true) - $start_time;
+ worldpay_audit_echo("$file parsed in $time");
+ if ( count( $recon_data ) ){
+ return $recon_data;
+ }
+ return false;
+}
+
+/** EVERYTHING BELOW THIS LINE IS NONSENSE. * */
+
+/**
+ * Returns just the payment method and submethod codes from data contained in
+ * the $wr1 line.
+ * @param array $wr1 Transaction parsed into an array
+ * @return array
+ */
+function worldpay_audit_make_payment_method_info( $wr1 ){
+ $return = array();
+ if ( $wr1['Payment-method'] === 'CC' ){
+ $return['payment_method'] = 'cc';
+ $return['payment_submethod'] =
worldpay_audit_cc_company_to_submethod( $wr1['Creditcard-company'] );
+ } else {
+ if (array_key_exists( 'WbC Payment product ID', $wr1 )){
+ $subm = worldpay_audit_get_payment_submethod( $wr1['WbC
Payment product ID'] );
+ $return['payment_method'] = $subm['group'];
+ $return['payment_submethod'] = $subm['code'];
+ } else {
+ //dying here so I can run it against reality
+ worldpay_audit_echo(print_r($wr1, true));
+ worldpay_audit_echo("Need to handle this payment method
and submethod.");
+ die();
+ }
+ }
+
+ return $return;
+}
+
+
+
+//GODDAMN IT, it's trying to subdir the subdir.
+function worldpay_audit_move_completed_wr1( $file, $nearly = false ){
+ $subdir = 'completed';
+ if ($nearly){
+ $subdir .= '-nearly-' . date('Y-m-d', time());
+ }
+ $files_directory = variable_get( 'worldpay_audit_recon_completed_dir',
WP_AUDIT_WR1_COMPLETED_DIR );
+ $completed_dir = $files_directory . '/' . $subdir;
+ if ( !is_dir( $completed_dir ) ){
+ if ( !mkdir ( $completed_dir, 0700 )){
+ $message = "Could not make $completed_dir";
+ watchdog('globalcollect_audit', $message);
+ return false;
+ }
+ }
+
+ $filename = basename( $file );
+ $newfile = $completed_dir . '/' . $filename;
+
+ if (!rename( $file, $newfile )){
+ $message = "Unable to move $file to $newfile";
+ watchdog('globalcollect_audit', $message);
+ return false;
+ }
+ worldpay_audit_echo("Moved $file to $newfile");
+ return true;
+}
+
+
+function worldpay_audit_get_log_data_by_order_id($order_id, $log) {
+
+ //Funny story: This was never order_id. It was always MERCHANTREFERENCE...
+ //@TODO: Rename everything. :(
+ //Even funnier story: It doesn't seem to matter very much. This appears to
+ //be the only place that we take the non-logged XML order_id seriously. The
+ //donor data is rebuilt below in this function, from the original request.
+ if (!is_numeric($order_id)) {
+ throw new Exception("$order_id is not numeric; Tampering may have
occurred.");
+ }
+ $cmd = "grep <MERCHANTREFERENCE>$order_id</MERCHANTREFERENCE> $log";
+ worldpay_audit_echo( __FUNCTION__ . ' ' . $cmd, true );
+
+ //echo $cmd . "\n";
+ $ret = array();
+ exec( escapeshellcmd($cmd), $ret, $errorlevel );
+
+ if ( count( $ret ) > 0 ){
+
+ //get the xml, and the contribution_id... and while we're at
it, parse the xml.
+ $xml = null;
+ $full_xml = false;
+ $contribution_id = null;
+
+
+ //see if we have an old merchantreference (= order id ) or a new one.
+ //the new ones are contribution_tracking_id + . + random 4 digits.
+ $newstyle = wp_recon_get_ctid_from_old_order_id($order_id);
+ if ($newstyle) {
+ $contribution_id = $newstyle;
+ }
+
+ foreach ( $ret as $line ){
+ if ( is_null( $xml ) ){
+ //look for the raw xml.
+ if ( strpos( $line, 'RETURNED FROM CURL' ) ){
+ $xmlstart = strpos( $line, '<XML>' );
+ //$xmlend = strpos( $line, '</XML>' ) +
6;
+ $xmlend = strpos( $line, '</XML>' );
+ if ( $xmlend ){
+ $full_xml = true;
+ $xmlend += 6;
+ $xml = substr($line, $xmlstart,
$xmlend - $xmlstart);
+ } else {
+ //this is a broken line, and it
won't load... but we can still parse what's left of the thing, the slow way.
+ $xml = substr($line, $xmlstart);
+ }
+ }
+ }
+ if ( is_null( $contribution_id ) ){
+ //get the contribution tracking id.
+ $ctid_end = strpos( $line, 'Raw XML Response' );
+ if ( $ctid_end > 0 ){
+ $ctid_start = strpos( $line,
'_gateway:' ) + 9;
+ $ctid = substr($line, $ctid_start,
$ctid_end - $ctid_start);
+ $contribution_id = trim( $ctid, ' :' );
+ }
+ }
+ }
+
+ if ( is_null( $contribution_id ) || is_null( $xml ) ){
+ worldpay_audit_echo("OH NOES. Couldn't find enough info
in this log for order id $order_id!");
+ if ( !is_null( $contribution_id ) ){
+ worldpay_audit_echo( $contribution_id );
+ }
+ if ( !is_null( $xml ) ){
+ worldpay_audit_echo("...but we also have xml,
so that's odd.");
+ }
+ //TODO: This needs more love. We can't go on without
both parts, but we should log that we found some of it.
+
+ return null;
+ } else {
+ //go on with your bad self
+ //echo "$order_id: '$contribution_id'. XML: \n$xml\n";
+ //now parse the xml...
+
+ $donor_data = array();
+
+ if ( $full_xml ){
+ $xmlobj = new DomDocument;
+ $xmlobj->loadXML($xml);
+
+ $parent_nodes = array(
+ 'ORDER',
+ 'PAYMENT'
+ );
+
+ foreach ( $parent_nodes as $parent_node ){
+ foreach (
$xmlobj->getElementsByTagName( $parent_node ) as $node ) {
+ foreach ( $node->childNodes as
$childnode ) {
+ if ( trim(
$childnode->nodeValue ) != '' ) {
+
$donor_data[$childnode->nodeName] = $childnode->nodeValue;
+ }
+ }
+ }
+ }
+ } else {
+
+ $search_for_nodes = array(
+ 'ORDERID' => true,
+ 'AMOUNT' => true,
+ 'CURRENCYCODE' => true,
+ 'PAYMENTPRODUCTID' => true,
+ 'ORDERTYPE' => true,
+ 'EMAIL' => true,
+ 'FIRSTNAME' => false,
+ 'SURNAME' => false,
+ 'STREET' => false,
+ 'CITY' => false,
+ 'STATE' => false,
+ 'COUNTRYCODE' => true,
+ 'ZIP' => false,
+ );
+
+ foreach ( $search_for_nodes as $node =>
$mandatory ){
+ $tmp =
worldpay_audit_get_partial_xml_node_value( $node, $xml );
+ if ( !is_null( $tmp ) ){
+ $donor_data[$node] = $tmp;
+ } else {
+ if ( $mandatory ){
+ $wd_message =
__FUNCTION__ . ": Mandatory field $node missing for $contribution_id.
Aborting.";
+
worldpay_audit_log_error( $wd_message, 'MISSING_MANDATORY_DATA' );
+ return null;
+ } else {
+ $donor_data[$node] = '';
+ }
+ }
+ }
+ }
+
+ $return['contribution_id'] = $contribution_id;
+ $return['donor_data'] = $donor_data;
+ return $return;
+ }
+ }
+ return false;
+}
+
+
+function worldpay_audit_get_payment_submethod( $payment_product ){
+ //get the payment product as a number. Then, look it up.
+ $map = array(
+ 11 => array(
+ //GC Bank Transfer (new)
+ 'code' => 'bt',
+ 'group' => 'bt',
+ ),
+ //Credit Cards: If the card is specified, land it in
the right specific bucket.
+ //otherwise: cc
+ 1 => array(
+ 'code' => 'visa',
+ 'group' => 'cc',
+ ),
+ 3 => array(
+ 'code' => 'mc',
+ 'group' => 'cc',
+ ),
+ 2 => array(
+ 'code' => 'amex',
+ 'group' => 'cc',
+ ),
+ 117 => array(
+ 'code' => 'maestro',
+ 'group' => 'cc',
+ ),
+ 118 => array(
+ 'code' => 'solo',
+ 'group' => 'cc',
+ ),
+ 124 => array(
+ 'code' => 'laser',
+ 'group' => 'cc',
+ ),
+ 125 => array(
+ 'code' => 'jcb',
+ 'group' => 'cc',
+ ),
+ 128 => array(
+ 'code' => 'discover',
+ 'group' => 'cc',
+ ),
+ 130 => array(
+ 'code' => 'cb',
+ 'group' => 'cc',
+ ),
+//non-recurring pre-SEPA DD in country-alpha order
+ 703 => array(
+ 'code' => 'dd_at',
+ 'group' => 'dd',
+ ),
+ 706 => array(
+ 'code' => 'dd_be',
+ 'group' => 'dd',
+ ),
+ 707 => array(
+ 'code' => 'dd_ch',
+ 'group' => 'dd',
+ ),
+ 702 => array(
+ 'code' => 'dd_de',
+ 'group' => 'dd',
+ ),
+ 709 => array(
+ 'code' => 'dd_es',
+ 'group' => 'dd',
+ ),
+ 704 => array(
+ 'code' => 'dd_fr',
+ 'group' => 'dd',
+ ),
+ 705 => array(
+ 'code' => 'dd_gb',
+ 'group' => 'dd',
+ ),
+ 708 => array(
+ 'code' => 'dd_it',
+ 'group' => 'dd',
+ ),
+ 701 => array(
+ 'code' => 'dd_nl',
+ 'group' => 'dd',
+ ),
+//recurring pre-SEPA DD in country-alpha order
+ 713 => array(
+ 'code' => 'dd_at',
+ 'group' => 'dd',
+ ),
+ 716 => array(
+ 'code' => 'dd_be',
+ 'group' => 'dd',
+ ),
+ 717 => array(
+ 'code' => 'dd_ch',
+ 'group' => 'dd',
+ ),
+ 712 => array(
+ 'code' => 'dd_de',
+ 'group' => 'dd',
+ ),
+ 719 => array(
+ 'code' => 'dd_es',
+ 'group' => 'dd',
+ ),
+ 714 => array(
+ 'code' => 'dd_fr',
+ 'group' => 'dd',
+ ),
+ 715 => array(
+ 'code' => 'dd_gb',
+ 'group' => 'dd',
+ ),
+ 718 => array(
+ 'code' => 'dd_it',
+ 'group' => 'dd',
+ ),
+ 711 => array(
+ 'code' => 'dd_nl',
+ 'group' => 'dd',
+ ),
+//SEPA DD
+ 770 => array(
+ 'code' => 'dd_sepa',
+ 'group' => 'dd',
+ ),
+//ewallets
+ 840 => array(
+ //paypal
+ 'code' => 'ew_paypal',
+ 'group' => 'ew',
+ ),
+ 841 => array(
+ //webmoney
+ 'code' => 'ew_webmoney',
+ 'group' => 'ew',
+ ),
+ 843 => array(
+ //moneybookers
+ 'code' => 'ew_moneybookers',
+ 'group' => 'ew',
+ ),
+ 845 => array(
+ //cashu :)
+ 'code' => 'ew_cashu',
+ 'group' => 'ew',
+ ),
+ 849 => array(
+ //yandex
+ 'code' => 'ew_yandex',
+ 'group' => 'ew',
+ ),
+ 500 => array(
+ //bpay
+ 'code' => 'bpay',
+ 'group' => 'obt',
+ ),
+ 805 => array(
+ //nordea
+ 'code' => 'rtbt_nordea_sweden',
+ 'group' => 'rtbt',
+ ),
+ 809 => array(
+ //iDeal (exists)
+ 'code' => 'rtbt_ideal',
+ 'group' => 'rtbt',
+ ),
+ 810 => array(
+ //enets
+ 'code' => 'rtbt_enets',
+ 'group' => 'rtbt',
+ ),
+ 836 => array(
+ //sofort (exists!)
+ 'code' => 'rtbt_sofortuberweisung',
+ 'group' => 'rtbt',
+ ),
+ 856 => array(
+ //EPS
+ 'code' => 'rtbt_eps',
+ 'group' => 'rtbt',
+ ),
+ 1503 => array(
+ //Boleto
+ 'code' => 'cash_boleto',
+ 'group' => 'cash',
+ ),
+ );
+
+ return $map[$payment_product];
+
+}
+
+/**
+ * *** Normalization Helpers ***
+ * Everything from here down is a function meant to simplify the way we
+ * reference values returned in the wr1 record.
+ * PLEASE NOTE: I haven't decided if I hate this approach or not yet.
+ * If it turns out I hate it, I'll just write a normalization function for
+ * everything and run 'em all through the seive.
+ */
+
+
+function wp_recon_echochar( $record ){
+ if ($record['Record-category'] === '-'){
+ switch ( $record['Record-type'] ){
+ case 'CR':
+ return 'r';
+ break;
+ case 'CB':
+ return 'k';
+ break;
+ default:
+ return '-';
+ }
+ } else {
+ switch ( $record['Payment-method'] ){
+ case 'CC':
+ return 'c';
+ break;
+ case 'BA':
+ case 'OB':
+ return 'b';
+ break;
+ case 'PP': //this actually means "ewallets" to us
+ return 'e';
+ break;
+ case 'DD': //Direct Debit. Obviously.
+ return 'd';
+ break;
+ default:
+ return '+';
+ }
+ }
+}
+
+//This is wrong. All of these things actually contain MerchantReference.
+//@TODO: Burn it all down. Maybe do something with the pieces.
+function wp_recon_order_id( $record ){
+ $ordered_keys = array(
+ 'Order-number',
+ 'Additional-reference',
+ 'Reference-original-payment',
+ );
+
+ if ( recon_is_negative_txn( $record ) ){
+ $ordered_keys = array(
+ 'Reference-original-payment',
+ 'Order-number-original',
+ 'Order-number',
+ );
+ if ( $record['Record-type'] === 'CR' ){
+ unset ( $ordered_keys[0] );
+ }
+ }
+
+ return wp_recon_key_grinder($record, $ordered_keys, 'order ID');
+}
+
+function wp_recon_currency( $record ){
+ $ordered_keys = array(
+ 'Transaction-currency',
+ 'Payment-currency',
+ 'Invoice-currency-deliv',
+ 'Order-currency-deliv',
+ 'Invoice-currency-local',
+ 'Order-currency-local',
+ 'Currency-due',
+ 'Over-under-currency-local',
+ );
+ return wp_recon_key_grinder($record, $ordered_keys, 'currency', true);
+}
+
+function wp_recon_date( $record ){
+ $ordered_keys = array(
+ 'Date-authorised',
+ 'Date-order',
+ 'Date-due',
+ );
+ return wp_recon_key_grinder($record, $ordered_keys, 'date');
+}
+
+function wp_recon_date_to_timestamp( $record ){
+ $parsed = date_parse( recon_date($record) );
+
+ return mktime(
+ '0',
+ '0',
+ '0',
+ $parsed['month'],
+ $parsed['day'],
+ $parsed['year']
+ );
+}
+
+function wp_recon_amount( $record ){
+ $ordered_keys = array(
+ 'Payment-amount',
+ 'Amount-due',
+ );
+ return wp_recon_key_grinder($record, $ordered_keys, 'amount');
+}
+
+function wp_recon_is_negative_txn( $record ){
+ $ordered_keys = array(
+ 'Amount-sign_payment',
+ 'Amount-sign_amount',
+ );
+ $sign = wp_recon_key_grinder($record, $ordered_keys, 'sign');
+ if ( $sign === '-' ) {
+ return true;
+ }
+ return false;
+}
+
+function wp_recon_get_negative_type( $record ){
+ $refund_type = false;
+ if ($record['Record-category'] === '-'){
+ switch ( $record['Record-type'] ){
+ case 'CR':
+ $refund_type = 'refund';
+ break;
+ case 'CB':
+ $refund_type = 'chargeback';
+ break;
+ }
+ }
+
+ return $refund_type;
+}
--
To view, visit https://gerrit.wikimedia.org/r/159761
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4020e3ac9e84b1b2c3bfbb44a874abd95e8e5df8
Gerrit-PatchSet: 1
Gerrit-Project: wikimedia/fundraising/crm
Gerrit-Branch: master
Gerrit-Owner: Katie Horn <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
